Creating and using metadata in Talend Studio

In this tutorial, discover how creating and using metadata in Talend Studio can help you save a lot of development time.

This tutorial makes use of a .csv file. If you do not have a .csv file, click the Downloads tab and save customers_unordered.csv.

This tutorial makes use of a database. If you do not have a database, click the Downloads tab and save customers_unordered.sql. You must import the database into a compatible program.

Understanding metadata in Talend Studio

Talend Studio allows you to create and run Jobs using predefined components. You can configure each component as either a Built-in or a Repository component. A Repository component is saved as metadata.

For Built-in components, information (such as how to read the file and what it contains):

is defined in the component,
applies only to this component,
cannot be reused with any other component.

For Repository components, information (such as how to read the file and what it contains):

is saved as metadata,
can be efficiently and consistently reused,
can be easily maintained because changes to metadata can be propagated to all Jobs that use it.

Creating a Talend Studio project

Creating a project is the first step to using Talend Studio. Projects allow you to better organize your work.

Procedure

Select Create a new project.
Enter a name for your project.
Example
TalendDemo
Click Create.
Click Finish.

Results

Your project opens. You are ready to work in Talend Studio.

Creating a Job to use metadata

Talend Studio projects contain Jobs. In Jobs, you can build workflows through components, which allow you to complete specific actions.

Before you begin

Select the Integration perspective (Window > Perspective > Integration).

Procedure

In Repository, right-click Job Designs.
1. Click Create Standard Job.
In the Name field, enter a name.
Example
useMetadata
Optional: In the Purpose field, enter a purpose.
Example
Display the use of Metadata in Talend Studio
Optional: In the Description field, enter a description.
Example
A simple job to demonstrate built-in properties vs. metadata

Tip: Enter a Purpose and Description to stay organized.
Click Finish.

Results

The Designer opens an empty Job.

Metadata configuration

By configuring metadata, you can configure reusable information across all your Talend Studio components.

Creating a metadata definition

Creating a metadata definition allows you to set up reusable information across all of your components.

Before you begin

This tutorial makes use of a .csv file. If you do not have a .csv file, click the Downloads tab and save customers_unordered.csv.
This tutorial also makes use of another delimited file. If you do not have another delimited file, click the Downloads tab and save directors.txt.

Procedure

In the Repository, expand Metadata then right-click File delimited and click Create file delimited.
In the Name field, enter a name.
Example
Customers
Optional: In the Purpose field, enter a purpose.
Example
Creating reusable metadata thanks to a .csv file
Optional: In the Description field, enter a description.
Example
Reusable shareable customer metadata

Tip: Enter a Purpose and Description to stay organized.
Click Next.
You are brought to Step 2 in the wizard.
Click Browse and select the file of your choice in the File Explorer.
Click Next.
You are brought to Step 3 in the wizard.
Optional: Define the parse settings.

Example
- Under File Settings, select your Field Separator and change it, if needed.
  Note: The most common Field Separator is ;
- Under Preview, select Set heading row as column names. The Header field automatically fills with the value 1, meaning the first row of your file has the file headers.
Tip: Under Preview, click Refresh Preview to check the parsing results.
Click Next.
You are brought to Step 4 in the wizard.
Optional: In the Name field, enter a name.
Example
customersSchema
Optional: Update the Schema so it is identical to the structure of the sample file.
Example
Change the Number Type to String. A postcode number does not serve an arithmetic function.
Click Finish.

Results

In the Repository, under Metadata, you can find and use your metadata.

Configuring a component through metadata

Configuring a component through metadata allows you to configure the component with predefined information.

Before you begin

You must have created a metadata definition (see Creating a metadata definition).

Procedure

Click inside the Designer.
Add a tFileInputDelimited component.

Note: By default, the component is configured with Built-in parameters.
Double-click the tFileInputDelimited component.
1. In the Property Type list, select Repository.
2. Click the […] button next to the Repository field.
3. Under Metadata > File delimited, select a metadata definition.
  Example
  customers 0.1
  
  All the Component fields are filled in with the metadata information. They now appear in gray, indicating that they belong to the metadata, and not to the component.

Results

You have succesfully configured a component through metadata.

Creating a metadata definition from a database

In Talend Studio, you can fetch a metadata definition from a database, making the best use of your existing resources.

Before you begin

This tutorial makes use of a database. If you do not have a database, click the Downloads tab and save customers_unordered.sql. You must import the database into a compatible program.

Procedure

In the Repository, expand Metadata then right-click Db Connections and click Create connection.
You are brought to Step 1 in the wizard.
In the Name field, enter a name.
Example
MySQL
Optional: In the Purpose field, enter a purpose.
Example
Demonstrate how to fetch metadata from a database
Optional: In the Description field, enter a description.
Example
Fetching CSV data imported into a MySQL database

Tip: Enter a Purpose and Description to stay organized.
Click Next.
You are brought to Step 2 in the wizard.
Enter your connection details.

Tip: To check the connection to the database, click Test connection.
Click Finish.
You are brought to the Designer.
Right-click your metadata in the Repository.
1. Click Retrieve Schema.
  You are brought to the Filter for the Table window.
2. Click Next.
  You are brought to the Add a Schema on repository window.
3. Select the check box next to the table name.
4. Click Next.
  The database schema displays.
5. Optional: Change the Schema parameters.
6. Click Finish.

Results

All table schemas have been imported as metadata and can be reused in your components.

Using metadata to read a database and displaying the results

You can read and manage databases in Talend Studio, allowing you to integrate them into your data management workflows.

Before you begin

This tutorial makes use of a database. If you do not have a database, click the Downloads tab and save customers_unordered.sql. You must import the database into a compatible program.
You must have created a metadata definition from a database (see Creating a metadata definition from a database).

Procedure

In the Repository, expand Metadata > Db Connections.
Drag-and-drop a database metadata on the Designer.
1. Select a tDBInput component.
  The component inherits the database schema.
In the Designer, add a tLogRow component.
Right-click the tDBInput component.
1. Select Row > Main.
2. Click on the tLogRow component to link the two.
Optional: In the tLogRow component, select the Table Mode.
In the Run view, click Run.

Results

The tDBInput component, configured through metadata, reads your database, and the tLogRow component displays its content on the console.