First steps using Big Data in Talend Studio - 8.0

Creating a project is the first step to using Talend Studio. Projects allow you to better organize your work.

1. Select Create a new project.
2. Enter a name for your project.

   Example

   TalendDemo
3. Click Create.
4. Click Finish.

Talend Studio projects contain Jobs. In Jobs, you can build workflows through components, which allow you to complete specific actions.

1. In Repository, right-click Job Designs.
   1. Click Create Standard Job.
2. In the Name field, enter a name.

   Example

   ReadWriteHDFS
3. Optional: In the Purpose field, enter a purpose.

   Example

   Read/Write data in HDFS
4. Optional: In the Description field, enter a description.

   Example

   Standard job to write and read customers data to and from HDFS
   Tip: Enter a Purpose and Description to stay organized.
5. Click Finish.

You can create a Hadoop cluster metadata definition to be able to quickly configure component with your Hadoop cluster information. Talend Studio also allows you to import a cluster metadata definition.

1. In the Repository, expand Metadata, right-click Hadoop Cluster and click Create Hadoop Cluster.
2. In the Name field, enter a name.

   Example

   MyHadoopCluster
3. Optional: In the Purpose field, enter a purpose.

   Example

   Cluster connection metadata
4. Optional: In the Description field, enter a description.

   Example

   Metadata to connect to a Amazon EMR cluster
   Tip: Enter a Purpose and Description to stay organized.
5. Click Next.
6. Select a Distribution.

   Example

   Select Amazon EMR and EMR 5.15.0 (Hadoop 2.8.3).
7. Select a Version.

   Example

   Select EMR 5.15.0 (Hadoop 2.8.3).
8. Select Enter manually Hadoop services.
9. Click Finish.
   You are brought to the Hadoop Cluster Connection window.
10. Enter your Connection details.

    Example

    • Namecode URI: hdfs://hadoopcluster:8020
    • Resource Manager: hadoopcluster:8032
    • Resource Manager Scheduler: hadoopcluster:8030
    • Job History: hadoopcluster:10020
    • Staging directory: /user
11. Enter your Authentication details.

    Example

    • User name: student
12. Optional: Click Check Services.
13. Click Finish.

You can import your Hadoop cluster configuration to create a Hadoop cluster metadata definition to be able to quickly configure components with its information. Talend Studio also allows you to create a cluster metadata definition from scratch.

1. In the Repository, expand Metadata, right-click Hadoop Cluster and click Create Hadoop Cluster.
2. In the Name field, enter a name.

   Example

   MyHadoopCluster_files
3. Optional: In the Purpose field, enter a purpose.

   Example

   Cluster connection metadata
4. Optional: In the Description field, enter a description.

   Example

   Metadata to connect to a Cloudera CDH cluster
   Tip: Enter a Purpose and Description to stay organized.
5. Click Next.
6. Select a Distribution.

   Example

   Select Cloudera.
7. Select a Version.

   Example

   Select Cloudera CDH6.1.1 [Built in].
8. Select Import configuration from local files.
9. Click Next.
10. Under Location, select the file of your choice in the File Explorer.
11. Select your modules.

    Example

    Select HDFS or YARN.
12. Click Finish.

    Example

    You are brought to the Hadoop Cluster Connection window, and your Connection details have been entered already.
13. Optional: Click Check Services.
14. Click Finish.

With the help of the tRowGenerator component, Talend Studio can create random data to help you test its features.

1. Add a tRowGenerator component.
   This component helps you generate random data for testing purposes.
2. Double-click the tRowGenerator component.
   You are brought in the tRowGenerator configuration window.
3. Click the plus button to add a Column.
   1. In the Column field, enter a name.

      Example

      1. CustomerID
      2. FirstName
      3. LastName
   2. Select the column Types.

      Example

      1. For CustomerID, select the Integer Type.
      2. For FirstName and LastName, select the String Type.
   3. Select the column Functions.

      Example

      1. For CustomerID, select the Numeric.random(int,int) function.

         This function generates random numbers.

      2. For FirstName, select the TalendDataGenerator.getFirstName() function.

         This function generates random first names.

      3. For LastName, select the TalendDataGenerator.getLastName() function.

         This function generates random last names.

4. Optional: Configure your Columns.

   Example

   For CustomerID, in the Function parameters tab, enter a max value of 1000.
5. Optional: Enter the number of your choice in the Number of Rows for RowGenerator field.

   Example

   Enter 1000 to create a 1000 customers.
6. Click OK.

What to do next

Click the Preview button in the Preview tab to try this feature.

Using the tHDFSOutput component, you can write data to HDFS.

1. In the Repository, expand Metadata > Hadoop Cluster, then expand the Hadoop cluster metadata of your choice.
   1. Drag-and-drop the HDFS metadata onto the Designer.
      You are brought to the Components window.
   2. Select a tHDFSOutput component.
2. Add an input component.

   Example

   Add a tRowGenerator component to generate fictional data for testing purposes (see Generating random data).
3. Right-click the input component.
   1. Select Row > Main
      
   2. Click on the tHDFSOutput component to link the two.
4. Double-click the tHDFSOutput component.

   The component is already configured with the predefined HDFS metadata connection information.

5. In the File Name field, enter the file path and name of your choice.
6. Optional: In Action, select Overwrite.

Using the tHDFSInput component, you can read data from HDFS.

1. In the Repository, expand Metadata > Hadoop Cluster, then expand the Hadoop cluster metadata of your choice.
   1. Drag-and-drop the HDFS metadata onto the Designer.
   2. Select a tHDFSInput component.
2. Double-click the tHDFSInput component.

   The component is already configured with the predefined HDFS metadata connection information.

3. In the File Name field, enter the file path and name of your choice.
4. Click the […] button next to Edit schema.
   
5. Click the plus button to add a Column.
   1. In the Column field, enter a name.

      Example

      1. CustomerID
      2. FirstName
      3. LastName
   2. Select the column Types.

      Example

      1. For CustomerID, select the Integer Type.
      2. For FirstName and LastName, select the String Type.
   3. Click OK.
6. Right-click the tRowGenerator component.
   1. Select Trigger > On Subjob OK.
   2. Click on the tHDFSInput component to link the two.
7. Add a tSortRow component.
8. Right-click the tHDFSInput component.
   1. Select Row > Main
   2. Click on the tSortRow component to link the two.
9. Double-click the tSortRow component.
   1. Click Sync columns.
      The tSortRow component inherits the schema from the tHDFSInput component.
10. Click the plus button.
    The first column of your tHDFSInput component schema appears.
11. Add a tLogRow component.
12. Right-click the tSortRow component.
    1. Select Row > Main
    2. Click on the tLogRow component to link the two.
       This is what your Designer should look like.

       
13. Double-click the tLogRow component.
    1. Select Table (print values in cells of a table).
14. In the Run view, click Run.