Using Spark with Talend Data Mapper - 7.1

1. In Talend Studio, navigate to Repository > Metadata.
2. Right-click Hadoop Cluster and select Create Hadoop Cluster.
3. Enter a name for your cluster and click Next.
4. Select your distribution, Cloudera in this example, and select the version.
5. Select Retrieve configuration from Ambari or Cloudera and click Next.
6. Enter your Cloudera Manager credentials.
   
7. Click Connect to populate the Discovered clusters section, where you can fetch the services from.
8. Click Next.
9. Click Check Services to verify if all the services are activated. Then check the service status.

Create a structure for your map.

firstName,lastName,age
John,Doe,20
Jane,Doe,35
Kid,Doe,02

1. Open the Mapping perspective and navigate to Data Mapper > Hierarchical Mapper.
2. Right-click Structures and select New > Structure.
3. Select Import a structure definition and click Next.
4. Select CSV and click Next.
5. Select the sample file to use and click Next.
6. Enter the name of the structure and click Next.
7. In the CSV properties, select the Skip header reading check box, then click Next and Finish.

After you have created a Hadoop Cluster and a Structure, design the Big Data Batch Job including the tHDFSConfiguration, tHMapInput, and tLogRow components.

1. Open the Integration perspective and navigate to Repository > Job Designs.
2. Right-click Big Data Batch and select Create Big Data Batch Job.
3. Enter the necessary details to create the Job.
4. Drag the Hadoop Cluster metadata you created into the Job Design and select the tHDFSConfiguration component.
5. Add a tHMapInput and a tLogRow and connect these using Row > Main connection.
   1. Enter Output, when prompted for the output name.
6. Double-click the tLogRow and define its schema:
   1. Click the […] button next to Edit schema.
   2. In the Output (Input) section, click the + to add three new columns and name them firstName, lastName and age.
   3. Click the button to copy the columns to tLogRow_1 (Output).
   
7. Click the tHMapInput and open the Basic Settings tab.
   1. Select the Define a storage configuration component check box and select the tHDFSConfiguration component as the chosen storage.
   2. Specify the input file in the Input field.
   3. Click the […] button next to Configure Component and select the structure you created earlier.
   4. Select CSV in the Input Representation drop-down list.
   5. Click Next and add the input file in the Sample File field, then click Run to check the number records found.
      
   6. Click Finish.

Map the elements from the input to the output structure and run the Job.

1. Drag the input row element on the output OutputRecord element.
   All the elements are automatically mapped.

   
2. Click Test Run to see a preview of the output.
   In this example, it looks like this when displayed as JSON:

   {
     "Output" : {
       "OutputRecord" : [ {
         "firstName" : "firstName",
         "lastName" : "lastName",
         "age" : "age"
       }, {
         "firstName" : "John",
         "lastName" : "Doe",
         "age" : "20"
       }, {
         "firstName" : "Jane",
         "lastName" : "Doe",
         "age" : "35"
       }, {
         "firstName" : "Kid",
         "lastName" : "Doe",
         "age" : "02"
       } ]
     }
   }

3. Save the map and go back to the Big Data Batch Job.
4. Open the Run tab and click Run to execute the Job.