ON THIS PAGE

DID THIS PAGE HELP YOU?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!

Configuring a HDFS connection to run on Spark

Using the tHDFSConfiguration component, you can connect your HDFS filesystem to Spark.

This tutorial makes use of a Hadoop cluster. You must have a Hadoop cluster available to you.
You must also have HDFS metadata configured (see Creating a Hadoop cluster metadata definition and Importing a Hadoop cluster metadata definition).

In the Repository, expand Metadata > Hadoop Cluster, then expand the Hadoop cluster metadata of your choice.
1. Expand the HDFS folder of your Hadoop cluster metadata.
2. Drag-and-drop the HDFS metadata onto the Designer.
3. Select a tHDFSConfiguration component.
  The Hadoop Configuration Update Confirmation window opens.
Click OK.

Talend Studio updates the Spark configuration so that it corresponds to your cluster metadata.

In the Run view, click Spark Configuration. The execution is configured with the HDFS connection metadata.

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!