Setting up context-smart Hadoop connections

Setting up a connection to a given Hadoop distribution in Repository allows you to avoid configuring that connection each time when you need to use it in your Jobs.

When defining this connection, you can contextualize the connection parameters using values from different Hadoop environments such as from a test and a production environments, in order to adjust the connection and the Jobs using the connection to the proper environments at runtime by only one click.

The security configuration such as the Kerberos parameters cannot be contextualized. Therefore, ensure that the values you use for security work in all the involved environments among which the Hadoop connection switches.

If available in your Studio, the advanced Spark properties and the advanced Hadoop properties you define cannot be contextualized either. For this reason, ensure that these properties are valid for all the involved environments among which the Hadoop connection switches.

Setting up the Hadoop connection

You need first to set up the connection to a given Hadoop environment.

In this article, a Cloudera distribution is used for demonstration purposes.

Before you begin

Ensure that the client machine on which Talend Studio is installed can recognize the host names of the nodes of the Hadoop cluster to be used. For this purpose, add the IP address/hostname mapping entries for the services of that Hadoop cluster in the hosts file of the client machine.

For example, if the host name of the Hadoop Namenode server is talend-cdh550.weave.local, and its IP address is 192.168.x.x, the mapping entry reads 192.168.x.x talend-cdh550.weave.local.
The Hadoop cluster to be used has been properly configured and is running.
The Integration perspective is active.
Cloudera is the example distribution of the current article. If you are using a different distribution, you may need to bear in mind the particular prerequisites explained as follows:
- If you need to connect to MapR from the Studio, ensure that you have installed the MapR client in the machine where the Studio is, and added the MapR client library to the PATH variable of that machine. According to the MapR documentation, the library or libraries of a MapR client corresponding to each OS version can be found under MAPR_INSTALL\/hadoop\hadoop-VERSION/lib/native. For example, the library for Windows is \lib\native\MapRClient.dll in the MapR client jar file. For further information, see the following link from MapR: http://www.mapr.com/blog/basic-notes-on-configuring-eclipse-as-a-hadoop-development-environment-for-mapr.
- If you need to connect to a Google Dataproc cluster, set the path to the Google credentials file associated with the service account to be used in the environment variables of your local machine, so that the Check service feature of the metadata wizard can properly verify your configuration.
  
  For further information how to set the environment variable, see Getting Started with Authentication of Google documentation.

Procedure

In the Repository tree view of your Studio, expand Metadata and then right-click Hadoop cluster.
Select Create Hadoop cluster from the contextual menu to open the Hadoop cluster connection wizard.
Fill in generic information about this connection, such as Name and Description and click Next to open the [Hadoop Configuration Import Wizard] window that allows you to select the distribution to be used and the manual or the automatic mode to configure the connection.
- Retrieve configuration from Ambari or Cloudera: if you are using a Hortonworks Data Platform or a Cloudera CDH cluster and your cluster contains its specific management platform: Hortonworks Ambari for Hortonworks Data Platform and Cloudera manager for Cloudera CDH, select this check box to directly import the configuration.
- Import configuration from local files: when you have obtained or you can obtain the configuration files (mainly the *-site.xml files), for example, from the administrator of the Hadoop cluster or downloaded directly from the Web-based cluster management service, use this option to import the properties directly from those files.
- Enter manually Hadoop services: you click Finish and manually enter the connection parameters.
On either the automatic approaches or the manual approach, the parameters you need to define are:
- Namenode URI: enter the URI of the NameNode machine of the cluster to be used.
- Resource Manager and Resource Manager scheduler: enter the URI pointing to the machine used by the Resource Manager service of your cluster and the address of its scheduler, respectively.
- Job history: enter the location of the JobHistory server of your cluster. This allows the metrics information of the current Job to be stored in that JobHistory server.
- Staging directory: enter this directory defined in your Hadoop cluster for temporary files created by running programs. Typically, this directory can be found under the yarn.app.mapreduce.am.staging-dir property in the configuration files such as yarn-site.xml or mapred-site.xml of your distribution.
- Use datanode hostname: select this check box to allow the Job to access datanodes via their hostnames. This actually sets the dfs.client.use.datanode.hostname property to true.
- The User name field is available when you are not using Kerberos to authenticate. In the User name field, enter the login user name for your distribution. If you leave it empty, the user name of the machine hosting the Studio will be used.
Verify whether your cluster is security-enabled and bear in mind that the security configuration cannot be contextualized.

If you are accessing the Hadoop cluster running with Kerberos security, select this check box, then, enter the Kerberos principal names for the ResourceManager service and the JobHistory service in the displayed fields. This enables you to use your user name to authenticate against the credentials stored in Kerberos. These principals can be found in the configuration files of your distribution, such as in yarn-site.xml and in mapred-site.xml.

If you need to use a Kerberos keytab file to log in, select Use a keytab to authenticate. A keytab file contains pairs of Kerberos principals and encrypted keys. You need to enter the principal to be used in the Principal field and the access path to the keytab file itself in the Keytab field. This keytab file must be stored in the machine in which your Job actually runs, for example, on a Talend Jobserver.

Note that the user that executes a keytab-enabled Job is not necessarily the one a principal designates but must have the right to read the keytab file being used. For example, the user name you are using to execute a Job is user1 and the principal to be used is guest; in this situation, ensure that user1 has the right to read the keytab file to be used.
Add the advanced Hadoop properties if they are required by your cluster and bear in mind that these properties cannot be contextualized. Click the [...] button to open the properties table and add the property or properties to be customized. Then at runtime, these changes will override the corresponding default properties used by the Studio for its Hadoop engine.
If your Studio supports designing Apache Spark Jobs and your cluster expects some advanced Spark properties, select the Use Spark properties check box to open the properties table and add the property or properties to be used. Bear in mind that these properties cannot be contextualized.

When you reuse this connection in your Apache Spark Jobs, the advanced Spark properties you have added here are automatically added to the Spark configurations for those Jobs.
If you are using Cloudera V5.5+ to run your MapReduce or Apache Spark Batch Jobs, you can select the Use Cloudera Navigator check box to make use of Cloudera Navigator to trace the lineage of given data flow to discover how this data flow was generated by a Job. But bear in mind that the Cloudera Navigator configuration cannot be contextualized.
With this option activated, you need to set the following parameters:
- Username and Password: this is the credentials you use to connect to your Cloudera Navigator.
- Cloudera Navigator URL : enter the location of the Cloudera Navigator to be connected to.
- Cloudera Navigator Metadata URL: enter the location of the Navigator Metadata.
- Activate the autocommit option: select this check box to make Cloudera Navigator generate the lineage of the current Job at the end of the execution of this Job.
  
  Since this option actually forces Cloudera Navigator to generate lineages of all its available entities such as HDFS files and directories, Hive queries or Pig scripts, it is not recommended for the production environment because it will slow the Job.
- Kill the job if Cloudera Navigator fails: select this check box to stop the execution of the Job when the connection to your Cloudera Navigator fails.
  
  Otherwise, leave it clear to allow your Job to continue to run.
- Disable SSL validation: select this check box to make your Job to connect to Cloudera Navigator without the SSL validation process.
  
  This feature is meant to facilitate the test of your Job but is not recommended to be used in a production cluster.
Click the Check services button to verify that the Studio can connect to the NameNode and the ResourceManager services you have specified in this wizard. A dialog box pops up to indicate the checking process and the connection status. If it shows that the connection fails, you need to review and update the connection information you have defined in the connection wizard.
Click Finish to validate your changes and close the wizard.

The newly set-up Hadoop connection displays under the Hadoop cluster folder in the Repository tree view. This connection has no sub-folders until you create connections to any element under that Hadoop distribution.

Contextualizing the Hadoop connection parameters

Contextualize the Hadoop connection parameters to make this connection portable over different Hadoop environments such as a test environment and a production environment.

Before you begin

Ensure that the client machine on which Talend Studio is installed can recognize the host names of the nodes of the Hadoop cluster to be used. For this purpose, add the IP address/hostname mapping entries for the services of that Hadoop cluster in the hosts file of the client machine.

For example, if the host name of the Hadoop Namenode server is talend-cdh550.weave.local, and its IP address is 192.168.x.x, the mapping entry reads 192.168.x.x talend-cdh550.weave.local.
The Hadoop cluster to be used has been properly configured and is running.
A Hadoop connection has been properly set up following the explanations in Setting up the Hadoop connection.
The Integration perspective is active.
Cloudera is the example distribution of the current article. If you are using a different distribution, you may need to bear in mind the particular prerequisites explained as follows:
- If you need to connect to MapR from the Studio, ensure that you have installed the MapR client in the machine where the Studio is, and added the MapR client library to the PATH variable of that machine. According to the MapR documentation, the library or libraries of a MapR client corresponding to each OS version can be found under MAPR_INSTALL\/hadoop\hadoop-VERSION/lib/native. For example, the library for Windows is \lib\native\MapRClient.dll in the MapR client jar file. For further information, see the following link from MapR: http://www.mapr.com/blog/basic-notes-on-configuring-eclipse-as-a-hadoop-development-environment-for-mapr.
- If you need to connect to a Google Dataproc cluster, set the path to the Google credentials file associated with the service account to be used in the environment variables of your local machine, so that the Check service feature of the metadata wizard can properly verify your configuration.
  
  For further information how to set the environment variable, see Getting Started with Authentication of Google documentation.

Procedure

In the Repository tree view of your Studio, expand Metadata > Hadoop cluster and double-click the Hadoop connection you created following Setting up the Hadoop connection.
Click Next to go to the step 2 window of this wizard and click Export as context.
In the Create/Reuse a context group wizard, select Create a new repository context and click Next.
Enter a name for the context group, for example smart_connection, and click Next.

A read-only view of this context group is created and automatically filled with the parameters of the given Hadoop connection you defined in Setting up the Hadoop connection.

You may also notice that not all of the connection parameters are added to the context group, meaning that they are not all contextualized, as expected.
Click Finish to validate the creation and switch back to the step 2 window of the Hadoop connection wizard.

The connection parameters have been automatically set to use the context variables and become read-only.
Click Finish.

The new context group, named smart_connection, has been created under the Contexts node.
In Repository, double-click this new context group to open the Create/Edit a context group wizard.
Click Next to pass to step 2 in order to edit the context variables.
Click the [+] button on the right of the table to add a new context.
Click New and enter the name of this new context, for example, prod.
Click OK to validate the changes and close the New context wizard. The new context is added to the context list.
Click OK and close the Configure contexts wizard to go back to the Create/Edit a context group wizard.
Define the new context to contain the connection parameter values for a different Hadoop cluster, for example, your production one.
Click Finish to validate the changes and accept the propagation.
Back to the Hadoop cluster node in Repository, double-click the Hadoop connection you are contextualizing to open its wizard.
In the step 2 window of this wizard, ensure that the Use custom Hadoop configuration check box is selected and click Configuration to open the Hadoop configuration wizard.

The prod context is displayed in the wizard and the message "Please import the jar." next to it prompts you to import the Hadoop configuration file specific to the Hadoop cluster this prod context is created for.

You can also notice that the Default context that was the first context generated for this given Hadoop connection, smart_connection, already possesses a Hadoop configuration jar file. This jar file was automatically generated at the end the process of defining this Hadoop connection and creating the Default context for it.

You can also select the Set path to custom the Hadoop configure JAR check box to specify the path to the configuration JAR file.
Click the field of this "Please import the jar." message to display the [...] button and click this button to open the Hadoop configuration import wizard wizard.

This step starts the same process as explained in Setting up the Hadoop connection to either automatically or manually set up the Hadoop configuration. But at the end of this process, this step is meant to generate only the appropriate Hadoop configuration jar file for the prod context but not to create a new Hadoop connection item under the Hadoop cluster node.
Click OK to validate the changes and then click Finish to validate the contextualization and close the Hadoop connection wizard.

If prompted, click Yes to accept the propagation.
The Hadoop connection is now contextualized and you can continue to create child connections to its elements such as HBase, HDFS and Hive etc. based on this connection. Each connection wizard contains the Export as context button that you can use to contextualize each connection.

Results

When you reuse these connections via the Property type list in a given component in your Jobs, these contexts are listed in the Run view of the Job at your disposal.

Reusing a contextualized Hadoop connection in a Job

Before you begin

An empty Job has been created and opened in the workspace of the Studio.
A Hadoop connection and its child connections have been properly set up following the explanations in Setting up the Hadoop connection.

Procedure

In the workspace of the Studio, drop a Hadoop related component, for example, tHDFSConnection. This means that you must have created a contextualized HDFS connection under the contextualized Hadoop connection to be used.
Double-click this component to open its Component view.
From the Property type list, select Repository and click the [...] button to open the view of the repository content.
Select the HDFS connection to be used and click OK to validate the selection.
In the pop-up dialog box, click Yes to accept adding the contexts defined for the connection to the Job and in the window that is displayed, select the listed contexts.
Click OK to validate the addition.

Results

The contexts available for use can then be seen in the Run view of the Job at your disposal.

Creating a new Hadoop configuration context outside the Studio (optional)

You can contextualize the Hadoop connection for a Job without using the Studio.

When you do not have a Studio at hand but need to deploy a Job in a Hadoop environment different from the Hadoop environments already defined for this Job, you can take the manual approach to add a new Hadoop connection context.

If a Job is using a contextualized Hadoop connection that has two contexts, for example Default and Dev, after being built out of the Studio, the lib folder of the built artifact (the Job zip) contains two special jars for the given Hadoop environments. The name of these jars follows a pattern: "hadoop-conf-[name_of_the_metadata_in_the_repository]_[name_of_the_context].jar".

The jar to be used at runtime is defined by the context used in the command you can read from the .bat file or the .sh file of the Job.

The following line is an example of this command, which calls the Default context:

java -Xms256M -Xmx1024M -cp .;../lib/routines.jar;../lib/antlr-runtime-3.5.2.jar;../lib/avro-1.7.6-cdh5.10.1.jar;../lib/commons-cli-1.2.jar;../lib/commons-codec-1.9.jar;../lib/commons-collections-3.2.2.jar;../lib/commons-configuration-1.6.jar;../lib/commons-lang-2.6.jar;../lib/commons-logging-1.2.jar;../lib/dom4j-1.6.1.jar;../lib/guava-12.0.1.jar;../lib/hadoop-auth-2.6.0-cdh5.10.1.jar;../lib/hadoop-common-2.6.0-cdh5.10.1.jar;../lib/hadoop-hdfs-2.6.0-cdh5.10.1.jar;../lib/htrace-core4-4.0.1-incubating.jar;../lib/httpclient-4.3.3.jar;../lib/httpcore-4.3.3.jar;../lib/jackson-core-asl-1.8.8.jar;../lib/jackson-mapper-asl-1.8.8.jar;../lib/jersey-core-1.9.jar;../lib/log4j-1.2.16.jar;../lib/log4j-1.2.17.jar;../lib/org.talend.dataquality.parser.jar;../lib/protobuf-java-2.5.0.jar;../lib/servlet-api-2.5.jar;../lib/slf4j-api-1.7.5.jar;../lib/slf4j-log4j12-1.7.5.jar;../lib/talend_file_enhanced_20070724.jar;mytestjob_0_1.jar; local_project.mytestjob_0_1.myTestJob --context=Default %*

In this example, switching from Default to Dev results in changing the Hadoop configuration which will be loaded in the Job at runtime.

Adding a new Hadoop connection context manually to the built Job

You can manually add a Hadoop environment to the Job, without the help of the Studio.

Following the example described in Creating a new Hadoop configuration context outside the Studio (optional), add a Prod Hadoop environment.

Before you begin

This Job must be using contextualized Hadoop connections. This means that your Job is using the Repository property type to reuse the Hadoop connection for which contexts have been defined.

You can search for further information about how to use metadata in a Job on Talend Help Center.

You can search for further information about how to use metadata in a Job on Talend Help Center (https://help.talend.com).

For further information about how to define contexts for a Hadoop connection in the Studio, see Contextualizing the Hadoop connection parameters.
The Job you need to deploy must have been properly built from the Studio and is unzipped.

You can search for further information about how to build Jobs to deploy and execute them on any server, independent of Talend Studio, on Talend Help Center.

You can search for further information about how to build Jobs to deploy and execute them on any server, independent of Talend Studio, on Talend Help Center (https://help.talend.com).

Procedure

In the contexts folder, duplicate the Dev.properties and rename it Prod.properties.
In the lib folder, duplicate the hadoop-conf-cdh5100_Dev.jar and rename it hadoop-conf-cdh5100_Prod.jar.
Open the hadoop-conf-cdh5100_Prod.jar and replace the configuration files with those from the production cluster.

Results

You can then use the Prod context in the command to load the Prod configuration in the Job.