Creating a new Hadoop configuration context outside the Studio (optional) - 7.3

Setting up context-smart Hadoop connections

Version
7.3
Language
English
Product
Talend Big Data
Talend Data Fabric
Talend Open Studio for Big Data
Talend Real-Time Big Data Platform
Module
Talend Studio
Content
Design and Development > Designing Jobs > Hadoop distributions
Last publication date
2024-02-06

You can contextualize the Hadoop connection for a Job without using the Studio.

When you do not have a Studio at hand but need to deploy a Job in a Hadoop environment different from the Hadoop environments already defined for this Job, you can take the manual approach to add a new Hadoop connection context.

If a Job is using a contextualized Hadoop connection that has two contexts, for example Default and Dev, after being built out of the Studio, the lib folder of the built artifact (the Job zip) contains two special jars for the given Hadoop environments. The name of these jars follows a pattern: "hadoop-conf-[name_of_the_metadata_in_the_repository]_[name_of_the_context].jar".

The jar to be used at runtime is defined by the context used in the command you can read from the .bat file or the .sh file of the Job.

The following line is an example of this command, which calls the Default context:

java -Xms256M -Xmx1024M -cp .;../lib/routines.jar;../lib/antlr-runtime-3.5.2.jar;../lib/avro-1.7.6-cdh5.10.1.jar;../lib/commons-cli-1.2.jar;../lib/commons-codec-1.9.jar;../lib/commons-collections-3.2.2.jar;../lib/commons-configuration-1.6.jar;../lib/commons-lang-2.6.jar;../lib/commons-logging-1.2.jar;../lib/dom4j-1.6.1.jar;../lib/guava-12.0.1.jar;../lib/hadoop-auth-2.6.0-cdh5.10.1.jar;../lib/hadoop-common-2.6.0-cdh5.10.1.jar;../lib/hadoop-hdfs-2.6.0-cdh5.10.1.jar;../lib/htrace-core4-4.0.1-incubating.jar;../lib/httpclient-4.3.3.jar;../lib/httpcore-4.3.3.jar;../lib/jackson-core-asl-1.8.8.jar;../lib/jackson-mapper-asl-1.8.8.jar;../lib/jersey-core-1.9.jar;../lib/log4j-1.2.16.jar;../lib/log4j-1.2.17.jar;../lib/org.talend.dataquality.parser.jar;../lib/protobuf-java-2.5.0.jar;../lib/servlet-api-2.5.jar;../lib/slf4j-api-1.7.5.jar;../lib/slf4j-log4j12-1.7.5.jar;../lib/talend_file_enhanced_20070724.jar;mytestjob_0_1.jar; local_project.mytestjob_0_1.myTestJob --context=Default %*

In this example, switching from Default to Dev results in changing the Hadoop configuration which will be loaded in the Job at runtime.