Find the hdp.version value to be used - 6.4

The HDP version variable issue in MapReduce Jobs and Spark Jobs

author
Talend Documentation Team
EnrichVersion
6.4
EnrichProdName
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Real-Time Big Data Platform
task
Design and Development > Designing Jobs > Hadoop distributions > Hortonworks
Design and Development > Designing Jobs > Job Frameworks > MapReduce
Design and Development > Designing Jobs > Job Frameworks > Spark Batch
Design and Development > Designing Jobs > Job Frameworks > Spark Streaming
EnrichPlatform
Talend Studio

Procedure

Identify the version of your Hortonwork cluster.
  • You may directly ask the administrator of your cluster about the correct version to use.

  • You can also check the /usr/hdp/ directory of each machine in your cluster. This folder usually contains several versions and a symbolic link reading current points to the latest value you should use. For example:
    [root@sandbox /]# ls  -lth /usr/hdp/
    total 16K
    drwxr-xr-x 50 root root 4.0K Jun  5 07:59 2.6.0.3-8
    drwxr-xr-x 32 root root 4.0K May  5 13:19 2.5.0.0-1245
    drwxr-xr-x  3 root root 4.0K May  5 13:18 share
    drwxr-xr-x  2 root root 4.0K May  5 12:48 current -> 2.6.0.3-8

    In this example, the version to be used is hdp.version=2.6.0.3-8.

Results

Then you need to configure the Talend Studio Job to be used.
  1. For Spark Jobs, see Resolve the hdp.version variable issue for Spark Jobs.

  2. For MapReduce Jobs, see Resolve the hdp.version variable issue for MapReduce Jobs.