Finding the properties to be set - 6.3

Enabling the HDFS High Availability feature in the Studio

EnrichVersion
6.3
task
Data Governance > Third-party systems > File components (Integration) > HDFS components
Data Quality and Preparation > Third-party systems > File components (Integration) > HDFS components
Design and Development > Third-party systems > File components (Integration) > HDFS components
EnrichPlatform
Talend Studio
You need to find the properties from this hdfs-site.xml file of the Hadoop cluster in order to replicate them in the Studio.

Procedure

  1. You need to find the dfs.nameservices property.

    For example, this property might read:

    <property>
       <name>dfs.nameservices</name>
       <value>nameservice1</value>
    </property>

    The value of this property is fundamental because it defines the name of the new nameservice and is used to define the other properties required by the HA feature. Therefore, you need to use this value, nameservice1 in this example, to find the other properties to be replicated.

  2. Use the nameservice1 value to find the following properties. Note this value nameservice1 is taken for demonstration purposes only.
    <property>
      <name>dfs.client.failover.proxy.provider.nameservice1</name>
      <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>
    <property>
      <name>dfs.ha.namenodes.nameservice1</name>
      <value>namenode90,namenode96</value>
    </property>

    The value, namenode90,namenode96 in this example, of the dfs.ha.namenodes.nameservice1 property defines the IDs of the NameNodes in this nameservice. The IDs are separated by coma (,).

  3. Use the NameNode IDs defined in the dfs.ha.namenodes.nameservice1 property to find the following properties:
    <property>
      <name>dfs.namenode.rpc-address.nameservice1.namenode90</name>
      <value>cdh4ha:8020</value>
    </property>
    <property>
      <name>dfs.namenode.rpc-address.nameservice1.namenode96</name>
      <value>cdh4ha2:8020</value>
    </property>

    They define the RPC address of each NameNode in this new nameservice.

Results

All the properties to be replicated have been found and you need to set them in the Hadoop properties table provided in the Studio.