Resolve the hdp.version variable issue for Spark Jobs

Procedure

Define the hdp.version parameter in your cluster.

The easiest way of doing this is to add this parameter to the yarn-site.xml configuration file.
1. In Ambari, click the Yarn service on the service list on the left, then click Configs to open the configuration page and click the Advanced tab.
2. Scroll down the page to find the Custom yarn-site list at the end of page and click Custom yarn-site to show this list.
3. Click Add property to open the [Add property] dialog box.
4. Enter hdp.version=2.6.0.3-8, the version number you found by following the procedure at the beginning of this article about how to find the hdp.version value to be used, and click Add to validate the changes. The hdp.version parameter appears in the Custom yarn-site parameter list.
5. Click Save to validate the new configuration and restart the services to implement the hdp.version parameter in the yarn-site.xml file.
In the Studio, open the Spark Job to be used and click the Run tab to open its view.
Click Spark configuration, then in the view, select the Set hdp.version check box and enter, within double quotation marks, the same version number you have entered in the cluster. In this example, it is 2.6.0.3-8.

This procedure explains only the actions to be performed to solve the HDP version issue for a Spark Job. You need properly configure the other parts of your Job before being able to run it successfully.

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!