Add the Azure specific properties to the Spark configuration of your Databricks cluster so that your cluster can access Azure Storage.
You need to do this only when you want your Talend Jobs for Apache Spark to use Azure Blob Storage or Azure Data Lake Storage with Databricks.
Before you begin
- Ensure that your Spark cluster in Databricks has been properly created and is running and its version is 3.5 LTS. For further information, see Create Databricks workspace from Azure documentation.
- You have an Azure account.
- The Azure Blob Storage or Azure Data Lake Storage service to be used has been properly created and you have the appropriate permissions to access it. For further information about Azure Storage, see Azure Storage tutorials from Azure documentation.
On the Configuration tab of your Databricks cluster
page, scroll down to the Spark tab at the bottom of the
- Click Edit to make the fields on this page editable.
In this Spark tab, enter the Spark properties regarding
the credentials to be used to access your Azure Storage system.
Option Description Azure Blob Storage When you need to use Azure Blob Storage with Azure Databricks, add the following Spark property:
Azure Data Lake Storage When you need to use Azure Data Lake Storage with Databricks, add the following Spark properties, each per line:
spark.hadoop.dfs.adls.oauth2.access.token.provider.type ClientCredential spark.hadoop.dfs.adls.oauth2.client.id <your_app_id> spark.hadoop.dfs.adls.oauth2.credential <your_authentication_key> spark.hadoop.dfs.adls.oauth2.refresh.url https://login.microsoftonline.com/<your_app_TENANT-ID>/oauth2/token
If you need to run Spark Streaming Jobs with Databricks, in the same
Spark tab, add the following property to define a
default Spark serializer. If you do not plan to run Spark Streaming Jobs, you
can ignore this step.
- Restart your Spark cluster.
- In the Spark UI tab of your Databricks cluster page, click Environment to display the list of properties and verify that each of the properties you added in the previous steps is present on that list.