Using the Databricks cluster clone wizard (recommended) - Cloud

Talend Cloud Management Console for Pipelines User Guide

author
Talend Documentation Team
EnrichVersion
Cloud
EnrichProdName
Talend Cloud
task
Administration and Monitoring > Managing projects
Administration and Monitoring > Managing users
Deployment > Deploying > Executing Tasks
Deployment > Scheduling > Scheduling Tasks
EnrichPlatform
Talend Management Console

Cloning an existing cluster is the recommended way to create an interactive Databricks cluster that is compatible with Talend Cloud Pipeline Designer.

Procedure

  1. Log in to Talend Cloud Pipeline Designer and execute a pipeline using a Databricks run profile that was configured with the New Cluster option in Talend Cloud Management Console.
    Note: This option creates a cluster then runs the pipeline and stops the cluster. To avoid recreating a supported cluster from scratch, this cluster will be cloned.
  2. Log in to your Databricks account and select the last terminated cluster in the Automated Cluster list.
  3. Click the Clone icon in the Actions column to open the clone wizard.
    1. Change the cluster configuration according to your needs but make sure you keep all the advanced configuration as they are.
    2. In the Tags tab of the advanced configuration section, add the following tag to indicate that the cluster is created for Talend Cloud Pipeline Designer:
      Key: TALEND_TPD_CLUSTER_TYPE

      Value: TPD_COMPATIBLE_INTERACTIVE_CLUSTER_1.0

    3. Click Create cluster to finalize the creation operation.