Cloning an existing cluster is the recommended way to create an interactive Databricks cluster that is compatible with Talend Cloud Pipeline Designer.
- Log in to Talend Cloud Pipeline Designer and execute a pipeline using a Databricks run profile that was configured with the New Cluster option in Talend Cloud Management Console.Note: This option creates a cluster then runs the pipeline and stops the cluster. To avoid recreating a supported cluster from scratch, this cluster will be cloned.
- Log in to your Databricks account and select the last terminated cluster in the Automated Cluster list.
- Click the Clone icon in the Actions column to open the clone wizard.
- Change the cluster configuration according to your needs but make sure you keep all the advanced configuration as they are.
- In the Tags tab of the advanced configuration section, add the following tag to indicate that the cluster is created for Talend Cloud Pipeline Designer:Key: TALEND_TPD_CLUSTER_TYPE
- Click Create cluster to finalize the creation operation.