Troubleshooting: Resolving Databricks runtime incompatibilities - Cloud

Talend Cloud Management Console for Pipelines User Guide

author
Talend Documentation Team
EnrichVersion
Cloud
EnrichProdName
Talend Cloud
task
Administration and Monitoring > Managing projects
Administration and Monitoring > Managing users
Deployment > Deploying > Executing Tasks
Deployment > Scheduling > Scheduling Tasks
EnrichPlatform
Talend Management Console
When upgrading to a newer Remote Engine Gen2 that uses a Databricks cluster, you might have compatibility issues during pipeline executions and you might see this error message in the logs:
Incompatible runtime found : Cluster runtime version =/= Engine runtime version

Cause

This error is due to the fact that Talend Cloud Pipeline Designer requires a Databricks cluster with compatible libraries in order to submit pipelines from a Remote Engine Gen2.

Troubleshooting

Procedure

  1. In order to resolve this issue, log in to your Databricks account and go to the Clusters page.
  2. Select your existing cluster and go to the Libraries tab to uninstall the old libraries.

    By removing these old dependencies the compatibility issue should be solved and you should be able to execute pipelines on your new engine with the same Databricks cluster.