Available in...
Big Data
Big Data Platform
Cloud Big Data
Cloud Big Data Platform
Cloud Data Fabric
Data Fabric
Real-Time Big Data Platform
Spark Universal is a mechanism that allows Talend Studio to be
compatible with every big data distribution for a given Spark version. You choose a
Spark version and upload a Hadoop configuration JAR file that contains all the necessary
information to connect to your cluster.
Spark Universal modes and environments support
Talend Studio supports the following modes and environments, depending on the Spark
versions:
| Spark 2.4.x | Spark 3.0.x | Spark 3.1.x | Spark 3.2.x | Spark 3.3.x | Spark 3.4.x | |
| Local mode |  |
|
|
|
|
|
| Standalone |  |
|
|
|
|
|
| Yarn cluster mode | |
|
|
|
|
|
| Databricks | |
|
|
|
|
|
| Dataproc | |
|
|
|
|
|
| Cloudera Data Engineering | |
|
|
|
|
|
| Kubernetes | |
|
|
|
|
|
| Spark-submit scripts | |
|
|
|
|
|
| Synapse | |
|
|
|
|
|
| HDInsight | |
|
|
|
|
|
| EMR Serverless | |
|
|
|
|
|
Note:
- Azure Synapse Analytics with Spark Universal 3.2.x and 3.3.x is only supported in Spark Batch Jobs.
- Spark-submit script with Spark Universal 3.3.x is only supported in Spark Batch Jobs.
Spark Universal distributions support
Talend Studio supports the following distributions in Yarn cluster mode,
depending on the Spark versions:
For example, if you want to connect to an Amazon EMR 6.2 cluster, you
need to select Spark 3.0.x version and then upload the Hadoop configuration JAR file
that contains all the
| Spark 2.4.x |
|
| Spark 3.0.x |
|
| Spark 3.1.x |
|
| Spark 3.2.x |
|
| Spark 3.3.x |
|
*-site.xml files related to the cluster.This list of distribution is not exhaustive, you can use Yarn cluster with any other distribution if the Spark version matches, but keep in mind that they have not been officially tested by Talend and thus not guaranteed to work.