Big Data Platform
Cloud Big Data
Cloud Big Data Platform
Cloud Data Fabric
Real-Time Big Data Platform
Spark Universal is a mechanism that allows Talend Studio to be compatible with every big data distribution for a given Spark version. You choose a Spark version and upload a Hadoop configuration JAR file that contains all the necessary information to connect to your cluster.
Spark Universal modes and environments support
Talend Studio supports the following modes and environments, depending on the Spark versions:
|Spark 2.4.x||Spark 3.0.x||Spark 3.1.x||Spark 3.2.x||Spark 3.3.x|
|Yarn cluster mode|
|Cloudera Data Engineering|
Spark Universal distributions support
Talend Studio supports the following distributions in Yarn cluster mode, depending on the Spark versions:
For example, if you want to connect to an Amazon EMR 6.2 cluster, you
need to select Spark 3.0.x version and then upload the Hadoop configuration JAR file
that contains all the
*-site.xmlfiles related to the cluster.
This list of distribution is not exhaustive, you can use Yarn cluster with any other distribution if the Spark version matches, but keep in mind that they have not been officially tested by Talend and thus not guaranteed to work.