Big Data
Big Data Platform
Data Fabric
Real-Time Big Data Platform
Cloud Hadoop distributions
Talend supports the following cloud platforms for Big Data. Click your cloud platform to see the Big data support information.
- Amazon EMR
- Databricks on AWS
- Databricks on Azure
- Databricks on Google Cloud Platform (GCP)
- Google Dataproc
- Microsoft HDInsight
Old versions of the supported Big Data platforms are being retired by their vendors. Talend ceases to support a version once this version reaches its date of end of support set by its vendor.
Talend and its community provide you with the convenience to keep using a version its vendor ceases to support in Talend products. For this reason, this version can be still listed in the following tables and available in the products but Talend stops providing support for this version.
Talend supports the minor versions of the platform versions listed in the following tables.
| Amazon EMR version | Supported frameworks | Supported Hadoop elements in Spark batch | Supported Hadoop elements in Spark streaming | Supported Hadoop elements in Standard |
|---|---|---|---|---|
| v5.29.0 (Hadoop 2.8.5) |
Standard Spark v2.4 |
HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive Sqoop |
| v6.2.0 (Hadoop 3.2.1) |
Standard Spark v3.0 |
HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive Sqoop |
| v6.3.x (Hadoop 3.2.3) | Spark Universal v3.1.x |
HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive |
- |
| v6.4.x (Hadoop 3.2.3) | Spark Universal v3.1.x |
HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive |
- |
| v6.5.x (Hadoop 3.2.3) | Spark Universal v3.1.x |
HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive |
- |
| v6.6.0 (Hadoop 3.2.3) | Spark Universal v3.2.x |
HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive |
- |
| v6.7.0 (Hadoop 3.2.3) | Spark Universal v3.2.x |
HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive |
- |
| v6.8.0 (Hadoop 3.3.1) | Spark Universal v3.3.x |
HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive |
- |
| v6.9.0 (Hadoop 3.3.1) | Spark Universal v3.3.x |
HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive |
- |
| v6.10.0 (Hadoop 3.3.1) | Spark Universal v3.3.x |
HBase HDFS HCatalog Hive |
HBase HDFS HCatalog Hive |
- |
The supported Amazon EMR versions for the tAmazonEMRManage component is 5.29.0.
| Databricks on AWS version | Supported frameworks | Supported elements in Spark batch | Supported elements in Spark streaming | Supported elements in Standard |
|---|---|---|---|---|
| 5.5 LTS | Standard Spark v2.4 |
Hive S3 DynamoDB Snowflake MongoDB TDM components tDataprepRun |
Hive S3 DynamoDB Kinesis Snowflake MongoDB TDM components as beta tDataprepRun |
DBFS |
| 6.4 | Standard Spark v2.4.5 |
Azure Blob Storage ADLS Gen2 Snowflake DeltaLake |
Azure Blob Storage ADLS Gen2 |
DBFS |
| 7.3 LTS | Standard Spark v3.0.1 |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Kinesis Azure Blob Storage |
DBFS |
| 8.x | Spark Universal v3.1.x | Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Kinesis Azure Blob Storage |
- |
| 9.x | Spark Universal v3.1.x | Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Kinesis Azure Blob Storage |
- |
| 10.x |
Spark Universal v3.2.x |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Kinesis Azure Blob Storage |
- |
| 11.x |
Spark Universal v3.3.x |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Kinesis Azure Blob Storage |
- |
| 12.x |
Spark Universal v3.3.x |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Kinesis Azure Blob Storage |
- |
| Databricks on Azure version | Supported frameworks | Supported elements in Spark batch | Supported elements in Spark streaming | Supported elements in Standard |
|---|---|---|---|---|
| 5.5 LTS | Standard Spark v2.4 |
Hive Azure Blob Storage ADLS Gen1 ADLS Gen2 Snowflake DeltaLake MongoDB TDM components tDataprepRun |
Hive Azure Blob Storage ADLS Gen1 ADLS Gen2 Snowflake DeltaLake MongoDB TDM components tDataprepRun |
DBFS |
| 6.4 | Standard Spark v2.4.5 |
Azure Blob Storage ADLS Gen2 Snowflake DeltaLake |
Azure Blob Storage ADLS Gen2 |
DBFS |
| 7.3 LTS | Standard Spark v3.0.1 |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
DBFS |
| 8.x | Spark Universal v3.1.x | Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
- |
| 9.x | Spark Universal v3.1.x | Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
- |
| 10.x |
Spark Universal v3.2.x |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
- |
| 11.x |
Spark Universal v3.3.x |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
- |
| 12.x |
Spark Universal v3.3.x |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Kinesis Azure Blob Storage |
- |
| Databricks on GCP version | Supported frameworks | Supported elements in Spark batch | ⓘ Available in: Data Fabric Real-Time Big Data Platform |
Supported elements in Standard |
|---|---|---|---|---|
| 8.x | Spark Universal v3.1.x | Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
- |
| 9.x | Spark Universal v3.1.x | Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
- |
| 10.x |
Spark Universal v3.2.x |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
- |
| 11.x |
Spark Universal v3.3.x |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen1 ADLS Gen2 Azure Blob Storage |
- |
| 12.x |
Spark Universal v3.3.x |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Azure Blob Storage |
Hive S3 DynamoDB Snowflake MongoDB DeltaLake ADLS Gen2 Kinesis Azure Blob Storage |
- |
| Google Dataproc version | Supported frameworks | Supported elements in Spark batch | ⓘ Available in: Data Fabric Real-Time Big Data Platform |
Supported elements in Standard |
|---|---|---|---|---|
| 2.0.x |
Standard Spark Universal v3.1.x |
Hive BigQuery Google Storage Avro Delimited Parquet Positional XML JSON |
Hive BigQuery Google Storage Avro Delimited Parquet Positional XML JSON Google PubSub |
Hive |
| Microsoft HDInsight version | Supported frameworks | Supported elements in Spark batch | Supported elements in Spark streaming | Supported elements in Standard |
|---|---|---|---|---|
| 4.0 | Spark v2.3 (deprecated) and v2.4 | ADLS Gen2 Azure Blob Storage Hive |
ADLS Gen2 Azure Blob Storage Hive |
ADLS Gen2 Azure Blob Storage Hive |