tDeltaLakeOutput properties for Apache Spark Batch - 7.3

Define how to save the dataset

Either Metastore, Files, or Merge.

Metastore: Stores data in table format in a metastore.

Files: Stores data in Delta format in files.

Merge: Stores data by merging it into an existing Delta table.

For more information about merging data in Delta tables, see Upsert into a table using merge in the Databricks documentation.

Define a storage configuration component

Select the configuration component to be used to provide the configuration information for the connection to the target file system such as HDFS.

If you leave this check box clear, the target file system is the local system.

The configuration component to be used must be present in the same Job. For example, if you have dropped a tHDFSConfiguration component in the Job, you can select it to write the result in a given HDFS system.

This field is available only when you select Files from the Define the source of the dataset drop-down list in the Basic settings view.

Property type

Either Built-In or Repository.

Built-In: No property data stored centrally.

Repository: Select the repository file where the properties are stored.

The properties are stored centrally under the Hadoop Cluster node of the Repository tree.

The fields that come after are pre-filled in using the fetched data.

For further information about the Hadoop Cluster node, see the Getting Started Guide.

Schema and Edit Schema

A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. When you create a Spark Job, avoid the reserved word line when naming the fields.

Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

• View schema: choose this option to view the schema only.

• Change to built-in property: choose this option to change the schema to Built-in for local changes.

• Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion. If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the Repository Content window.

Spark automatically infers data types for the columns in a PARQUET schema. In a Talend Job for Apache Spark, the Date type is inferred and stored as int96.

Built-In: You create and store the schema locally for this component only.

Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs.

Folder/File

Browse to, or enter the path pointing to the data to be used in the file system.

The button for browsing does not work with the Spark Local mode; if you are using the other Spark Yarn modes that the Studio supports with your distribution, ensure that you have correctly configured the connection in a configuration component in the same Job, such as tHDFSConfiguration. Use the configuration component depending on the filesystem to be used.

This field is available only when you select Files from the Select how to save the dataset drop-down list in the Basic settings view.

Action

Delta Lake systematically creates slight differences between the upload time of a file and the metadata timestamp of this file. Bear in mind these differences when you need to filter data.

This option is available only when you select Files or Metastore from the Select how to save the dataset drop-down list in the Basic settings view.

Source type

The source type of the input data, which can be either Dataset or SQL.

This field is available only when you select Merge from the Select how to save the dataset drop-down list in the Basic settings view.

Dataset: select this value if the source data is a dataset that comes from a flow of data.

SQL: select this value if you want to use an SQL statement to retrieve the source data from a Delta table.

Database

Enter, in double quotation marks, the name of the Delta Lake database to be used.

This field is available only when you select Metastore or Merge from the Select how to save the dataset drop-down list in the Basic settings view.

Target table

Enter, in double quotation marks, the name of the table to be used.

This field is available only when you select Metastore or Merge from the Select how to save the dataset drop-down list in the Basic settings view.

External path

Enter, in double quotation marks, the path pointing to a filesystem, different from the DBFS, to store the data. It can either be a ADLS Gen2 filesystem or a S3 filesystem.

This field is available only when you select Metastore from the Select how to save the dataset drop-down list in the Basic settings view.

Optimize

Enter the SQL statement to optimize the layout of the data.

For more information about optimizing Delta Lake data, see Optimize (Delta Lake on Databricks) in the Databricks documentation.

This field is available only when you select Metastore from the Select how to save the dataset drop-down list in the Basic settings view.

Merge on

Indicate the input and output columns on which to apply the merge operation. For each pair of input and output columns, you must specify a function. The resulting clause will condition which merge actions to perform. These merge actions are defined in the When matched and When not matched options.

For example, suppose an Airline company works with data related to flight events. They regularly handle changes in flight departure dates and need to update their current data with the new dates. To merge this new data into existing events based on the ID of the flight, they might indicate flightId as input and output columns and the== function. Then, they can define the merge actions to perform when the result of this condition returns true. In this case, the condition is true when the values of flightId in the input and output columns are equal.

This field is available only when you select Merge from the Select how to save the dataset drop-down list in the Basic settings view.

When matched

Select this check box to define merge actions to perform when the functions defined in the Merge on table return true. You can define one merge action for each of the two When matched options. If you define two When matched options, the first one must have a clause condition and the two options are evaluated in the order in which they are defined.

In the condition field, you can enter a condition statement to further refine the data on which to apply the merge action. If you specify a condition, the merge action is performed on a given row only if the condition is true for this row. Using conditions can speed up the merge. The condition must follow the format targetTable.column = this.column, where targeTable is the name of the target table. With Dataset source type, this is the name of the connection. With SQL source type, this must be replaced with the Table alias field.

For example, suppose that an airline data table contains the number of passengers of a flight and a boolean that indicates if a flight has become more crowded. The following condition limits the scope of the merge action to the flights that have become more crowded based on the number of passengers: flightEvents.nbOfPeople < this.nbOfPeople. The merge action could then be to update the value of isFlightMoreCrowded to true with the UPDATE SET* action.

This field is available only when you select Merge from the Select how to save the dataset drop-down list in the Basic settings view.

When not matched

Select this check box to define merge actions to perform when the functions defined in the Merge on table return false.

In the condition field, you can enter a condition statement to further refine the data on which to apply the merge action. If you specify a condition, the merge action is performed on a given row only if the condition is true for this row. Using conditions can speed up the merge. The condition must follow the format targetTable.column = this.column, where targeTable is the name of the target table. With Dataset source type, this is the name of the connection. With SQL source type, this must be replaced with the Table alias field.

For example, suppose that an airline data table is partitioned by date and the merge action has the following condition: flightEvents.date = current_date(). The merge action then applies when the functions defined in the merge on table return false and only on the data that corresponds to flights that occurred on the current date.

This field is available only when you select Merge from the Select how to save the dataset drop-down list in the Basic settings view.

If you leave this check box and the Overwrite Schema check box clear, only the columns of the existing data are used.

The schema of your datasets often evolves through time. Select this check box to use the schemas of the incoming data to overwrite the schemas of the existing data.

If you leave this check box and the Merge Schema check box clear, only the columns of the existing data are used.

Usage rule

This component is used as an end component and requires an input link.

Delta Lake systematically creates slight differences between the upload time of a file and the metadata timestamp of this file. Bear in mind these differences when you need to filter data.

This Delta Lake layer is built on top of your Data Lake system, thus to be connected as part of your Data Lake system using the configuration component corresponding to your Data Lake system, for example, tAzureFSCofiguration.

Spark Connection

This connection is effective on a per-Job basis.