tConvertType properties for Apache Spark Streaming

tConvertType

EnrichVersion
Cloud
6.4
task
Data Quality and Preparation > Third-party systems > Processing components (Integration)
Data Governance > Third-party systems > Processing components (Integration)
Design and Development > Third-party systems > Processing components (Integration)
EnrichPlatform
Talend Studio

These properties are used to configure tConvertType running in the Spark Streaming Job framework.

The Spark Streaming tConvertType component belongs to the family.

The component in this framework is available only if you have subscribed to Talend Real-time Big Data Platform or Talend Data Fabric.

Basic settings

Schema and Edit Schema

A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. The schema is either Built-In or stored remotely in the Repository.

Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

  • View schema: choose this option to view the schema only.

  • Change to built-in property: choose this option to change the schema to Built-in for local changes.

  • Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion. If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the [Repository Content] window.

 

Built-In: You create and store the schema locally for this component only. Related topic: see Talend Studio User Guide.

 

Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs. Related topic: see Talend Studio User Guide.

Auto Cast

This check box is selected by default. It performs an automatic java type conversion.

Manual Cast

This mode is not visible if the Auto Cast check box is selected. It allows you to precise manually the columns where a java type conversion is needed.

Die on error

Select this check box to stop the execution of the Job when an error occurs.

Set empty values to Null before converting

This check box is selected to set the empty values of String or Object type to null for the input data.

Usage

Usage rule

This component is used as an intermediate step.

This component, along with the Spark Streaming component Palette it belongs to, appears only when you are creating a Spark Streaming Job.

Note that in this documentation, unless otherwise explicitly stated, a scenario presents only Standard Jobs, that is to say traditional Talend data integration Jobs.

Spark Connection

You need to use the Spark Configuration tab in the Run view to define the connection to a given Spark cluster for the whole Job. In addition, since the Job expects its dependent jar files for execution, you must specify the directory in the file system to which these jar files are transferred so that Spark can access these files:
  • Yarn mode: when using Google Dataproc, specify a bucket in the Google Storage staging bucket field in the Spark configuration tab; when using other distributions, use a tHDFSConfiguration component to specify the directory.

  • Standalone mode: you need to choose the configuration component depending on the file system you are using, such as tHDFSConfiguration or tS3Configuration.

This connection is effective on a per-Job basis.