tKafkaOutput Standard properties - Cloud - 8.0

Kafka

Version
Cloud
8.0
Language
English
Product
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Open Studio for Big Data
Talend Real-Time Big Data Platform
Module
Talend Studio
Content
Data Governance > Third-party systems > Messaging components (Integration) > Kafka components
Data Quality and Preparation > Third-party systems > Messaging components (Integration) > Kafka components
Design and Development > Third-party systems > Messaging components (Integration) > Kafka components

These properties are used to configure tKafkaOutput running in the Standard Job framework.

The Standard tKafkaOutput component belongs to the Internet family.

The component in this framework is available in all Talend products with Big Data and in Talend Data Fabric.

Basic settings

Input type

Select the type of messages to be read by Kafka from the drop-down list:
  • byte[]: the component receives messages serialized into byte arrays.
  • ProducerRecord: the component receives messages serialized as key/value pair. The message key and the message value can be serialized as Avro.

This property is available since Kafka 2.2.1.

Schema and Edit schema

A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. When you create a Spark Job, avoid the reserved word line when naming the fields.

Note that the schema of this component is read-only. It stores the messages to be published.

Use an existing connection

Select this check box and in the Component List drop-down list, select the desired connection component to reuse the connection details you already defined.

Version

Select the version of the Kafka cluster to be used.

Broker list

Enter the addresses of the broker nodes of the Kafka cluster to be used.

The form of this address should be hostname:port. This information is the name and the port of the hosting node in this Kafka cluster.

If you need to specify several addresses, separate them using a comma (,).

Topic name

Enter the name of the topic you want to publish messages to. This topic must already exist.

This property is only available when you select Byte[] from the Input type drop-down list.

Compress the data

Select the Compress the data check box to compress the output data.

Use SSL/TLS

Select this check box to enable the SSL or TLS encrypted connection.

This check box is available since Kafka 0.9.0.1.

Set keystore

Select this check box to enable the SSL or TLS encrypted connection via a tSetKeystore component.

Then you need to use the tSetKeystore component in the same Job to specify the encryption information.

This check box is available when you select the Use SSL/TLS check box.

Note: This option is available when you have installed the 8.0.1-R2022-05 Studio Monthly update or a later one delivered by Talend. For more information, check with your administrator.

Use Kerberos authentication

If the Kafka cluster to be used is secured with Kerberos, select this check box to display the related parameters to be defined:

  • JAAS configuration path: enter the path, or browse to the JAAS configuration file to be used by the Job to authenticate as a client to Kafka.

    This JAAS file describes how the clients, the Kafka-related Jobs in terms of Talend , can connect to the Kafka broker nodes, using either the kinit mode or the keytab mode. It must be stored in the machine where these Jobs are executed.

    Talend , Kerberos or Kafka does not provide this JAAS file. You need to create it by following the explanation in Configuring Kafka client depending on the security strategy of your organization.

  • Kafka brokers principal name: enter the primary part of the Kerberos principal you defined for the brokers when you were creating the broker cluster. For example, in this principal kafka/kafka1.hostname.com@EXAMPLE.COM, the primary part to be used to fill in this field is kafka.

  • Set kinit command path: Kerberos uses a default path to its kinit executable. If you have changed this path, select this check box and enter the custom access path.

    If you leave this check box clear, the default path is used.

  • Set Kerberos configuration path: Kerberos uses a default path to its configuration file, the krb5.conf file (or krb5.ini in Windows) for Kerberos 5 for example. If you have changed this path, select this check box and enter the custom access path to the Kerberos configuration file.

    If you leave this check box clear, a given strategy is applied by Kerberos to attempt to find the configuration information it requires. For details about this strategy, see the Locating the krb5.conf Configuration File section in Kerberos requirements.

For further information about how a Kafka cluster is secured with Kerberos, see Authenticating using SASL.

This check box is available since Kafka 0.9.0.1.

Advanced settings

Kafka properties

Add the Kafka new producer properties you need to customize to this table.

For more information about the new producer properties you can define in this table, see the section describing the new producer configuration from the official Kafka documentation.

Set Headers

Select this check box to add headers to messages to be sent.

This feature is available to Kafka 1.1.0 onwards.

Use schema registry

Select this check box to use Confluent Schema Registry and to display the related parameters to be defined:
  • URL: enter the Schema Registry instance URL.
  • Basic authentication: select this check box and enter your credentials in the Username and Password.
  • Set schema registry keystore: select this check box to enable the SSL or TLS encrypted connection. Then you need to use the tSetKeystore component in the same Job to specify the encryption information. This check box is not available when you already set a tSetKeystore in the Basic settings view of the component because Kafka SSL configuration is reused for schema registry.
  • Key deserializer and Value deserializer: select the schema format to use for the key and value from the drop-down list. The default Custom serializer is org.apache.kafka.common.serialization.ByteArraySerializer.

For more information about Schema Registry, see the Confluent documentation.

This option is only available when you select ProducerRecord from the Input type drop-down list in the Basic settings view.

Note: This option is available when you have installed the 8.0.1-R2022-01 Studio Monthly update or a later one delivered by Talend. For more information, check with your administrator.

tStatCatcher Statistics

Select this check box to gather the processing metadata at the Job level as well as at each component level.

Global Variables

ERROR_MESSAGE

The error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is selected.

NB_LINE

The number of rows processed. This is an After variable and it returns an integer.

NB_ERRORS

The number of rows processed with errors. This is an After variable and it returns an integer.

NB_SUCCESS

The number of rows successfully processed. This is an After variable and it returns an integer.

Usage

Usage rule

This component is an end component. It requires a tJavaRow or tJava component to transform the incoming data into serialized byte arrays.

The following sample shows how to construct a statement to perform this transformation:

output_row.serializedValue = input_row.users.getBytes();

In this code, the output_row variable represents the schema of the data to be output to tKafkaOutput and output_row.serializedValue the single read-only column of that schema; the input_row variable represents the schema of the incoming data and input_row.users the input column called users to be transformed to byte arrays by the getBytes() method.