General parameters - 7.3

Talend Change Data Capture User Guide

Version
7.3
Language
English
Product
Talend Change Data Capture
Module
Talend Change Data Capture
Content
Data Governance > Third-party systems > Database components (Integration) > Change Data Capture
Data Quality and Preparation > Third-party systems > Database components (Integration) > Change Data Capture
Design and Development > Third-party systems > Database components (Integration) > Change Data Capture
Last publication date
2023-12-22

The following tables list parameters that are configurable in the distribution properties, unless specified. They can be added in the connection string or the separate configuration file of the Kafka connector.

For more details, see https://docs.confluent.io/platform/current/clients/librdkafka/html/md_CONFIGURATION.html.

The tables are listed in big categories:

To ensure the delivery of data

Parameter Description

bootstrap.servers (parameter in the Manager: broker)

Indicates the number of acknowledgements the leader broker must receive from ISR brokers before responding to the request:

By default, it is set to all (the process waits until all in-sync replicas have received the data).

Acks (parameter in the Manager: topic.acks or topic.request.required.acks)

Indicates the number of acknowledgements the leader broker must receive from ISR brokers before responding to the request:

By default, it is set to all (the process waits until all in-sync replicas have received the data).

In case of connection issues to the Kafka cluster

Parameter Description

Retries (parameter in the Manager: retries or message.send.max.retries)

Indicates the number of retries to send a failing message.

Note: Retrying may cause reordering unless enable.idempotence is set to true.

By default, it is set to 1000000.

retry.backoff.ms (parameter in the Manager: retry.backoff.ms)

Indicates the initial time to wait before reconnecting to a broker after the connection has been closed.

By default, it is set to 100.

To preserve message order

Parameter Description
max.in.flight.requests.per.connection (parameter in the Manager: max.in.flight.requests.per.connection or max.in.flight) Indicates the maximum number of in-flight requests per broker connection.

By default, it is set to 1000000.

For performance running

Parameter Description

batch.size

Indicates the maximum size (in bytes) of all messages batched in one MessageSet.
Note: This parameter is not configurable in the distribution properties but other parameters are available and may be equivalent:
  • queue.buffering.max.messages: Maximum number of messages allowed on the producer queue. This queue is shared by all topics and partitions.
  • queue.buffering.max.kbytes: Maximum total message size sum allowed on the producer queue. This queue is shared by all topics and partitions. This property has higher priority than queue.buffering.max.messages.
  • message.max.bytes: Maximum Kafka protocol request message size.
  • message.copy.max.bytes: Maximum size for messages to be copied to the buffer. Messages larger than this will be passed by reference (zero-copy) at the expense of larger iovecs.

linger.ms (parameter in the Manager: linger.ms or queue.buffering.max.ms)

Indicates the alias for queue.buffering.max.ms: delay in milliseconds to wait for messages in the producer queue to accumulate before constructing message batches (MessageSets) to transmit to brokers.

By default, it is set to 1000000.
buffer.memory Indicates the total bytes of memory the producer can use to buffer records waiting to be sent to the server.
Note: This parameter is not configurable in the distribution properties but other parameters are available and may be equivalent: See batch.size.

Transport Layer Authentication and Security

Parameter Description

security.protocol

Protocol used to communicate with brokers.

Possible values: sasl_ssl , ssl, sasl_plaintext

Example: security.protocol=sasl_ssl

sasl.username SASL username to be used with PLAIN and SASL-SCRAM mechanisms.
sasl.password SASL password to be used with the PLAIN and SASL-SCRAM mechanisms.
sasl.mechanisms SASL mechanism to be used for authentication.

Possible values: GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER.

Example: sasl.mechanisms=SCRAM-SHA-256

ssl.ca.location Path to the CA certificate to verify the broker key.

Example: ssl.ca.location=%P%ca-cert.cert %P% will be replaced by the DestinationPath\Data path.

Talend Change Data Capture parameters

Parameter Description

config | cfg | cf

Path to the configuration file.

The parameters can be put in a configuration file.

Example: config=%P%producer.config %P% corresponds to Talend Change Data Capture data path: DestinationPath\Data.

tkey | tk Type of serialization of the key. The key is used internally by the Kafka connector of Talend Change Data Capture.

For more details, see Connection string: details on Kafka key setup.

codepage | cp Conversion file used by the target when the source is an AS/400 IBMi machine.
tr Allows to get metrics on topic for each partition in a log file. This file is stored in the folder .\DestinationPath\Data\Kafka\ and named REE_SSSMMM_statKfk.log.

Metrics are saved only when the distribution is stopped.

mode | md To enable or disable template script.

0= Template script disabled

1 = Template script enabled

0 = Default