General parameters

The following tables list parameters that are configurable in the distribution properties, unless specified. They can be added in the connection string or the separate configuration file of the Kafka connector.

For more details, see https://docs.confluent.io/platform/current/clients/librdkafka/html/md_CONFIGURATION.html.

The tables are listed in big categories:

To ensure the delivery of data

Parameter	Description
bootstrap.servers (parameter in the Manager: broker)	Indicates the number of acknowledgements the leader broker must receive from ISR brokers before responding to the request: By default, it is set to `all` (the process waits until all in-sync replicas have received the data).
Acks (parameter in the Manager: topic.acks or topic.request.required.acks)	Indicates the number of acknowledgements the leader broker must receive from ISR brokers before responding to the request: By default, it is set to `all` (the process waits until all in-sync replicas have received the data).

Parameter

Description

bootstrap.servers (parameter in the Manager: broker)

Indicates the number of acknowledgements the leader broker must receive from ISR brokers before responding to the request:

By default, it is set to all (the process waits until all in-sync replicas have received the data).

Acks (parameter in the Manager: topic.acks or topic.request.required.acks)

Indicates the number of acknowledgements the leader broker must receive from ISR brokers before responding to the request:

By default, it is set to all (the process waits until all in-sync replicas have received the data).

In case of connection issues to the Kafka cluster

Parameter	Description
Retries (parameter in the Manager: retries or message.send.max.retries)	Indicates the number of retries to send a failing message. Information noteNote: Retrying may cause reordering unless enable.idempotence is set to true. By default, it is set to `1000000`.
retry.backoff.ms (parameter in the Manager: retry.backoff.ms)	Indicates the initial time to wait before reconnecting to a broker after the connection has been closed. By default, it is set to `100`.

Parameter

Description

Retries (parameter in the Manager: retries or message.send.max.retries)

Indicates the number of retries to send a failing message.

Note: Retrying may cause reordering unless enable.idempotence is set to true.

By default, it is set to 1000000.

retry.backoff.ms (parameter in the Manager: retry.backoff.ms)

Indicates the initial time to wait before reconnecting to a broker after the connection has been closed.

By default, it is set to 100.

To preserve message order

Parameter	Description
max.in.flight.requests.per.connection (parameter in the Manager: max.in.flight.requests.per.connection or max.in.flight)	Indicates the maximum number of in-flight requests per broker connection. By default, it is set to `1000000`.

For performance running

Parameter	Description
batch.size	Indicates the maximum size (in bytes) of all messages batched in one MessageSet. Information noteNote: This parameter is not configurable in the distribution properties but other parameters are available and may be equivalent: queue.buffering.max.messages: Maximum number of messages allowed on the producer queue. This queue is shared by all topics and partitions. queue.buffering.max.kbytes: Maximum total message size sum allowed on the producer queue. This queue is shared by all topics and partitions. This property has higher priority than queue.buffering.max.messages. message.max.bytes: Maximum Kafka protocol request message size. message.copy.max.bytes: Maximum size for messages to be copied to the buffer. Messages larger than this will be passed by reference (zero-copy) at the expense of larger iovecs.
linger.ms (parameter in the Manager: linger.ms or queue.buffering.max.ms)	Indicates the alias for queue.buffering.max.ms: delay in milliseconds to wait for messages in the producer queue to accumulate before constructing message batches (MessageSets) to transmit to brokers. By default, it is set to `1000000`.
buffer.memory	Indicates the total bytes of memory the producer can use to buffer records waiting to be sent to the server. Information noteNote: This parameter is not configurable in the distribution properties but other parameters are available and may be equivalent: See batch.size.

Transport Layer Authentication and Security

Parameter	Description
security.protocol	Protocol used to communicate with brokers. Possible values: `sasl`_`ssl` , `ssl`, `sasl_plaintext` Example: security.protocol=sasl_ssl
sasl.username	SASL username to be used with PLAIN and SASL-SCRAM mechanisms.
sasl.password	SASL password to be used with the PLAIN and SASL-SCRAM mechanisms.
sasl.mechanisms	SASL mechanism to be used for authentication. Possible values: `GSSAPI`, `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`, `OAUTHBEARER`. Example: sasl.mechanisms=SCRAM-SHA-256
ssl.ca.location	Path to the CA certificate to verify the broker key. Example: ssl.ca.location=%P%ca-cert.cert %P% will be replaced by the DestinationPath\Data path.

Talend Change Data Capture parameters

Parameter	Description
config \| cfg \| cf	Path to the configuration file. The parameters can be put in a configuration file. Example: config=%P%producer.config %P% corresponds to Talend Change Data Capture data path: DestinationPath\Data.
tkey \| tk	Type of serialization of the key. The key is used internally by the Kafka connector of Talend Change Data Capture. For more details, see Connection string: details on Kafka key setup.
codepage \| cp	Conversion file used by the target when the source is an AS/400 IBMi machine.
tr	Allows to get metrics on topic for each partition in a log file. This file is stored in the folder .\DestinationPath\Data\Kafka\ and named REE_SSSMMM_statKfk.log. Metrics are saved only when the distribution is stopped.
mode \| md	To enable or disable template script. `0`= Template script disabled `1` = Template script enabled `0` = Default

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!

Leave your feedback here