Configuring Amazon Kinesis connector - 7.3

Talend Change Data Capture User Guide

Version
7.3
Language
English
Product
Talend Change Data Capture
Module
Talend Change Data Capture
Content
Data Governance > Third-party systems > Database components (Integration) > Change Data Capture
Data Quality and Preparation > Third-party systems > Database components (Integration) > Change Data Capture
Design and Development > Third-party systems > Database components (Integration) > Change Data Capture
Last publication date
2023-12-22

The configuration of Amazon Kinesis target is performed both when distributing a target table and after the distribution is performed.

Before you begin

Make sure you know the directory destination path where Talend Change Data Capture files are installed. This path is called <DestinationPath> in the following steps.

Procedure

  1. Prepare your distribution: open the Manager and follow this procedure until step 7 only: Preparing replication.
  2. Click Distribute to open the Property tab.
  3. In Database type, select Kinesis.
  4. Enter the mandatory parameters: region, accesskey and secretkey of your AWS account.
    For more information, see Getting Your AWS Access Keys.
    Note: The secretkey is only encrypted when surrounded by { }.
  5. Click OK. In the connection string field, you can see the result of the parameters entered before.
    Example of final connection string:
    region=eu-west-3;accesskey=BLIA6LOAH4P;secretkey={171459653AC006671D3FA2FA61334C};retry=12;traceretry=1;traceresult=1
  6. To add optional parameters to this connection string, follow the table below:
    Parameter Description
    mode Sets the behavior of the connector to send data to Amazon Kinesis:
    • mode = 0 or single mode: each record is sent as a unit and the connector waits for a response after each sending.
    • mode = 1 or packet mode: the records are sent by packet (500 records max.) and the connector waits for a response for the whole packet. This mode is selected by default.
    • mode=2 or packet threaded: the records are sent by packets (500 records max.) and distributed on the whole threads. The connector waits for a response for each packet.

    Default is 1.

    queue Sets the size of the queue used when mode = 2.

    The queue is used to dispatch packets between threads.

    Enter a numeric value. Default is 50.

    row Sets the maximum record number of a packet (500 records max.).

    This option speeds up the data sending process to the Amazon Kinesis target.

    Default is 50.

    retry Sets the maximum attempts in the PutRecords function to Amazon Kinesis. When the maximum is reached, the distribution stops in error in the case of Rate exceeded.
    • retry = -1 means no retry is made and the distribution stops in error in the case of Rate exceeded.
    • retry = 0 means the retry is limited to 1000.
    • option retry is not available when mode = 0

    Default is 100 and it is recommended not to exceed 1000.

    This option speeds up the data sending process to the Amazon Kinesis target.

    thread

    Set the number of threads used when mode = 2.

    Default is 2.

    This option speeds up the data sending process to the Amazon Kinesis target.
    trace This option is used to trace specific events into the log file (DDMSGQ.dmm):
    • trace1 to get traces when:
      • putRecordsRequestEntryList reaches 500 records,
      • putRecordsRequestEntryList reaches the maximum size of 5 MB.
    • trace2 to get traces for BeginAction, CommitAction, ClearCache.
    • trace3 to get traces for CreateCache, ClearCache, AddCache, UpdateCache, FindCache.

    Default is 0.

    traceretry This option is used to trace retry attempts to Amazon Kinesis into the log file (DDMSGQ.dmm):
    • traceretry = 0 means there is no trace.
    • traceretry = 2 means the trace is activated.

    Default is 0.

    traceresult This option is used to get metrics on retry attempts and shard distribution for each stream in a log file. This file is in the folder .\<DestinationPath>\Data\Kinesis\ and named REE_SSSMMM_statKDS.log.
    Note: Metrics are saved only when the distribution is stopped.

    traceresult = 1 means the parameter is activated.

    Default is 0.

    traceskip This option is used to trace, into the log file (DDMSGQ.dmm), the records that are ignored after the restart of a distribution that was in error.

    traceskip = 1 means the parameter is activated.

    Default is 0.

    verbose This option is used to set verbosity of the distribution.

    verbose = 1 means the parameter is activated.

    Default is 0.