Failed to send a large document to the Talend Dictionary Service server - 7.2

author
Talend Documentation Team
EnrichVersion
7.2
EnrichProdName
Talend Big Data Platform
Talend Data Fabric
Talend Data Management Platform
Talend Data Services Platform
Talend MDM Platform
Talend Real-Time Big Data Platform
task
Data Quality and Preparation > Managing semantic types
Installation and Upgrade
EnrichPlatform
Talend Data Preparation
Talend Data Stewardship

Sending a large document to the Talend Dictionary Service server

Sending a large document to the Talend Dictionary Service server may fail due to Apache Kafka memory limitations.

When using the command line tool for sending a document to the Talend Dictionary Service server, an error message is returned if the input document exceeds the maximum configured size of a request the socket server will accept:

Failed >>> Http Status: 406 -> input document is too large

You can edit configuration properties to handle large messages with Apache Kafka.

Procedure

  1. Edit the following properties in the <Kafka_path>/config/server.properties configuration file:
       
    message.max.bytes Specifies the maximum size of a message that the server can receive. The number value for this property must be in sync with the maximum fetch size your consumers use.
    socket.request.max.bytes Specifies the maximum number of bytes in a socket request. A request can consist of data from multiple topic partitions and therefore can contain many messages. The number value for this property should be smaller than the Java heap size.
  2. If you modified the value for message.max.bytes, add the following properties to <Data_Preparation_Path>/config/application.properties and <Data_Stewardship_Path>/config/data-stewardship.properties:
    • spring.cloud.stream.kafka.bindings.input.consumer.fetch.message.max.bytes
    • spring.cloud.stream.kafka.bindings.category.consumer.fetch.message.max.bytes
    • spring.cloud.stream.kafka.bindings.regEx.consumer.fetch.message.max.bytes

    The value set for these properties must match the message.max.bytes value set in <Kafka_path>/config/server.properties.

    For more information, see Apache Kafka documentation.