Failed to send a large document to the dictionary server

author
Talend Documentation Team
EnrichVersion
6.3
2.1
EnrichProdName
Talend Real-Time Big Data Platform
Talend Data Services Platform
Talend Data Management Platform
Talend MDM Platform
Talend Data Fabric
Talend Big Data Platform
task
Installation and Upgrade
Data Quality and Preparation > Managing semantic types
Design and Development > Third-party systems > Messaging components (Integration) > Kafka components
Data Quality and Preparation > Third-party systems > Messaging components (Integration) > Kafka components
Data Governance > Third-party systems > Messaging components (Integration) > Kafka components
EnrichPlatform
Talend Data Stewardship

Failed to send a large document to the dictionary server

Sending a large document to the dictionary server may fail due to Apache Kafka memory limitations.
Symptoms/Description

When using the command line tool for sending a document to the dictionary server, an error message is returned if the input document exceeds the maximum configured size of a request the socket server will accept:

Failed >>> Http Status: 406 -> input document is too large
Resolution

If you need to handle large messages in Apache Kafka:

  1. Edit the following properties in the <Kafka_path>/config/server.properties configuration file:

    Property Description
    message.max.bytes
    Specifies the maximum size of a message that the server can receive. The number value for this property must be in sync with the maximum fetch size your consumers use.
    socket.request.max.bytes
    Specifies the maximum number of bytes in a socket request. A request can consist of data from multiple topic partitions and therefore can contain many messages. The number value for this property should be smaller than the Java heap size.
  2. If you modified the value for message.max.bytes , add the following properties to <Data_Preparation_Path>/config/application.properties and <Data_Stewardship_Path>/config/data-stewardship.properties :
    • spring.cloud.stream.kafka.bindings.input.consumer.fetch.message.max.bytes

    • spring.cloud.stream.kafka.bindings.category.consumer.fetch.message.max.bytes

    • spring.cloud.stream.kafka.bindings.regEx.consumer.fetch.message.max.bytes

The value set for these properties must match the message.max.bytes value set in <Kafka_path>/config/server.properties .

For more information, see Apache Kafka documentation .