Chunking - Cloud - 8.0

Talend ESB Container Administration Guide

Talend Cloud
Talend Data Fabric
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for ESB
Talend Real-Time Big Data Platform
Talend ESB
Talend Runtime
Administration and Monitoring
Design and Development
Installation and Upgrade
Last publication date
There are a number of parameters related to chunking, so we discuss this here in more detail. There are two ways of putting the body of a message into an HTTP stream:
  1. Standard scheme: this is used by most browsers. It consists in specifying a Content-Length header in the HTTP headers. This allows the receiver to know how much data is coming and when to stop reading.

    The problem with this approach is that the length needs to be pre-determined. The data cannot be streamed as generated as the length needs to be calculated upfront.

    Thus, if chunking is turned off, we need to buffer the data in a byte buffer (or temp file if it is too large) so that the Content-Length can be calculated.

  2. Chunked scheme: with this mode, the data is sent to the receiver in chunks. Each chunk is preceded by a hexadecimal chunk size. When a chunk size is 0, the receiver knows that all the data has been received.

    This mode allows better streaming, as we just need to buffer a small amount, up to 8K by default. When the buffer fills, the chunk is written out.

Some parameters in Configuration properties allow you to specify the details of the chunking.

In general, the Chunked scheme will perform better as the streaming can take place directly. However, there are some problems with chunking:
  • Many proxy servers don't understand it, especially older proxy servers. Many proxy servers want the Content-Length up front so they can allocate a buffer to store the request before passing it onto the real server.
  • Some of the older Web Services stacks also have problems with Chunking - specifically, older versions of .NET.
Note: If you are getting strange errors (generally not SOAP faults, but other HTTP-type errors) when trying to interact with a service, try turning off chunking to see if that helps.