Skip to main content Skip to complementary content

Enabling input streaming for a DSQL map

Availability-noteBeta

Use input streaming to avoid memory issues when working with large input data.

In a regular DSQL map execution, the input is first loaded into memory before the DSQL script is executed. While this provides the best performances, it may cause memory issues when the input data is large. With input streaming, the records are processed in blocks of 1000 records and the memory is freed after each block.

You can only enable streaming on one element in your map. If you enable it on another element, streaming will be disabled on the first element.

Before you begin

You have created a DSQL map.

Procedure

  1. Open your DSQL map.
  2. Right-click the output looping element on which you want to enable streaming and click Enable streaming.
    Information noteImportant: This option is available only if the element selected meets the following conditions:
    • It is a looping element.
    • It contains a FROM clause referring directly to an input looping element, not through an expression.
    • It is a Group element, not a primitive element.
  3. Save your map

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here