Options - 6.3

Talend ESB Mediation Developer Guide

EnrichVersion
6.3
EnrichProdName
Talend Data Fabric
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for ESB
Talend Real-Time Big Data Platform
task
Design and Development
EnrichPlatform
Talend ESB

Name

Default Value

Description

overwrite

true

The file can be overwritten

append

false

Append to existing file. Notice that not all HDFS file systems support the append option.

bufferSize

4096

The buffer size used by HDFS

replication

3

The HDFS replication factor

blocksize

67108864

The size of the HDFS blocks

fileType

NORMAL_FILE

It can be SEQUENCE_FILE, MAP_FILE, ARRAY_FILE, or BLOOMMAP_FILE, see Hadoop

fileSystemType

HDFS

It can be LOCAL for local filesystem

keyType

NULL

The type for the key in case of sequence or map files. See below.

valueType

TEXT

The type for the key in case of sequence or map files. See below.

splitStrategy

 

A string describing the strategy on how to split the file based on different criteria. See below.

openedSuffix

opened

When a file is opened for reading/writing the file is renamed with this suffix to avoid to read it during the writing phase.

readSuffix

read

Once the file has been read is renamed with this suffix to avoid to read it again.

initialDelay

0

For the consumer, how much to wait (milliseconds) before to start scanning the directory.

delay

0

The interval (milliseconds) between the directory scans.

pattern

*

The pattern used for scanning the directory

chunkSize

4096

When reading a normal file, this is split into chunks producing a message per chunk.

connectOnStartup

true

Camel 2.9.3/2.10.1: Whether to connect to the HDFS file system on starting the producer/consumer. If false then the connection is created on-demand. Notice that HDFS may take up till 15 minutes to establish a connection, as it has hardcoded 45 x 20 sec redelivery. By setting this option to false allows your application to startup, and not block for up till 15 minutes.

owner

 

Camel 2.13/2.12.4: The file owner must match this owner for the consumer to pickup the file. Otherwise the file is skipped.

KeyType and ValueType

  • NULL it means that the key or the value is absent

  • BYTE for writing a byte, the java Byte class is mapped into a BYTE

  • BYTES for writing a sequence of bytes. It maps the java ByteBuffer class

  • INT for writing java integer

  • FLOAT for writing java float

  • LONG for writing java long

  • DOUBLE for writing java double

  • TEXT for writing java strings

BYTES is also used with everything else, for example, in Camel a file is sent around as an InputStream, int this case is written in a sequence file or a map file as a sequence of bytes.