URI Options - 6.3

Talend ESB Mediation Developer Guide

EnrichVersion
6.3
EnrichProdName
Talend Data Fabric
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for ESB
Talend Real-Time Big Data Platform
task
Design and Development
EnrichPlatform
Talend ESB

Common

Name

Default Value

Description

autoCreate

true

Automatically create missing directories in the file's pathname. For the file consumer, that means creating the starting directory. For the file producer, it means creating the directory the files should be written to.

bufferSize

128kb

Write buffer, sized in bytes.

fileName

null

Use Expression such as File Language to dynamically set the filename. For consumers, it is used as a filename filter. For producers, it is used to evaluate the filename to write. If an expression is set, it takes precedence over the CamelFileName header. ( Note: The header itself can also be an Expression).

The expression options support both String and Expression types. If the expression is a String type, it is always evaluated using the File Language.

If the expression is an Expression type, the specified Expression type is used; this allows you, for instance, to use OGNL expressions. For the consumer, you can use it to filter filenames, so you can for instance consume today's file using the File Language syntax: mydata-${date:now:yyyyMMdd}.txt. Starting with Camel 2.11, the producers support the CamelOverruleFileName header which will take precedence over any existing CamelFileName header. CamelOverruleFileName is used only once, and helps avoid temporarily storing a CamelFileName and needing to restore it afterwards.

flatten

false

Flatten is used to flatten the file name path to strip any leading paths, so it is purely the file name. This allows you to consume recursively into sub-directories. However, for example, if you write the files to another directory they will be written in a (flat) single directory.

Setting this to true on the producer ensures that any file name received in CamelFileName header will be stripped of any leading paths.

charset

null

This option is used to specify the encoding of the file, and camel will set the Exchange property with Exchange.CHARSET_NAME with the value of this option. You can use this on the consumer, to specify the encodings of the files, which allow Camel to know the charset it should load the file content in case the file content is being accessed. Likewise when writing a file, you can use this option to specify which charset to write the file as well.

copyAndDelete OnRenameFail

true

Whether to fallback and do a copy and delete file, in case the file could not be renamed directly. This option is not available for the [FTP|FTP2] component.

renameUsingCopy

false

Perform rename operations using a copy and delete strategy. This is primarily used in environments where the regular rename operation is unreliable (e.g. across different file systems or networks). This option takes precedence over the copyAndDeleteOnRenameFail parameter that will automatically fall back to the copy and delete strategy, but only after additional delays.

Consumer

Name

Default Value

Description

initialDelay

1000

Milliseconds before polling the file or directory starts.

delay

500

Milliseconds before the next poll of the file or directory.

useFixedDelay

true

Controls if fixed delay or fixed rate is used. See ScheduledExecutorService in JDK for details.

runLoggingLevelTRACEThe consumer logs a start/complete log line when it polls. This option allows you to configure the logging level for that.

recursive

false

if it is consuming a directory, it will look for files in all the sub-directories as well.

delete

false

If true, the file will be deleted after it is processed

noop

false

If true, the file is not moved or deleted in any way. This option is good for readonly data, or for ETL type requirements. If noop=true, Camel will set idempotent=true as well, to avoid consuming the same files over and over again.

preMove

null

If a file is to be moved before processing, use Expression such as File Language to dynamically specify the target directory name. For example to move in-progress files into the order directory set this value to order.

move

.camel

If a file is to be moved after processing, use Expression such as File Language to dynamically set the target directory name. To move files into a .done subdirectory just enter .done.

moveFailed

null

Expression (such as File Language) used to dynamically set a different target directory when moving files in case of processing (configured via move setting defined above) failed. For example, to move files into a .error subdirectory use: .error. Note: When moving the files to the "fail" location Camel will handle the error and will not pick up the file again.

include

null

Is used to include files, if filename matches the regex pattern.

exclude

null

Is used to exclude files, if filename matches the regex pattern.

antIncludenullAnt style filter inclusion, for example {{antInclude=**/*.txt}}. Multiple inclusions may be specified in comma-delimited format.
antExcludenullAnt style filter exclusion. If both antInclude and antExclude the latter takes precedence. Multiple exclusions may be specified in comma-delimited format.
antFilter-CaseSensitivetrueStarting with Camel 2.11, whether Ant filters are case sensitive or not.

idempotent

false

Option to use the Idempotent Consumer EIP pattern to let Camel skip already processed files. This will by default use a memory based LRUCache that holds 1000 entries. If noop=true then idempotent will be enabled as well to avoid consuming the same files over and over again.

idempotentKey

Expression

Starting with Camel 2.11, use of a custom idempotent key. By default the absolute path of the file will be used. Camel's File Language can be used to specify the file name and size: idempotentKey=${file:name}-${file:size}.

idempotent-Repository

null

Pluggable repository as a org.apache.camel. processor.idempotent.MessageIdRepository class. This will by default use MemoryMessageIdRepository if none is specified and idempotent is true.

inProgress-Repository

memory

A pluggable in-progress repository org.apache.camel.spi. IdempotentRepository. The in-progress repository is used to account the current in-progress files being consumed. By default a memory based repository is used.

filter

null

Pluggable filter as a org.apache.camel.component.file. GenericFileFilter class. This will skip files if filter returns false in its accept() method.

sorter

null

Pluggable sorter as a java.util.Comparator <org.apache.camel.component.file.GenericFile> class.

sortBy

null

Built-in sort using the File Language. Supports nested sorts, so you can have a sort by file name and as a second group sort by modified date. See sorting section below for details.

readLock

marker-File

Used by consumer, to only poll the files if it has exclusive read-lock on the file (that is, the file is not in-progress or being written). Camel will wait until the file lock is granted. This option provides the build in strategies:

markerFile Camel creates a marker file (fileName.camelLock) and then holds a lock on it. This option is *not* available for the FTP component.

changed is using file length/modification timestamp to detect whether the file is currently being copied or not. This will at least use 1 sec. to determine this, so this option cannot consume files as fast as the others, but can be more reliable as the JDK IO API cannot always determine whether a file is currently being used by another process. The option readLockCheckInterval can be used to set the check frequency. Note the FTP option fastExistsCheck can be enabled to speed up this readLock strategy, if the FTP server supports the LIST operation with a full file name (some servers may not). not avail for the FTP component.

fileLock is for using java.nio.channels.FileLock. This option is not available for the FTP component. This approach should be avoided when accessing a remote file system via a mount/share unless that file system supports distributed file locks.

rename is for using a try to rename the file as a test if we can get exclusive read-lock.

none is for no read locks at all. Note the read locks changed, fileLock and rename will also use a markerFile as well, to ensure not picking up files that may be in process by another Camel consumer running on another node (eg cluster). This is supported only by the file component (not the ftp component).

readLockTimeout

10000

Optional timeout in milliseconds for the read-lock, if supported by the read-lock. If the read-lock could not be granted and the timeout triggered, then Camel will skip the file. At next poll Camel, will try the file again, and this time maybe the read-lock could be granted. Use a value of 0 or lower to indicate forever. Currently fileLock, changed and rename support the timeout. Note: for the FTP component the default value is 20000.

readLockCheck-Interval

1000

Interval in milliseconds for the read-lock, if supported by the read lock. This interval is used for sleeping between attempts to acquire the read lock. For example when using the changed read lock, you can set a higher interval period to cater for slow writes . The default of 1 sec. may be too fast if the producer is very slow writing the file.

readLock-MinLength

1

This option applied only for readLock=changed. This option allows you to configure a minimum file length. By default Camel expects the file to contain data, and thus the default value is 1. You can set this option to zero to allow consuming zero-length files.

readLockLoggingLevel

WARN

Starting with Camel 2.12: Logging level used when a read lock could not be acquired. By default a WARN is logged. You can change this level, for example to OFF to not have any logging. This option is only applicable for readLock of types: changed, fileLock, rename.

directoryMust-Exist

false

Similar to startingDirectoryMustExist but this applies during polling recursive sub-directories.

doneFileName

null

If provided, Camel will only consume files if a done file exists. This option configures what file name to use. Either you can specify a fixed name, or you can use dynamic placeholders. The done file is always expected in the same folder as the original file. See using done file and writing done file sections for examples.

exclusiveRead-LockStrategy

null

Pluggable read-lock as a org.apache.camel.component.file. GenericFileExclusiveReadLockStrategy implementation.

maxMessages-PerPoll

0

An integer that defines the maximum number of messages to gather per poll. By default, no maximum is set. It can be used to set a limit of, for example, 1000 to avoid having the server read thousands of files as it starts up. Set a value of 0 or negative to disable it. You can use the eagerMaxMessagesPerPoll option and set this to false to allow to scan all files first and then sort afterwards.

eagerMax-MessagesPerPoll

true

Allows for controlling whether the limit from maxMessagesPerPoll is eager or not. If eager then the limit is during the scanning of files. Whereas false would scan all files, and then perform sorting. Setting this option to false allows to sort all files first, and then limit the poll. Note that this requires a higher memory usage as all file details are in memory to perform the sorting.

minDepth0The minimum depth to start processing when recursively processing a directory. Using minDepth=1 means the base directory. Using minDepth=2 means the first sub directory.
maxDepthInteger. MAX_VALUEThe maximum depth to traverse when recursively processing a directory.

processStrategy

null

A pluggable org.apache.camel.component.file. GenericFileProcessStrategy allowing you to implement your own readLock option or similar. Can also be used when special conditions must be met before a file can be consumed, such as a special ready file exists. If this option is set then the readLock option does not apply.

startingDirect-oryMustExist

false

whether the starting directory must exist. Keep in mind that the autoCreate option is default enabled, which means the starting directory is normally auto created if it doesn't exist. You can disable autoCreate and enable this to ensure the starting directory must exist. It will then throw an exception if the directory doesn't exist.

pollStrategynullA pluggable org.apache.camel.spi. PollingConsumerPollingStrategy allowing you to provide your custom implementation to control error handling usually occurred during the poll operation *before* an Exchange has been created and routed in Camel. In other words the error occurred while the polling was gathering information, for instance access to a file network failed so Camel cannot access it to scan for files. The default implementation will log the caused exception at WARN level and ignore it.
sendEmpty-MessageWhenIdlefalseIf the polling consumer did not poll any files, you can enable this option to send an empty message (no body) instead.
consumer.bridge-ErrorHandlerfalseAllows for bridging the consumer to the Camel routing Error Handler, which mean any exceptions occurred while trying to pickup files, or the likes, will now be processed as a message and handled by the routing Error Handler. By default the consumer will use the org.apache.camel.spi.ExceptionHandler to deal with exceptions, that by default will be logged at WARN/ERROR level and ignored.
scheduled-ExecutorServicenullAllows for configuring a custom/shared thread pool to use for the consumer. By default each consumer has its own single threaded thread pool. This option allows you to share a thread pool among multiple file consumers.
schedulernull Camel 2.12: To use a custom scheduler to trigger the consumer to run. See more details at Polling Consumer, for example there is a Quartz2, and Spring based scheduler that supports CRON expressions.
backoffMultiplier0 Camel 2.12: To let the scheduled polling consumer backoff if there has been a number of subsequent idles/errors in a row. The multiplier is then the number of polls that will be skipped before the next actual attempt is happening again. When this option is in use then backoffIdleThreshold and/or backoffErrorThreshold must also be configured. See more details at Polling Consumer.
backoffIdleThreshold0 Camel 2.12: The number of subsequent idle polls that should happen before the backoffMultipler should kick-in.
backoffErrorThreshold0 Camel 2.12: The number of subsequent error polls (failed due some error) that should happen before the backoffMultipler should kick-in.

Default behavior for file consumer

  • By default the file is locked for the duration of the processing.

  • After the route has completed, files are moved into the .camel subdirectory, so that they appear to be deleted.

  • The File Consumer will always skip any file whose name starts with a dot, such as ., .camel, .m2 or .groovy.

  • Only files (not directories) are matched for valid filename, if options such as: include or exclude are used.

Producer

Name

Default Value

Description

fileExist

Override

What to do if a file already exists with the same name. The following values can be specified: Override, Append, Fail, Ignore, Move, and TryRename (Camel 2.11.1).

  • Override, which is the default, replaces the existing file.

  • Append adds content to the existing file.

  • Fail throws a GenericFileOperation-Exception, indicating that there is already an existing file.

  • Ignore silently ignores the problem and does not override the existing file, but assumes everything is okay. The Move option will move any existing files, before writing the target file. The corresponding moveExisting option must be configured. The option eagerDeleteTargetFile can be used to control what to do if an moving the file, and there exists already an existing file, otherwise causing the move operation to fail.

  • TryRename (Camel 2.11.1) is only applicable if tempFileName option is in use. This allows to try renaming the file from the temporary name to the actual name, without doing any exists check. This check may be faster on some file systems and especially FTP servers.

tempPrefix

null

This option is used to write the file using a temporary name and then, after the write is complete, rename it to the real name. Can be used to identify files being written and also avoid consumers (not using exclusive read locks) reading in-progress files. Is often used by FTP when uploading big files.

tempFileName

null

The same as tempPrefix option but offering a more fine grained control on the naming of the temporary filename as it uses the File Language .

moveExisting

null

Expression used to compute file name to use when fileExist=Move is configured. To move files into a backup subdirectory just enter backup. This option supports only the following File Language tokens: "file:name", "file:name.ext", "file:name.noext", "file:onlyname", "file:onlyname.noext", "file:ext", and "file:parent". Notice the "file:parent" is not supported by the FTP component, as the FTP component can move existing files only to a relative directory based on the current directory.

keepLastModified

false

If enabled, will keep the last modified timestamp from the source file (if any). This will use the Exchange. FILE_LAST_MODIFIED header to located the timestamp. This header can contain either a java.util.Date or long with the timestamp. If the timestamp exists and the option is enabled it will set this timestamp on the written file. Note: This option only applies to the file producer. You cannot use this option with any of the ftp producers.

eagerDeleteTarget-File

true

Whether or not to eagerly delete any existing target file. (This option only applies when you use fileExists=Override and the tempFileName option). You can use this to disable deleting the target file before the temp file is written. For example you may have large files and want the target file to persist while the temp file is being written. Setting eagerDeleteTargetFile to false ensures the target file is only deleted until the very last moment, just before the temp file is being renamed to the target filename. This option is also used to control whether to delete any existing files when fileExist=Move is enabled and an existing file is present. If this option copyAndDeleteOnRenameFailis false, then an exception will be thrown if an existing file existed, if it's true, then the existing file is deleted before the move operation.

doneFileName

null

If provided, then Camel will write a second done file when the original file has been written. The done file will be empty. This option configures what file name to use. Either you can specify a fixed name. Or you can use dynamic placeholders. The done file will always be written in the same folder as the original file. See writing done file section for examples.

allowNullBody

false

Used to specify if a null body is allowed during file writing. If set to true then an empty file will be created, when set to false, and attempting to send a null body to the file component, a GenericFileWriteException of ""Cannot write null body to file" will be thrown. If the "fileExist" option is set to "Override"", then the file will be truncated, and if set to "append"" the file will remain unchanged.

forceWrites

true

Starting with Camel 2.10.5/2.11, whether to force syncing writes to the file system. You can turn this off if you do not want this level of guarantee, for example if writing to logs / audit logs etc; this would yield better performance.

Default behavior for file producer

By default it will override any existing file, if one exist with the same name.

Note

Override is the default for the file producer. This is also the default file operation using java.io.File - and also the default for the FTP library we use in the camel-ftp component.