Automating artifact cleanup for better performance of your Remote Engine - Cloud

Talend Remote Engine User Guide for Linux

Version
Cloud
Language
English (United States)
EnrichDitaval
linux
Product
Talend Cloud
Module
Talend Remote Engine
Content
Design and Development
Installation and Upgrade

Enable your Remote Engine to regularly and automatically delete useless artifacts to free up storage space.

This cleanup does not impact any ongoing task runs, because a task uses artifacts from the local Remote Engine repository only at the moment of deployment. Task-related artifacts are downloaded to this local repository when task is run for the first time. After that, these downloaded artifacts are reused for the following runs of the same task.

Note that an extreme security level of your antivirus system could impede the performance of this automatic cleanup, for example, prevent the deletion of temporary files. Reducing this level can often resolve this issue.

Procedure

  1. Open the <RemoteEngineInstallationDirectory>/etc/org.talend.ipaas.rt.deployment.agent.cfg file to edit it.
  2. Set the following parameters:
    • local.maven.clean.interval.days: the automatic cleanup is enabled once you put a value greater than 0.
      For example,
      local.maven.clean.interval.days=2
      Then the cleanup happens every two days.
    • local.maven.clean.max.gb.size: the maximum size (in GB) allocated for artifacts to be stored in Remote Engine.
      For example,
      local.maven.clean.max.gb.size=8
      If all the stored artifacts together exceed 8GB, when a scheduled cleanup service starts, the service deletes the oldest artifacts for which the last execution times are beyond the retention period, until the total size is reduced to be equal to or less than 8GB.
    • local.maven.clean.older.than.days: the retention period of an artifact, for example,
      local.maven.clean.older.than.days=3

      When you have an ongoing task expected to run for 4 days, the 3-day retention period in this example does not impact this task.

    Artifacts are deleted only when both of the following conditions are satisfied:
    • The total size of the stored artifacts exceeds the specified limit.
    • The old artifacts have their last execution times beyond specified retention period.

      An artifact is downloaded to the local Remote Engine repository only when a related task is run for the first time; after that, this artifact is reused in the following runs of the same task, with its last execution time updated.

  3. Save the file.

Results

A log message is returned every time when a scheduled cleanup is performed. This message starts with the following wording:
Remote Engine m2 artifacts cleanup performed:
If this message tells you that no artifact has been deleted, you may need to consider to decrease the size limit or shorten the retention period.
Note: This automatic cleanup does not delete files in the <RemoteEngineInstallationDirectory>/TalendJobServersFiles/archiveJobs/ folder and the \data\tmp\ folder:
  • Additional dependency files such as different log4j versions are stored in <RemoteEngineInstallationDirectory>/TalendJobServersFiles/archiveJobs/.
  • Temporary files related to Karaf is stored in \data\tmp\ and are automatically deleted when Karaf, along with the Remote Engine, stops running.