Automating artifact cleanup for better performance of your Remote Engine

Enable your Remote Engine to regularly and automatically delete useless artifacts to free up storage space.

This cleanup does not impact any ongoing task runs, because a task uses artifacts from the local Remote Engine repository only at the moment of deployment. Task-related artifacts are downloaded to this local repository when task is run for the first time. After that, these downloaded artifacts are reused for the following runs of the same task.

The highest security level of your antivirus system may impede the automatic cleanup process, for example, by preventing the deletion of temporary files. To address this issue, lowering the security level of the antivirus software can be a potential solution.

Note: Make sure each Remote Engine instance has its separate TalendJobServersFiles folder, as explained in Deploying multiple Remote Engines.

Procedure

Open the <RemoteEngineInstallationDirectory>/etc/org.talend.ipaas.rt.deployment.agent.cfg file to edit it.
Set the following parameters:
- local.maven.clean.interval.days: the automatic cleanup is enabled once you put a value greater than 0.
  For example,
  local.maven.clean.interval.days=2
  Then the cleanup happens every two days.
  Information noteNote: Ensure that there are no trailing spaces in the parameter value, as this results in engine pairing failure.
- local.maven.clean.max.gb.size: the maximum size (in GB) allocated for artifacts to be stored in <RemoteEngineInstallationDirectory>/data/m2/actions in Remote Engine.
  For example,
  local.maven.clean.max.gb.size=8
  If all the stored artifacts together exceed 8GB, when a scheduled cleanup service starts, the service deletes the oldest artifacts for which the last execution times are beyond the retention period, until the total size is reduced to be equal to or less than 8GB.
- local.maven.clean.older.than.days: the retention period of an artifact, for example,
```
local.maven.clean.older.than.days=3
```
  When you have an ongoing task expected to run for 4 days, the 3-day retention period in this example does not impact this task.
Artifacts are deleted only when both of the following conditions are satisfied:
- The total size of the stored artifacts exceeds the specified limit.
- The old artifacts have their last execution times beyond specified retention period.
  An artifact is downloaded to the local Remote Engine repository only when a related task is run for the first time; after that, this artifact is reused in the following runs of the same task, with its last execution time updated.
Save the file.

Results

A log message is returned every time when a scheduled cleanup is performed. This message starts with the following wording:

Remote Engine m2 artifacts cleanup performed:

If this message tells you that no artifact has been deleted, you may need to consider to decrease the size limit or shorten the retention period.

Note: This automatic cleanup does not delete files in the <RemoteEngineInstallationDirectory>/TalendJobServersFiles/archiveJobs/ folder and the \data\tmp\ folder:

Additional dependency files such as different log4j versions are stored in <RemoteEngineInstallationDirectory>/TalendJobServersFiles/archiveJobs/.
Temporary files related to Karaf is stored in \data\tmp\ and are automatically deleted when Karaf, along with the Remote Engine, stops running.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!

Leave your feedback here