Enable your Remote Engine to regularly and automatically delete useless artifacts to free up storage space.
This cleanup does not impact any ongoing task runs, because a task uses artifacts from the
local Remote Engine repository only at the moment of deployment. Task-related artifacts are
downloaded to this local repository when task is run for the first time. After that, these
downloaded artifacts are reused for the following runs of the same task.
The highest security level of your antivirus system may impede the automatic cleanup
process, for example, by preventing the deletion of temporary files. To address this issue,
lowering the security level of the antivirus software can be a potential solution.
Procedure
-
Open the <RemoteEngineInstallationDirectory>/etc/org.talend.ipaas.rt.deployment.agent.cfg
file to edit it.
-
Set the following parameters:
- local.maven.clean.interval.days: the automatic cleanup is
enabled once you put a value greater than 0.
For example,
local.maven.clean.interval.days=2
Then the cleanup
happens every two days.
Note: Ensure that there are no trailing spaces in the
parameter value, as this results in engine pairing
failure.
- local.maven.clean.max.gb.size: the maximum size (in GB)
allocated for artifacts to be stored in
<RemoteEngineInstallationDirectory>/data/m2/actions
in Remote Engine.
For example,
local.maven.clean.max.gb.size=8
If all the stored
artifacts together exceed 8GB, when a scheduled cleanup service starts, the
service deletes the oldest artifacts for which the last execution times are
beyond the retention period, until the total size is reduced to be equal to
or less than 8GB.
- local.maven.clean.older.than.days: the retention period
of an artifact, for example,
local.maven.clean.older.than.days=3
When you have
an ongoing task expected to run for 4 days, the 3-day retention period in
this example does not impact this task.
Artifacts are deleted only when both of the following conditions are satisfied:
- The total size of the stored artifacts exceeds the specified limit.
- The old artifacts have their last execution times beyond specified
retention period.
An artifact is downloaded to the local Remote Engine
repository only when a related task is run for the first time; after
that, this artifact is reused in the following runs of the same task,
with its last execution time updated.
-
Save the file.
Results
A log message is returned every time when a scheduled cleanup is performed. This
message starts with the following wording:
Remote Engine m2 artifacts cleanup performed:
If this message tells
you that no artifact has been deleted, you may need to consider to decrease the size limit
or shorten the retention period.
Note: This automatic cleanup does not delete files in the
<RemoteEngineInstallationDirectory>/TalendJobServersFiles/archiveJobs/
folder and the
\data\tmp\ folder:
- Additional dependency files such as different log4j versions are stored in
<RemoteEngineInstallationDirectory>/TalendJobServersFiles/archiveJobs/.
- Temporary files related to Karaf is stored in \data\tmp\ and
are automatically deleted when Karaf, along with the Remote Engine, stops
running.