CI builder-related Maven parameters - Cloud

Talend Software Development Life Cycle for Cloud - Best Practices Guide

Version
Cloud
Language
English
Product
Talend Cloud
Module
Talend Artifact Repository
Talend Management Console
Talend Studio
Content
Administration and Monitoring
Deployment
Design and Development
These parameters are detailed in the embedded help of the plugin (see the generate-mojo.html file) and can be used either in the mavenOpts (Jenkins)/mavenOptions(Azure DevOps) or the mvn(Jenkins)/options(Azure DevOps) commands of your CI build definition.
Maven parameter Value
Talend CommandLine parameters -Dproduct.path

Path to the Talend CommandLine installation directory.

Example:
-Dproduct.path=/opt/Talend/commandline8
From version 8 onwards, Talend Studio provides two scripts, commandline_upgrade.bat for Windows and commandline-linux_upgrade.sh for Linux, which allows you to install Talend CommandLine by upgrading either a brand new or an existing Talend Studio.
-Dp2Installer.path Path to the Installer files.

If -Dproduct.path is specified then the Talend CommandLine will be installed in the value specified for this parameter.

Example:
-Dproduct.path=/home/.installation/.commandline_8
-Dp2Installer.path=/home/.installation/.p2Installer_8
This allows you to use different paths for different product versions.
-Dlicense.path Path to the license of your Talend product.

If the licence path value is a URL and credentials are needed to access it, use the -Dlicense.remote.user and -Dlicense.remote.password parameters.

You can use this parameter at build time to update an expired license. To change a license, you can use this parameter and add the -Dinstaller.clean=true parameter if you are not using the mvn org.talend.ci:builder-maven-plugin:8.0.4:install command.

-Dtalend.studio.p2.base

The URL of the repository for Talend Studio feature packages, or the path to the unzipped folder of the Talend_Full_Studio_p2_repository-YYYYMMDD_HHmm-VA.B.C.zip archive for Talend Studio feature packages.

Example for the URL of the official Talend repository for Talend Studio feature packages:

-Dtalend.studio.p2.base=https://update.talend.com/Studio/8/base

Example for the path to the unzipped folder of the archive for Talend Studio feature packages:

-Dtalend.studio.p2.base=D:\Talend\v80\StudioP2Base

For more information, see Setting up update repositories for Talend Studio and Continuous Integration.

-Dtalend.studio.p2.update

The URL of the repository for Talend Studio updates, or the path to the unzipped folder of the patch zip assigned to you or the archive for Talend Studio updates.

If you want to migrate your projects to a newer version, you need to install patches, including Talend Studio monthly updates, manual patches, and component patches, using this parameter at build time. The main advantage of this parameter is that it allows you to test and apply a patch on one or several existing projects using CI before applying the patch to Talend Studio.

Example for the URL of the official Talend repository for the latest Talend Studio monthly update:

-Dtalend.studio.p2.update=https://update.talend.com/Studio/8/updates/latest

Example for the path to the unzipped folder of the archive for a Talend Studio monthly update:

-Dtalend.studio.p2.update=D:\Talend\v80\StudioR202201

For more information, see Setting up update repositories for Talend Studio and Continuous Integration.

(Optional)

-Declipse.keyring and -Declipse.password

-Dtalend.studio.p2.base.user and -Dtalend.studio.p2.base.pwd

-Dtalend.studio.p2.update.user and -Dtalend.studio.p2.update.pwd

If basic authentication is enabled in the Talend Studio update repositories, use these parameters to enter the credentials encrypted using the Eclipse secure storage.

This feature is available from 8.0 R2022-09 onwards.

Example:

-Dtalend.studio.p2.base=http://localhost:8081/Studio/8/base
-Dtalend.studio.p2.update=http://XYZ.talend.com:8081/Studio/8/updates/R2022-07v2/
-Declipse.keyring=<keyring-file-path>/mykeyring
-Declipse.password=<master-password-file-path>/mymaster
-Dtalend.studio.p2.base.user=admin
-Dtalend.studio.p2.base.pwd=admin123
-Dtalend.studio.p2.update.user=admin
-Dtalend.studio.p2.update.pwd=admin123

For more information on how to encrypt the credentials in the Talend Studio configuration files, see Basic authentication for update repositories in Talend Studio.

-Dsigner.path

-Dsigner.keystore.password

-Dsigner.key.password

-Dsigner.key.alias

(available from R2022-06)

Specify the custom Java KeyStore for the Job artifact signature when building the Job artifact packages, to prevent accidental or malicious modification of your Job artifact. By default, a Java KeyStore issued by Talend is used for the Job artifact signature.
  • -Dsigner.path: the path to your custom Java KeyStore.
  • -Dsigner.keystore.password: the KeyStore password, either Maven-encrypted or in plain text.
  • -Dsigner.key.password: the key password, either Maven-encrypted or in plain text.
  • -Dsigner.key.alias: the alias name associated with your KeyStore.

For more information about how to encrypt passwords using Maven, see Password Encryption.

After you publish your Job artifacts to Talend Cloud using Talend Cloud Publisher, the Remote Engine will validate their signature before running them. For more information, see Running trusted tasks with your custom signature.

Example for encrypted passwords:

mvn clean package
-f D:/Talend/ciworkspace/projectname/poms/pom.xml
-Dproduct.path=D:/Talend/Studio-80
-Dsigner.path=D:/Talend/JKS/code-signing_valid.jks
-Dsigner.keystore.password={ugVME8/sYhMHjXYNN74oemBjBxpsRFWb31etYRyiif8=}
-Dsigner.key.password={SozpGRQPFNUHQqDvo3gTW/hEv0LZ1FfKZk1fFngb1Fc=}
-Dsigner.key.alias=code-signing

Example for passwords in plain text:

mvn clean package
-f D:/Talend/ciworkspace/projectname/poms/pom.xml
-Dproduct.path=D:/Talend/Studio-80
-Dsigner.path=D:/Talend/JKS/code-signing_valid.jks
-Dsigner.keystore.password=keystore123
-Dsigner.key.password=key123
-Dsigner.key.alias=code-signing
-Dpatch.path Path to the patch you have installed or want to install (manual patches, custom component patches, hotfixes, etc.).

Note that this parameter is deprecated from 8.0 R2022-04 onwards. Alternatively, you can use the -Dtalend.studio.p2.update parameter.

Example if you have installed the patch locally:
-Dpatch.path=D:/patches/Patch_TPS-2222.zip,D:/patches/Patch_TPS-3333.zip
Example of patch to download:
-Dpatch.path=http://localhost:8085/nexus/content/repositories/releases/Patch_TPS-2222.zip,http://localhost:8085/nexus/content/repositories/releases/Patch_TPS-3333.zip

If credentials are needed to access the remote patch location, use the -Dpatch.remote.user and -Dpatch.remote.password parameters. The patch.remote.password parameter can set a password encrypted by Maven.

-Dinstaller.clean Performs a clean installation and removes the .metadata folder under your CI workspace directory before reinstalling Talend CommandLine.

Example: -Dinstaller.clean=true

-DforceUpdate Forces the installation of Talend CommandLine from the update site, even if a local installation is present.

Note that this parameter is deprecated from 8.0 R2022-04 onwards. Alternatively, you can use the -Dinstaller.clean parameter.

Example: -DforceUpdate=true

-Dinstall.org.eclipse.ecf.provider.filetransfer.httpclient4.retrieve.connectTimeout

-Dinstall.org.eclipse.ecf.provider.filetransfer.httpclient4.retrieve.readTimeout

-Dinstall.org.eclipse.ecf.provider.filetransfer.httpclient4.browse.connectTimeout

Timeout settings in milliseconds to indicate how long to wait for the connection to open before throwing an error. These parameters help prevent the CI server from being blocked when it does not receive a response soon enough. Their default values are all 12000 milliseconds.

Example:

-Dinstall.org.eclipse.ecf.provider.filetransfer.httpclient4.retrieve.connectTimeout=12000
-Dinstall.org.eclipse.ecf.provider.filetransfer.httpclient4.retrieve.readTimeout=12000
-Dinstall.org.eclipse.ecf.provider.filetransfer.httpclient4.browse.connectTimeout=12000
-Dgeneration.type Type local as the value to generate the sources of your artifacts and Tests locally without having to keep the Talend CommandLine up and running during the build execution.
Example:
-Dgeneration.type=local

The server generation type is no longer supported.

-DcveReportFilePath Allows you to define the path where you want to store the report containing the list of fixed vulnerabilities in your artifacts.

This parameter can only be used with the org.talend.ci:builder-maven-plugin:8.0.X:detectCVE Maven command.

For more information, see Detecting the fixed vulnerabilities (CVEs) of your artifacts while building.

Example:
-DcveReportFilePath=/home/talend/cvereport.csv
-Dcommandline.skip Type true as the value to skip the source code generation of your artifacts.

This parameter can only be used if you have previously generated your artifacts using Talend CommandLine.

Example:
-Dcommandline.skip=true
-Dtalend.profile.module Allows you to activate or deactivate profiles when generating POM files by setting its value to true or false.

If you have not installed the 8.0 R2022-06 Studio monthly update or a later one provided by Talend and if a main project has duplicated reference projects, you need to add this parameter and set its value to true when generating POM files for the main project to avoid duplicate projects module definition in the POM files and then specify profiles using the -P parameter when building projects and artifacts using the Maven package command. For more information, see Generating POM files for your projects.

If your projects are migrated from version 7.1, 7.2, or 7.3, the profiles can be activated or deactivated via selecting or clearing the Set reference project modules in profile check box under the Build > Maven view in the Project Settings dialog box in Talend Studio. You can override the profile settings in Talend Studio via the -Dtalend.profile.module parameter when generating POM files using CI.

Example for activating profiles:

-Dtalend.profile.module=true
-Dtalend.exclude.deleted Allows you to exclude modules of deleted items from the POM files by setting its value to true when generating POM files for your projects.

Example:

-Dtalend.exclude.delete=true
-DprojectFilter Allows you to build specific projects. Regular expressions are supported.

Example of filter on all projects with a name starting with CI:

-DprojectFilter="CI.*"
Example of filter on two specific projects projectA and projectB:
-DprojectFilter="projectA|projectB"
-DitemFilter Allows you to build specific project items. Regular expressions are supported.

Note that if -DitemFilter is used together with -DprojectFilter, -DitemFilter will be applied only on the projects filtered by -DprojectFilter. If the filtered projects have any reference projects that are not filtered by -DprojectFilter, the POM files will be generated for all items in these reference projects.

Example of filter on Job types:

  • -DitemFilter=(type=process) to filter on all Standard Jobs
  • -DitemFilter=(type=process_mr) to filter on all Big Data Map/Reduce and Spark Batch Jobs
  • -DitemFilter=(type=process_storm) to filter on all Big Data Storm and Spark Streaming Jobs
  • -DitemFilter=(type=route) to filter on all Routes

Example of filter on Job labels:

  • -DitemFilter=(type=process)and(label=jobA) to filter on one specific Job named jobA
  • -DitemFilter=(type=process)and(label%job*) to filter on Jobs whose names start with job

Example of filter on Job paths:

  • -DitemFilter=(type=process)and(path=Integration) to filter on Jobs located in a subfolder Integration
  • -DitemFilter=(type=process)and(path%Integration*) to filter on Jobs located in subfolders with a name starting with Integration

Example of filter on Job versions:

  • -DitemFilter=(type=process)and(version=lastVersion) to filter on the latest version of your Jobs

You also have the possibility to remove old versions of your project items in the Project Settings of Talend Studio.

Example of filter on the person who created the Job:

-DitemFilter=(type=process)and(author=doc@talend.com) to filter on Jobs whose author ID is doc@talend.com

Example of exclusion filter:

  • -DitemFilter=(!path=sandbox)and(type=process)and(label%jobA*)or(label%jobB*) to filter on Jobs with a name starting with jobA or jobB, but that are not in the sandbox folder
  • -DitemFilter=(!path%MainProcess/Import*)and(type=process)and(label%job*) to filter on Jobs with a name starting with job, but that are not in the subfolders with a name starting with Import under the MainProcess folder
  • -DitemFilter=(type=process)and(!label=job2)and(path%MainProcess/Export*) to filter on Jobs located in the subfolders with a name starting with Export under the MainProcess folder, and with the exception of the Job named job2
-DforceMavenOnline In case of dependency error when building or scheduling the execution of Jobs in offline mode, allows the Studio to download dependencies according to what is defined in your Maven settings file.

The error you may get is the following:

Plugin X.Y.Z or one of its dependencies could not be resolved: Cannot access central

It can also be avoided by re-generating all POM files before build time as these POM files would then contain all the dependencies needed to execute your Jobs.

-Dworkspace Allows you to define the current workspace project for which you want to build artifacts.

It can be used with different Maven builder commands like org.talend.ci:builder-maven-plugin:8.0.X:detectCVE or org.talend.ci:builder-maven-plugin:8.0.X:generateAllPoms for example.

Example:
-Dworkspace=/home/talend/talend_studio/CI/workspace
Custom component parameters -Dcomponents.* When deploying Jobs that use custom components to an artifact repository, specifies the repository in which the component libraries are stored.

This is only required for the custom components created using Talend Component Kit.

For more information, read this documentation.

Example:
-Dcomponents.nexus.user=<user>
-Dcomponents.nexus.password=<password>
-Dcomponents.nexus.url=http://localhost:8081/
-Dcomponents.nexus.repository=talend-custom-libs-release
-Dcomponents.nexus.repository.snapshot=talend-custom-libs-snapshot
Note: For older javajet-based components, refer to this guide.
Artifact repository parameter -DaltDeploymentRepository When deploying to an artifact repository (Nexus, Artifactory), specifies an alternative repository to which the project artifacts should be deployed.

Format: id::layout::url

Example:
-DaltDeploymentRepository=snapshots::default::http://localhost:8081/repository/snapshots/
Talend Cloud parameters -Pcloud-publisher Activates the Talend Cloud publisher plugin that allows you to publish to Talend Cloud.
-Dservice.url Specifies the Talend Cloud account in which the project artifacts should be deployed.

In the default script provided by Talend, the value corresponds to the ${CLOUD_URL} variable that is defined at the beginning of the script.

-Dcloud.token Specifies the access token generated for your Talend Cloud account.

Personal access tokens are generated from Talend Cloud Management Console

-Dcloud.publisher.environment Specifies the environment created in your Talend Cloud account in which the project artifacts should be deployed.
Example:
-Dcloud.publisher.environment=PROD
-Dcloud.publisher.workspace Specifies the workspace created in your Talend Cloud account in which the project artifacts should be deployed.
Example:
-Dcloud.publisher.workspace=CI_WORKSPACE
-Dcloud.publisher.screenshot Add this parameter with a true value if you want to publish the capture of the artifact to be deployed in your Talend Cloud account.
Example:
-Dcloud.publisher.screenshot=true
Docker parameters -Djkube.docker.push.registry Specifies the Docker registry where you want to share your image.
-Djkube.docker.username Specifies the Docker user corresponding to the account that will log in to the Docker registry.

In the default script provided by Talend, the value corresponds to the ${env.Dusername} variable, that is the Docker user defined in the credentials stored in Jenkins.

-Djkube.docker.password Specifies the Docker password corresponding to the account that will log in to the Docker registry.

In the default script provided by Talend, the value corresponds to the ${env.Dpassword} variable, that is the Docker password defined in the credentials stored in Jenkins.

-Djkube.docker.host Specifies the hostname corresponding to the account that will log in to the Docker registry.
-Dtalend.project.branch Specifies the tag and related tagged artifacts you want to build.
Example:
-Dtalend.project.branch=tags/mytag
-Djvm.arguments Specifies JVM arguments if needed, for example, to increase the memory available to Talend CommandLine.

If you have installed the 8.0 R2022-07 Studio monthly update or a later one provided by Talend, the JVM arguments are also applied to the Eclipse p2 director application that is used for installing Talend CommandLine. For more information, see Installing software using the p2 director application.

Example:
-Djvm.arguments=-Xms512m -Xmx4000m
-Dcodes.compile.failOnError To prevent the CI build to fail when encountering some routines with compilation errors that are not used in the artifacts you are building, add this parameter with a false value to skip them and continue the compilation.
Example:
-Dcodes.compile.failOnError=false