Talend custom Maven build options - Cloud

Talend Software Development Life Cycle for Cloud - Best Practices Guide

EnrichVersion
Cloud
EnrichProdName
Talend Cloud
EnrichPlatform
Talend Artifact Repository
Talend Management Console
Talend Studio
task
Administration and Monitoring
Deployment
Design and Development
Maven options you can pass when building, executing and publishing artifacts on your Continuous Integration server.
Talend leverages a standard Maven-based Continuous Integration (CI) implementation. A CI Maven command typically looks like this:
mvn \
        -f <project name>/poms/pom.xml \
        -am -pl <comma separated list of jobs modules> \
        -Dgeneration.type=local
        -Dlicense.path=<local path or url> \
        -Dupdatesite.path=<local path> \
        -Dpatch.path=<local path or url>
        clean deploy
Example:
mvn -f C:/CI_CD/MYPROJECT/poms/pom.xml clean deploy -fae -e -pl jobs/process/priorTest_0.1 -am -Dlicense.path=C:/CI_CD/license -Dupdatesite.path=http://localhost:8080/P2/ -Dpatch.path=D:/patches/Patch_TPS-2222.zip -Dproject.distributionManagement.snapshotRepository.url=http://localhost:8081/repository/snapshots/ -Dproject.distributionManagement.snapshotRepository.id=snapshots -DaltDeploymentRepository=snapshots::default::http://localhost:8081/repository/snapshots/

Talend-related Maven parameters

Maven parameter Value
Talend CommandLine parameters -Dupdatesite.path URL address of your server where the P2 repository with Talend CommandLine can be downloaded and installed at build time. This site also contains the Maven plugins and all the external libraries necessary to build your pipelines. For an example of how to embed the P2 repository in your Tomcat application, see Before scheduling the execution of your artifacts.
Example:
-Dupdatesite.path=http://localhost:8080/P2/
Warning: This parameter only applies to users who have the P2 update site archive, which is provided by Talend.

-DforceUpdate

-Dinstaller.clean

-U

Parameters needed when installing updates from the update site:
  • -DforceUpdate forces the installation of Talend CommandLine from the update site, even if a local installation is present.
    Example:
    -DforceUpdate=true
  • -Dinstaller.clean re-downloads all plugins necessary to execute your build successfully.
    Example:
    -Dinstaller.clean=true
  • -U (with no values) re-downloads all JAR files necessary to execute your build successfully.
-Dpatch.path Path to the patch you have installed or want to install (Talend Studio p2 patch, manual patch, component patch, etc.).
Example if you have installed the patch locally:
-Dpatch.path=D:/patches/Patch_TPS-2222.zip,D:/patches/Patch_TPS-3333.zip
Example of patch to download:
-Dpatch.path=http://localhost:8085/nexus/content/repositories/releases/Patch_TPS-2222.zip,http://localhost:8085/nexus/content/repositories/releases/Patch_TPS-3333.zip

If credentials are needed to access the remote patch location, use the -Dpatch.remote.user and -Dpatch.remote.password parameters. The patch.remote.password parameter can set a password encrypted by Maven.

-Dgeneration.type Type local as the value to generate the sources of your artifacts and Tests locally without having to keep the Talend CommandLine up and running during the build execution.
Example:
-Dgeneration.type=local

The server generation type is deprecated.

-Dlicense.path Path to the license of your Talend product.

If the licence path value is a URL and credentials are needed to access it, use the -Dlicense.remote.user and -Dlicense.remote.password parameters.

-Dcommandline.skip Type true as the value to skip the source code generation of your artifacts. This parameter can only be used if you have previously generated your artifacts using Talend CommandLine.
Example:
-Dcommandline.skip=true
-DprojectFilter Allows you to build only specific projects.
Example of filter on all projects with a name starting with CI:
-DprojectFilter="CI.*"
-DitemFilter Allows you to build only specific project items.

Example of filter on Job types:

-DitemFilter=(type=process) to filter on all Standard Jobs of the project

-DitemFilter=(type=process_mr) to filter on all Big Data Map/Reduce and Spark Batch Jobs of the project

-DitemFilter=(type=process_storm) to filter on all Big Data Storm and Spark Streaming Jobs of the project

-DitemFilter=(type=route) to filter on all Routes of the project

Example of filter on Job labels:

-DitemFilter=(type=process_mr)and(label=job_ProcessWeatherData) to filter on one specific Big Data Map/Reduce Job named job_ProcessWeatherData

-DitemFilter=(type=process)and(label%job_dev*) to filter on Jobs which names start with job_dev

Example of filter on Job paths:

-DitemFilter=(type=process)and(path=Integration) to filter on Jobs located in subfolder called Integration

-DitemFilter=(type=process)and(path%Integration*) to filter on Jobs located in subfolders with a name starting with Integration

Example of filter on the person who created the Job:

-DitemFilter=(type=process_storm)and(author=rbunch@talend.com) to filter on Big Data Storm Streaming Jobs whose author ID is rbunch@talend.com

Example of exclusion filter:

-DitemFilter=(!path=sandbox)and(type=process)and(label%job_Export*)or(label%job_Monitor*) to filter on Jobs with a name starting with job_Export or job_Monitor, but that are not in the sandbox folder.

-DitemFilter=(!path%MainProcess/Import*)and(type=process)and(label%job_Export*) to filter on Jobs with a name starting with job_Export, but that are not in the subfolders with a name starting with Import under the MainProcess folder.

Example of filter applied in order to execute Big Data Spark Batch Jobs located in the subfolders with a name starting with Export under the MainProcess folder, and with the exception of the Job named job_batch_feature22:

-DitemFilter=(type=process_mr)and(!label=job_batch_feature22)and(path%MainProcess/Export*)

-Dtalend.job.context and -Dtalend.job.applyContextToChildren When building or scheduling the execution of Jobs, specifies the context group to apply to all Jobs at build time.
Example to apply a context group named dev only to the parent Jobs and apply the default context group to the child Jobs:
-Dtalend.job.context=dev 
Example to apply a context group named dev to the parent Jobs and apply the same context group to the child Jobs:
-Dtalend.job.context=dev -Dtalend.job.applyContextToChildren=true 

If the context group you specify does not exist on the Job, the default context group will be applied.

Custom component parameters -Dcomponents.* When deploying Jobs that use custom components to an artifact repository, specifies the repository in which the component libraries are stored.
Example:
-Dcomponents.nexus.user=<user> 
-Dcomponents.nexus.password=<password> 
-Dcomponents.nexus.url=http://localhost:8081/
-Dcomponents.nexus.repository=talend-custom-libs-release 
-Dcomponents.nexus.repository.snapshot=talend-custom-libs-snapshot
Artifact repository parameter -DaltDeploymentRepository When deploying to an artifact repository (Nexus, Artifactory), specifies an alternative repository to which the project artifacts should be deployed.

Format: id::layout::url

Example:
-DaltDeploymentRepository=snapshots::default::http://localhost:8081/repository/snapshots/
Talend Cloud parameters -Dservice.url Specifies the Talend Cloud account in which the project artifacts should be deployed.

In the default script provided by Talend, the value corresponds to the ${CLOUD_URL} variable that is defined at the beginning of the script.

-Dcloud.publisher.environment Specifies the environment created in your Talend Cloud account in which the project artifacts should be deployed.
Example:
-Dcloud.publisher.environment=PROD
-Dcloud.publisher.workspace Specifies the workspace created in your Talend Cloud account in which the project artifacts should be deployed.
Example:
-Dcloud.publisher.workspace=CI_WORKSPACE
-Dcloud.publisher.screenshot Add this parameter with a true value if you want to publish the capture of the artifact to be deployed in your Talend Cloud account.
Example:
-Dcloud.publisher.screenshot=true
Docker parameters -Ddocker.push.registry Specifies the Docker registry where you want to share your image.
-Ddocker.push.username Specifies the Docker user corresponding to the account that will log in to the Docker registry.

In the default script provided by Talend, the value corresponds to the ${env.Dusername} variable, that is the Docker user defined in the credentials stored in Jenkins.

-Ddocker.push.password Specifies the Docker password corresponding to the account that will log in to the Docker registry.

In the default script provided by Talend, the value corresponds to the ${env.Dpassword} variable, that is the Docker password defined in the credentials stored in Jenkins.

-Dtalend.docker.name Specifies the name of the Docker image that will be created.
-Dstudio.prometheus.metrics Add this parameter with a true value while publishing to Docker in order to expose Camel metrics to Prometheus and provide more details about the deployed Routes (number of processed exchanges, number of errors, requests per seconds, etc).
-Dtalend.project.branch Specifies the tag and related tagged artifacts you want to build.
Example:
-Dtalend.project.branch=tags/mytag
-Xms -Xmx JVM arguments that allow you to increase the memory available to Maven.
-Dstudio.talendDebug When trying to debug build issues with the support team for example, add this parameter with a true value to get additional logs.
Example:
-Dstudio.talendDebug=true
-Dcodes.compile.failOnError To prevent the CI build to fail when encountering some routines with compilation errors that are not used in the artifacts you are building, add this parameter with a false value to skip them and continue the compilation.
Example:
-Dcodes.compile.failOnError=false