Adapting the script parameters to your environment - Cloud

Talend Software Development Life Cycle for Cloud - Best Practices Guide

EnrichVersion
Cloud
EnrichProdName
Talend Cloud
EnrichPlatform
Talend Artifact Repository
Talend Management Console
Talend Studio
task
Administration and Monitoring
Deployment
Design and Development

Configure the pipeline script provided by Talend to generate the project items according to the Maven phase you want to achieve and the Talend Cloud account or the repository you want to deploy your artifacts to.

Some parameters come from the settings file you have previously defined and the credentials you have stored in Jenkins. For more information about the Talend Maven parameters you can use when configuring your pipeline, see Talend custom Maven build options.

Procedure

  1. Open the Talend_Pipeline configuration page and go to the Script editor.
  2. Set your own values for the environment variables defined at the top of the script (project name, Job name/version/type, repository URL, etc.).
    Tip: Look for the XXX references in the script provided by Talend as an example and replace them with values corresponding to your environment.

    The script can be found in the Downloads tab of this page on Talend Help Center.

    Example

    Example of a project named CICD, a Job named job_feature956 with a 0.1 version and a Docker image named cicd4talend:
    env.PROJECT_GIT_NAME = 'CICD'
    env.PROJECT_NAME = env.PROJECT_GIT_NAME.toLowerCase()
    env.ARTIFACT = 'job_feature956'
    env.VERSION = '0.1'
    env.GIT_URL = 'https://github.com/talend/myrepo.git'
    env.CLOUD_URL = 'https://tmc.us.cloud.talend.com/inventory/'
    env.TYPE = "" // if big data = _mr
    env.UPDATESITE_URL 'http://localhost:8080/P2/'
    env.IMAGE_NAME = 'cd4talend'
    env.GIT_CREDENTIALS_ID = 'git'
    env.DOCKER_CREDENTIALS_ID = 'docker'

    If you are using a referenced project, you might need to adapt your pipeline script accordingly.

    Warning: Syntax requirements:
    • The value of the env.PROJECT_GIT_NAME variable must be in upper case otherwise it might cause build failure. If you used special characters in the project name or have any doubts on its final syntax you can check it in the parent pom.xml file of your project (talend.project.name variable in <project>/poms/pom.xml).
    • The value of the env.ARTIFACT variable must be in lower case even if the original name of your Job/Route/Service contained upper case as all artifacts are in lower case in the .pom files required to launch the build.
    • Do not add manually the "-SNAPSHOT" suffix to the version value. Snapshots and releases are decoupled from the version itself and must only be handled via the Studio preferences, see Changing the deployment version of each artifact at once.
  3. To define the Talend Cloud workspace and environment in which you want to deploy the project artifacts: edit the values stated in the mavenOpts line of the 'Build, Test and Publish artifacts to Cloud' stage.

    Example

    • mavenOpts: '-Dlicense.path=/home/talend/talend_studio/license -Dupdatesite.path=${UPDATESITE_URL} -Dservice.url=${CLOUD_URL} -Dcloud.publisher.environment=PROD
      -Dcloud.publisher.workspace=CI_WORKSPACE -Dcloud.publisher.screenshot=true'

      (Unix) allows you to deploy artifacts (with screenshots) into a Talend Cloud account inside a workspace named CI_WORKSPACE on a PROD environment. You need to specify your own Talend Cloud environment and workspace.

    Note:
    • If you have enabled Single Sign-On (SSO) to connect to Talend Cloud, you need to add the -Dcloud.token=<MyAccessToken> parameter to these commands.
    • The artifact published to Talend Cloud cannot be larger than 400MB.
  4. To define the Maven phase you want to achieve: make sure the phases stated in the 'Build, Test and Publish artifacts to artifact repository and 'Push to a Docker Registry' stages are the one you want to perform.
    For more information on the Maven phases, see the Maven documentation.
    Tip: All build profiles (-Pdocker, -Pnexus, etc.) are listed in the parent pom.xml file of your project. Some default parameters, like the docker image name or the Open JDK name, can also be overwritten in this file.

    Example

    • bat "mvn -f $PROJECT_GIT_NAME/poms/pom.xml clean -Pcloud-publisher deploy -fae -e -pl jobs/process${TYPE}/${ARTIFACT}_${VERSION} -am"
      (Windows) allows you to copy the final packaged artifacts to the Talend Cloud account of your choice.

      For examples of filters you can apply to the execution of your project items, see Filter the execution of your project on selected artifacts.

    • sh "mvn -f $PROJECT_GIT_NAME/poms/pom.xml deploy -Pdocker -fae -e -pl jobs/process${TYPE}/${ARTIFACT}_${VERSION} --context=Prod"
      (Unix) allows you to copy the final packaged artifacts to the Docker container of your choice, while changing the context environment to Prod (using --context). This environment must have been defined previously in Talend Studio, however if you did not define any context parameters you can use --context_param to define them at build time.
  5. To define the Docker registry in which you want to deploy the project artifacts: make sure that the Docker credentials stated in the mavenOpts line of the 'Push to a Docker registry' stage point to the credentials you have previously stored in Jenkins.

    Example

    mavenOpts: '-Dcommandline.skip=true -Dlicense.path=${LICENSE} -Dupdatesite.path=${UPDATESITE_URL} -Ddocker.push.registry=registry.hub.docker.com/${env.Dusername} -Ddocker.push.username=${env.Dusername} -Ddocker.push.password=${env.Dpassword} -Xms1024m -Xmx3096m')
    allows you to deploy artifacts as the Docker image you have previously defined in the Docker registry of your choice. You need to specify the Docker registry credentials refer to the credentials you have previously stored in Jenkins.
    Note: The -Dcommandline.skip=true parameter can only be used if you have previously generated your artifacts using Talend CommandLine.
  6. Click Save to take your changes into account.
Troubleshooting:

Troubleshooting:

If you get the following error while trying to publish ESB features:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy-file (deploy-file) on project XXX-feature:
*The parameters 'url' for goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy-file are missing or invalid*

Try adding these parameters to the Maven options for release artifacts:

-Dproject.distributionManagement.repository.url=http://localhost:8081/repository/releases/
-Dproject.distributionManagement.repository.id=releases

Or the following parameters for snapshot artifacts:

-Dproject.distributionManagement.snapshotRepository.url=http://localhost:8081/repository/snapshots/
-Dproject.distributionManagement.snapshotRepository.id=snapshots