Before scheduling the execution of your artifacts - Cloud - 7.3

Talend Software Development Life Cycle Best Practices Guide

Version
Cloud
7.3
Language
English
Product
Talend Big Data
Talend Big Data Platform
Talend Cloud
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Real-Time Big Data Platform
Module
Talend Administration Center
Talend Artifact Repository
Talend CommandLine
Talend JobServer
Talend Management Console
Talend Remote Engine
Talend Studio
Content
Administration and Monitoring
Deployment
Design and Development
Last publication date
2024-02-08
There are several prerequisites you should ensure before you start scheduling automatic artifact executions and deployment.
  • A Continuous Integration server/service is properly set up (here, Jenkins or Azure DevOps). For more information, see the Jenkins website and the Azure DevOps website.
  • You have previously created Test Cases in your project that you want to execute automatically. For more information, see Designing Tests.

    Note: When building and deploying your project, Test Cases will be generated as JUnit files and thus will be built before the packaging Maven phase.
  • The external libraries stored in Nexus that are needed to execute your Jobs have been properly installed in this project and your Nexus instance is started.

    If you are using Talend Administration Center, you have configured it to retrieve the external modules from Nexus. See the Talend Administration Center User Guide for more information.

  • You have a Docker environment installed and set up to push Docker images to your Docker registry. For more information, see the Docker website. Note that you can install a Docker container registry inside your Azure
  • If you are using Jenkins, you also need to check that a version superior or equal to Apache Maven 3 is installed in your server. For more information, see the Apache Maven website.

  • In order to display Git information (author, commit ID, commit date) in Talend Management Console when publishing artifacts using Continuous Integration builds, you need to set up your Git project repository with this hierarchy and copy (not move) every project to the ci-workspace folder.
    ci-workspace
        |- .repositories
             |- <any_unique_name>
                  |- .git
                  |- MAINPROJ1
             |- <any_unique_name>
                  |- .git
                  |- REFPROJ1
             |- <any_unique_name>
                  |- .git
                  |- REFPROJ2
                  |- REFPROJ3
        |- MAINPROJ1
        |- REFPROJ1
        |- REFPROJ2
        |- REFPROJ3
  • If you are using CI to build artifacts that require the Talend Data Mapper X12 HIPAA or HL7v2 packages, you need to:
    • extract the Talend_Full_Studio_p2_repository--YYYYYYYY_YYYY-VA.B.C.zip archive into <folder>/org.talend.transform.standard.p2.repository-A.B.C
    • install these packages in Talend CommandLine prior to building the Jobs using this command:
      Talend-Studio-win-x86_64.exe -nosplash -consoleLog -application org.eclipse.equinox.p2.director 
      -repository file:///<folder>/org.talend.transform.standard.p2.repository-A.B.C 
      -installIU org.talend.transform.standard.x12.hipaa.5010.feature.feature.group
Warning: The following prerequisites only apply to users who have the P2 update site archive, which is provided by Talend.

Starting from version 7.2 onwards, Talend CommandLine is automatically downloaded from the Talend update site and installed during build time to generate and execute your project artifacts. For more information, read Talend CI zero install workflow.

To download Talend CommandLine automatically at build time, you need to host the archive provided by Talend on your local HTTP server. For example, you can:
  • extract the Talend_Full_Studio_p2_repository--YYYYYYYY_YYYY-VA.B.C.zip archive in the webapp folder of Tomcat and rename it P2.
  • start Tomcat.
  • point to http://localhost:8080/P2/ when defining the -Dupdatesite.path parameter at build time. Replace <localhost> with the IP address or the hostname of the web server if the web browser IP is different from the machine you are on and replace <8080> with the Tomcat port you use.

Talend does not recommend running more than one Talend CommandLine at any given time on a machine. The easiest way to prevent this is to limit the number of executors on the servers to 1.

To summarize, you need the following applications: a Continuous Integration server or service (here, Azure or Jenkins), Git, and a server to host the P2 update site where the Talend CommandLine as well as other plugins necessary to build your artifacts are stored. Depending on where you want to deploy your artifacts, you may need access to an artifact repository (Nexus, Artifactory), Docker, or Talend Cloud.