Configuring Jenkins - 6.3

Talend Software Development Life Cycle Best Practices Guide

EnrichVersion
6.3
EnrichProdName
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Real-Time Big Data Platform
task
Administration and Monitoring
Deployment
Design and Development
EnrichPlatform
Talend Administration Center
Talend Artifact Repository
Talend CommandLine
Talend Repository Manager
Talend Studio

Creating the Jenkins projects

Prerequisites:

Before creating your build projects, check in the Jenkins configuration that:

  • the paths to your Jenkins, JDK (java 8 is needed), Git and Maven installation directories are properly set.

  • the credentials of your Subversion and or Git accounts are set and valid.

  1. Launch the Jenkins server.

  2. Create three Jenkins build projects (Maven projects):

    Project name (example)

    Goal

    GenerateSources

    Gets the sources (potentially at each commit) and converts them to Java classes with the Talend CI Builder. This project calls the CommandLine application that generates all the sources of the Jobs and Tests you have created in your project. See Configuring Jenkins to generate the sources.

    RunTests

    Project triggered at the end of the build project GenerateSources that runs all Test Cases at once. See Configuring Jenkins to run all tests .

    DeployToNexus

    Project triggered at the end of the build project RunTests that deploys the Jobs on the Nexus repository of your choice. See Configuring Jenkins to deploy Jobs on Nexus.

Next: Configuring Jenkins to generate the sources

Configuring Jenkins to generate the sources

In order to generate the sources of your Jobs and Tests, you need to:

  • link the BuildSources Jenkins project to the SVN/Git project where your sources are stored. See Link the Jenkins project to the SVN/Git project.

  • decide how to generate your sources:

    • Recommended: Locally (script mode): generate the sources of your Jobs and Tests locally without having to keep the CommandLine up and running during the build execution, as the CommandLine is called, used and shut down. It allows you to run several builds at the same time based on the same CommandLine installation, however you need to use a different workspace for each build.

    • Alternative: remotely (server mode): generate the sources of your Jobs and Tests remotely using the CommandLine which needs to be launched (often installed as a service).

      See Create the POM file to generate sources.

  • specify the parameters to generate sources in the BuildSources Jenkins project. See Specify the parameters to generate the sources.

Link the Jenkins project to the SVN/Git project
  • Link the Jenkins BuildSources project to the SVN or Git project in which you created your Test Cases. Note that several projects can be added.

    SVN only: Fill in the branch you want to retrieve (in this example, CI_PROJECT/trunk) and the base folder where you saved all your projects (here, projects) followed by a folder with the same name as the original technical name of the project (here, CI_PROJECT). In this way, the existing workspace folder will be completely replicated and the CommandLine application will be able to connect directly to the defined folder (projects) to find the projects.

    See below an example for a Git project.

Next: Create the POM file to generate sources

Create the POM file to generate sources
  1. Create a POM file (in XML format), called BuildJob_pom.xml for example, holding information on Maven project and its configuration and the instructions to generate your sources. It could look as follows (note that the parts in italics need to be modified):

    • Recommended: Local generation (script mode)

      When you choose to generate sources in script mode (local-generate), you do not need to launch the CommandLine as it will be launched and shut down automatically, but you do need to specify in this POM file:

      • properties linked to the CommandLine used to generate the sources:

        • <product.path>: path to CommandLine

        • <commandline.workspace>: path to the workspace which is the base folder where all the projects were checked out before

        • <projectsTargetDirectory>: folder where all the sources of your project will be generated

        • <deploy.version>: defines the version of the items to be published on Nexus and applies -SNAPSHOT tags to these items (can be Jobs, Routines, Test cases, etc.). This parameter affects the output result of the generation.

          Example of snapshot version: <deploy.version>1.0.0-SNAPSHOT</deploy.version>

          Example of release version: <deploy.version>1.0.0</deploy.version>

        Note that the only requirement is to store all projects in the workspace folder, this means you have the possibility create a Jenkins task that clones the projects and copy them (folders only) to the workspace.

      • the configuration parameters that allow you to customize the execution:

        • <itemFilter>: to filter generation on project items. This parameter allows you to build only specific Jobs:

          Example of filter on one specific Job with label job_dev:

          <itemFilter>(label=job_dev)</itemFilter>

          Example of filter on all Jobs located in subfolder called Integration :

          <itemFilter>(path=Integration)</itemFilter>

          Example of filter on all Jobs with TEST status:

          <itemFilter>(status=TEST)</itemFilter>

          Example of filter on all Jobs with name starting with jobExport, but that are not in the sandbox folder:

          <itemFilter>(!path=sandbox)and(label%jobExport*)</itemFilter>

        • <commandlineUser>: to define who is the CommandLine user

        • <jvmArguments>: to define the memory that will be allocated to the generation

      • the Nexus repository where the CI Builder plugin is stored: the parameter <pluginRepository> allows you to specify the URL of the Nexus thirdparty repository in which you have uploaded the CI Builder files.

      The Maven documentation of the CI Builder plugin is embedded in the CI Builder archive file, see the files in the <CIBuilder_home>/help folder for more information.

      <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
       <modelVersion>4.0.0</modelVersion>
      <groupId>org.talend</groupId>
      <artifactId>buildsources</artifactId>
      <version>A.B.C</version>
      
      <properties>
          <!-- Required. Commandline application workspace and Studio path, only for local(script) mode -->
          <commandline.workspace>commandline_workspace_path</commandline.workspace>   
          <product.path>studio_path</product.path>
          <!-- Optional. Specify target directory where generated sources will be stored -->
          <projectsTargetDirectory>${basedir}/projectSources</projectsTargetDirectory> 
          <!-- Optional. Specify version for the artifact to be built. Can be set for each Job independently -->
          <deploy.version>A.B.C-SNAPSHOT</deploy.version>
      </properties>
      
      <build>
          <plugins>
              <plugin>
                  <groupId>org.talend</groupId>
                  <artifactId>ci.builder</artifactId>
                  <version>A.B.C</version>
                  <executions>
                      <execution>
                          <phase>generate-sources</phase>
                          <goals>
                              <!-- local(script) mode -->
                              <goal>local-generate</goal>
                          </goals>
                          <configuration>
                              <!-- Optional. Specify CommandLine user -->
                              <commandlineUser>jobbuilder@talend.com</commandlineUser>
                              <!-- Optional. Jvm Parameters for local(script) mode -->
                              <!-- <jvmArguments>-ea -Xms512m -Xmx1300m -XX:+CMSClassUnloadingEnabled -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:+HeapDumpOnOutOfMemoryError</jvmArguments> -->
                              <!-- Optional. Parameter used to filter on specific Job status (TEST, DEV, PROD, etc)  -->
                              <!-- <itemFilter>(status=TEST)or(status=PROD)or(status="")</itemFilter> -->
                          </configuration>
                      </execution>
                  </executions>
              </plugin>
          </plugins>
      </build>
      
      <pluginRepositories>
          <!-- everything through Maven Central Proxy -->
              <pluginRepository>
                  <id>Central</id>
                  <name>Central</name>
                  <url>http://<host>:<port>/nexus/content/repositories/central</url>
              </pluginRepository>
              <pluginRepository>
                  <id>thirdparty</id>
                  <name>thirdparty</name>
                  <url>http://<host>:<port>/nexus/content/repositories/thirdparty/</url>
              </pluginRepository>
      </pluginRepositories>
      
      </project>
    • Alternative: Remote generation (server mode)

      When you choose to generate sources in server mode (generate), you need the CommandLine to be launched (the best practice would be to have it installed as a service) and you then need to specify its access parameters in the GenerateSources Jenkins project.

      <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
       <modelVersion>4.0.0</modelVersion>
      <groupId>org.talend</groupId>
      <artifactId>buildsources</artifactId>
      <version>A.B.C</version>
      <packaging>pom</packaging>
      
      <build>
          <plugins>
              <plugin>
                  <groupId>org.talend</groupId>
                  <artifactId>ci.builder</artifactId>
                  <version>A.B.C</version>
                  <executions>
                      <execution>
                          <phase>generate-sources</phase>
                          <goals>
                              <!-- server mode -->
                              <goal>generate</goal>
                          </goals>
                          <configuration>
                          </configuration>
                      </execution>
                  </executions>
              </plugin>
          </plugins>
      </build>
      
      <pluginRepositories>
          <!-- everything through Maven Central Proxy -->
              <pluginRepository>
                  <id>Central</id>
                  <name>Central</name>
                  <url>http://<host>:<port>/nexus/content/repositories/central</url>
              </pluginRepository>
              <pluginRepository>
                  <id>thirdparty</id>
                  <name>thirdparty</name>
                  <url>http://<host>:<port>/nexus/content/repositories/thirdparty/</url>
              </pluginRepository>
      </pluginRepositories>
      
      </project>

    For more information on Maven life cycle and goals, see the Apache Maven documentation.

  2. You also have the possibility to customize some of the information that are generated by default for your project from Talend Studio, through the menu Project Settings > Build > Maven > Default > Project: from here you can edit the default <groupId> that will be used when publishing on Nexus, the <version> as well as other default information. For more information, see the Talend Studio User Guide.

Next: Specify the parameters to generate the sources

Specify the parameters to generate the sources

Prerequisites:

In the BuildSources project configuration

  1. In the Root POM field of the Build area, set the path to the BuildJob_pom.xml file holding the instructions to generate sources.

  2. In the Goals and options field, enter the Maven instruction to generate sources :

    org.talend:ci.builder:6.3.2:<local->generate

    Use local-generate to generate sources in script mode (recommended), or use generate to generate sources remotely.

  3. (Optional) If you want to perform a non-local generation (not recommended), set the other necessary Maven parameters in the MAVEN_OPTS field of the Advanced part of the Build area (plugin execution information as well as parameters needed to generate the sources).

    Note that the Maven documentation of the CI Builder plugin is embedded in the CI Builder archive file, see the files in the <CIBuilder_home>/help folder for more information.

    Example :

    Field

    Parameters

    MAVEN_OPTS

    -Dcommandline.workspace=<jenkins_path>/workspace/BuildSources/projects
    -Dcommandline.host=localhost
    -Dcommandline.port=8002
    -Dcommandline.user=xxx@talend.com
    -DprojectsTargetDirectory=<jenkins_path>/workspace/RunTests/projectSources
    -DitemFilter=(status=TEST)

    For more information on these parameters, see Create the POM file to generate sources.

    Note that the only requirement is to store all projects in the workspace folder, this means you have the possibility create a Jenkins task that clones the projects and copy them (folders only) to the workspace.

  4. If your project is on Git, you need to specify the folder in which your projects will be checked out (Use custom workspace) in the Advanced part of the Build area. This path needs to be the same as the one you will use in your CommandLine (see Executing the Tests).

    Example: <jenkins_path>/workspace/BuildSources/projects

    You also have the possibility to check out one specific project in your repository by entering its name in Additional Behaviours > Sparse Checkout paths.

    Example: CI_PROJECT

  5. In the Advanced part of the Build area, select Settings file in filesystem in the Settings file list and point to the Maven settings file of the Studio with all the dependencies needed to generate the sources.

    Example: <studio_path>/configuration/maven_user_settings.xml

  6. (Optional) Add a post-build action to get it to run the next project, the RunTests project, that will run all the tests.

Next: Configuring Jenkins to run all tests

Configuring Jenkins to run all tests

Prerequisite: you have created and configured the GenerateSources Jenkins project to retrieve the sources of your Jobs and Tests, see Creating the Jenkins projects and Configuring Jenkins to generate the sources.

In the RunTests project configuration

  1. In the Root POM field of the Build area, set the path to the <jenkins_workspace>/RunTests/projectSources/pom.xml file generated previously.

  2. In the Goals and options field, enter the Maven instruction to run all Tests : test -fn -e

    This Jenkins project retrieves the POM file generated during source generation and use it to run all available Test Cases.

  3. In the Advanced part of the Build area, select Settings file in filesystem in the Settings file list in order to point to the Maven settings file of the Studio with all the jar dependencies needed to generate the sources.

  4. (Optional) Add a post-build action to trigger the DeployToNexus build.

    Note that if you add this step, artifacts will be deployed automatically to Nexus after Test executions following default instructions (default groupID, default version, etc.) held in the source POM file. These default values can be edited through the Studio project settings before generating the sources and their corresponding POM file.

Next: Configuring Jenkins to deploy Jobs on Nexus

Configuring Jenkins to deploy Jobs on Nexus

Prerequisites: You have generated your sources and have run all corresponding Tests, see Configuring Jenkins to generate the sources and Configuring Jenkins to run all tests.

In the DeployToNexus project configuration

  1. In the Root POM field of the Build area, set the path to the <jenkins_workspace>/RunTests/projectSources/pom.xml file generated previously.

  2. In the Goals and options field, enter the Maven instruction to deploy all the Jobs available into the Nexus artifact repository of your choice : deploy -fn -e

    This project deploys artifacts on Nexus by using the instructions held in the POM file.