Adding a JDBC driver to your engine - Cloud

Talend Remote Engine Gen2 Quick Start Guide

EnrichVersion
Cloud
EnrichProdName
Talend Cloud
EnrichPlatform
Talend Management Console
Talend Pipeline Designer
task
Deployment > Deploying > Executing Pipelines
Installation and Upgrade

You have the possibility to add a JDBC driver to the Remote Engine Gen2 in order to connect to databases that are not explicitly supported in Talend Cloud Data Inventory.

The generated SQL statements will be SQL-92 compatible.

Procedure

  1. Go to the Remote Engine Gen2 installation directory.
  2. Edit the following file that is the Docker configuration file used when starting the engine:
    default/docker-compose.yml if you are using the engine in the AWS USA, AWS Europe, AWS Asia-Pacific or Azure regions.

    eap/docker-compose.yml if you are using the engine as part of the Early Adopter Program.

  3. Find the livy container and add this JDBC driver configuration under the environment section:
    livy:
        image: ${LIVY_IMG}
        networks:
          default:
            aliases:
              - livy
        environment:
          LIVY_JOB_SPARK_SUBMIT_OPTS: "-Djdbc.drivers[10].id=<DATABASE_ID> -Djdbc.drivers[10].displayName=<DISPLAY_NAME> -Djdbc.drivers[10].order=<ORDER_IN_THE_LIST> -Djdbc.drivers[10].className=<JDBC_DRIVER_CLASS> -Djdbc.drivers[10].paths[0]=<MAVEN GROUP>:<MAVEN ARTIFACT>:jar:<MAVEN VERSION>"
          [...]

    where <DATABASE_ID> (the database identifier), <DISPLAY_NAME> (the database display name), <ORDER_IN_THE_LIST> (the position of the database in the list of connections in the app) , <JDBC_DRIVER_CLASS> (the database JDBC driver class) and <MAVEN GROUP>:<MAVEN ARTIFACT>:jar:<MAVEN VERSION> (the database artifact) need to be replaced with your own database information.

    Example

    For a DB2 database
    livy:
        image: ${LIVY_IMG}
        networks:
          default:
            aliases:
              - livy
        environment:
    		LIVY_JOB_SPARK_SUBMIT_OPTS: "-Djdbc.drivers[10].id=DB2 -Djdbc.drivers[10].displayName=DB2 -Djdbc.drivers[10].order=900 -Djdbc.drivers[10].className=com.ibm.db2.jcc.DB2Driver -Djdbc.drivers[10].paths[0]=com.ibm.db2:jcc:jar:11.5.0.0"
    		[...]
  4. Find the component-server container and add this configuration under the environment/_JAVA_OPTIONS section to add the JDBC driver to the list of connectors and to be able to call the corresponding services:
    component-server:
        image: ${COMPONENTS_IMG}
        networks:
          default:
            aliases:
              - component-server
        environment:
          [...]
          _JAVA_OPTIONS: -Xmx${COMPONENTS_MEM} -Dhttp=443
    		[...@=]
    		-Djdbc.drivers[10].id=<DATABASE_ID>
    		-Djdbc.drivers[10].displayName=<DISPLAY_NAME>
    		-Djdbc.drivers[10].order=<ORDER_IN_THE_LIST>
    		-Djdbc.drivers[10].className=<JDBC_DRIVER_CLASS>
    		-Djdbc.drivers[10].paths[0]=<MAVEN GROUP>:<MAVEN ARTIFACT>:jar:<MAVEN VERSION>
    		[...]

    Example

    For a DB2 database
    component-server:
        image: ${COMPONENTS_IMG}
        networks:
          default:
            aliases:
              - component-server
        environment:
          [...]
          _JAVA_OPTIONS: -Xmx${COMPONENTS_MEM} -Dhttp=443
            [...@=]
            -Djdbc.drivers[10].id=DB2
            -Djdbc.drivers[10].displayName=DB2
            -Djdbc.drivers[10].order=900
            -Djdbc.drivers[10].className=com.ibm.db2.jcc.DB2Driver
            -Djdbc.drivers[10].paths[0]=com.ibm.db2:jcc:jar:11.5.0.0
    		[...]
  5. Find the connectors container image and add this configuration under the environment section to make it download the JDBC driver library:
    connectors:
        image: talend/connectors:20200528132950116
        networks:
          - default
        environment:
          CONNECTORS_SETUP_OPTS: setup --component-jdbc-auto-download-drivers
            [...]
            --artifact=<MAVEN GROUP>:<MAVEN ARTIFACT>:jar:<MAVEN VERSION>
            [...]

    Example

    For a DB2 database
    connectors:
        image: talend/connectors:20200528132950116
        networks:
          - default
        environment:
          CONNECTORS_SETUP_OPTS: setup --component-jdbc-auto-download-drivers
            [...]
            --artifact=com.ibm.db2:jcc:jar:11.5.0.0
    		[...]
  6. Find the previewrunner container and configure it this way in order to make the data sample preview work from the dataset edition form in Talend Cloud Data Inventory:
    previewrunner:
        image: ${PREVIEW_RUNNER_IMG}
        user: root
        command: /opt/talend/app/bin/data-streams-previewrunner-ee-svc -Djdbc.drivers[10].id=<DATABASE_ID>
    	-Djdbc.drivers[10].displayName=<DISPLAY_NAME> -Djdbc.drivers[10].order=<ORDER_IN_THE_LIST>
    	-Djdbc.drivers[10].className=<JDBC_DRIVER_CLASS> -Djdbc.drivers[10].paths[0]=<MAVEN GROUP>:<MAVEN ARTIFACT>:jar:<MAVEN VERSION>
    	-Dhttp.port=80 -Dlogger.resource=dss-logback.xml -Dtalend.component.manager.m2.repository=/opt/talend/connectors
    	-Dtalend.component.manager.classloader.container.classesAndPackages=org.talend.components.common.

    Example

    For a DB2 database
    previewrunner:
        image: ${PREVIEW_RUNNER_IMG}
        user: root
        command: /opt/talend/app/bin/data-streams-previewrunner-ee-svc -Djdbc.drivers[10].id=DB2
    	-Djdbc.drivers[10].displayName=DB2 -Djdbc.drivers[10].order=900
    	-Djdbc.drivers[10].className=com.ibm.db2.jcc.DB2Driver -Djdbc.drivers[10].paths[0]=com.ibm.db2:jcc:jar:11.5.0.0
    	-Dhttp.port=80 -Dlogger.resource=dss-logback.xml -Dtalend.component.manager.m2.repository=/opt/talend/connectors
    	-Dtalend.component.manager.classloader.container.classesAndPackages=org.talend.components.common.
  7. Save your changes and restart your engine.