Skip to main content Skip to complementary content

Setting up Data Planes on a registered Kubernetes cluster

Availability-noteBeta

Register one of your existing Kubernetes clusters for the deployment of Talend Data Plane environments.

You need to provide one of your own Kubernetes clusters for Talend to deploy Data Plane environments on it. This Kubernetes cluster must be exclusively dedicated to Talend Data Planes.

If you want to first familiarize yourself with Talend Control Plane and Data Plane before registering your cluster, see Setting up a demo Data Plane for test purposes to set up a demo Data Plane for test purposes.

Before you begin

  • Generate access tokens:

    Once generated, a service account token expires after 30 minutes. If it expires, generate a new token using the POST method at the endpoint https://api.<env>.cloud.talend.com/security/oauth/token. For more information about generating a token, see Generating a service account token.

  • The Kubernetes cluster must be one of the following versions:
    • v1.25
    • v1.26
    • v1.27
  • Minimum requirements for the Kubernetes cluster to be registered:
    • Instance size: 16 GB of memory and 4 vCPU
    • Disk size: 100 GB
    • Number of nodes: 3
    • Longhorn v1.5.1, a distributed block storage system for Kubernetes, has been installed on your cluster. You can use the following command to quickly install a operational Longhorn for a quick start:
      kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.5.1/deploy/prerequisite/longhorn-iscsi-installation.yaml
      If you need a comprehensive production-ready installation and configuration of Longhorn, see instructions from the Longhorn documentation.

Procedure

  1. If you do not have any Kubernetes cluster yet, create one either locally or on a Cloud platform such as Azure or AWS.
  2. Create a Data Plane entity directly in Talend Management Console or using its API service.
    • Talend Management Console: see Adding a Data Plane. When doing this, do not toggle the Install Data Plane as a single package including Kubernetes cluster option on to ensure obtaining the command used for only registering your Kubernetes cluster.
    • API: send the following POST call to https://api.<env>.cloud.talend.com/processing/data-planes.
      curl -i -X POST \
       -H "Content-Type:application/json" \ 
       -H "Authorization:Bearer <personal_access_token_or_service_access_token>" \ 
       -d \ 
      '{ 
       "provider": "REGISTER",
       "name": "a_meaningful_demo_data_plane_name", 
       "description": "A meaningful description."
      }' \
      'https://api.<ENV>.cloud.talend.com/processing/data-planes'

      The provider field uses REGISTER. This generates a command used to register an existing Kubernetes cluster only.

      With this request, a RKE2 Kubernetes distribution is automatically installed on your cluster. For further information about a RKE2 distribution, see https://docs.rke2.io.

      For further information about this endpoint, see Create a new Data Plane.

    In both approaches, you receive a command to be used to register the cluster to Talend for the deployment of Talend Data Plane environments. Save this command for later use; it's only visible once, so ensure you have it available when needed.
    You also receive the ID of the Data Plane in the response. Note it down so that you can use it in later steps.
    • If you use Talend Management Console to create your Data Plane, you can find its ID in the URL when you click this Data Plane. For example, in this URL
      <Your_TMC_URL>/processing/data-planes/c-m-nt4l5zx8/overview
      The c-m-nt4l5zx8 corresponds to the ID of the selected Data Plane.
  3. Run this command in your Kubernetes cluster to register it to Talend Cloud. This may take 2 to 3 minutes.
  4. Optional: Send the following API request to verify that the cluster node has been discovered by the Data Plane controller.
    curl -i -X GET \
       -H "Authorization:Bearer <personal_access_token_or_service_access_token>" \
     'https://api.<env>.cloud.talend.com/processing/data-planes/<data_plane_id_you_obtained>'
    When you see the node is listed in the nodes field in the response, it means the Data Plane controller has successfully discovered your node. The node is ready to have a Data Plane environment deployed.

    For further information about this endpoint, see Get a Data Plane by its ID.

  5. Create the Data Plane environment to be connected to the Data Plane entity created above.
    A Data Plane environment is a Talend environment specific for Data Planes, scoping and managing processing resources such as services, tasks, or artifacts as a whole. You need to use Talend Management Console or API to create a Data Plane environment.

    The Environments - Manage permission (ID: TMC_ENVIRONMENT_MANAGEMENT) is required.

    Example

    • Talend Management Console: see Adding a Data Plane environment.
    • API: send the following request:
      curl -i -X POST \
       -H "Content-Type:application/json" \ 
       -H "Authorization:Bearer <personal_access_token_or_service_access_token>" \ 
       -d \ 
      '{ 
       "name": "a_meaningful_data_plane_environment_name", 
       "description": "A meaningful description.",
       "workspaceName": 'the_name_of_the_workspace_to_be_created_and_associtated'"
      }' \
      'https://api.<ENV>.cloud.talend.com/orchestration/dataplane-environments'

      For further information about this endpoint, see Create a Data Plane environment.

  6. Connect the Data Plane entity to the Data Plane environment to be used for task or plan executions.

    Use Talend Management Console or the following request to create the connection. This request also deploys that Data Plane environment to the demo Kubernetes cluster.

    The Engines - Manage permission (ID: TMC_CLUSTER_MANAGEMENT) is required.

Results

The Talend Data Plane has been successfully deployed in your Kubernetes and connected with Talend Control Plane, ready to be used to run your Job tasks as you usually do on the standard engines. For details about how to add the task to the Data Plane environment for a run, see Adding tasks to a Data Plane environment.

If you need to know what has been installed in your Kubernetes cluster, go to your Kubernetes machine and run the commands outlined in this official Kubernetes documentation.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!