Skip to main content Skip to complementary content

Setting up a demo Data Plane for test purposes

Availability-noteBeta

If you want to familiarize with the Data Plane processing before starting to implement full-scale Data Planes, you can quickly set up a demo Data Plane with a one-node cluster on your VM for easy access to Talend Control Plane and Data Plane resources.

This approach is optional and intended for test purposes only. It should not be used in a production environment.

Before you begin

  • Generate access tokens:

    Once generated, a service account token expires after 30 minutes. If it expires, generate a new token using the POST method at the endpoint https://api.<env>.cloud.talend.com/security/oauth/token. For more information about generating a token, see Generating a service account token.

  • Minimum system requirements for the VM to be used:
    • 16G of memory and 4 vCPU
    • Disk size: 50 GB
    • One of the following versions of Linux:
      • Ubuntu 22
      • Amazon Linux 2
      • Rocky Linux 9

Procedure

  1. Create a VM to be used for testing Control Plane and Data Plane.
    An online VM is recommended.
  2. Create a demo Data Plane entity called CUSTOM directly in Talend Management Console or using its API service.

    The Engines - Manage(ID: TMC_CLUSTER_MANAGEMENT) permission is required.

    • Talend Management Console: see Adding a Data Plane. When doing this, toggle the Install Data Plane as a single package including Kubernetes cluster option on to ensure obtaining the command used to create and register a new Kubernetes cluster on your VM.
    • API: send the following POST call to https://api.<env>.cloud.talend.com/processing/data-planes. For example,
      curl -i -X POST \
       -H "Content-Type:application/json" \ 
       -H "Authorization:Bearer <personal_access_token_or_service_access_token>" \ 
       -d \ 
      '{ 
       "provider": "CUSTOM",
       "name": "a_meaningful_demo_data_plane_name", 
       "description": "A meaningful description.",
       "oneNodeDemo": true, 
       "distribution": "rke2"
      }' \
      'https://api.<ENV>.cloud.talend.com/processing/data-planes'
      The provider field uses CUSTOM. This generates a command used to create and register a new Kubernetes cluster. The supported cluster types are rke2 and k3s, defined in the distribution field.

      For further information about this endpoint, see Create a new Data Plane.

    In both approaches, you receive a command to be used to install a one-node Kubernetes cluster on your VM and register it to Talend for the deployment of Talend Data Plane environments. Save this command for later use; it's only visible once, so ensure you have it available when needed.
    You also receive the ID of the Data Plane in the response. Note it down so that you can use it in later steps.
    • If you use Talend Management Console to create your Data Plane, you can find its ID in the URL when you click this Data Plane. For example, in this URL
      <Your_TMC_URL>/processing/data-planes/c-m-nt4l5zx8/overview
      The c-m-nt4l5zx8 corresponds to the ID of the selected Data Plane.
  3. Run the command to install and register this Kubernetes cluster.
  4. Optional: Send the following API request to verify that the cluster node has been discovered by the Data Plane controller.
    curl -i -X GET \
       -H "Authorization:Bearer <personal_access_token_or_service_access_token>" \
     'https://api.<env>.cloud.talend.com/processing/data-planes/<data_plane_id_you_obtained>'
    When you see the node is listed in the nodes field in the response, it means the Data Plane controller has successfully discovered your node. The node is ready to have a Data Plane environment deployed.

    For further information about this endpoint, see Get a Data Plane by its ID.

  5. Create the Data Plane environment to be connected to the Data Plane entity created above.
    A Data Plane environment is a Talend environment specific for Data Planes, scoping and managing processing resources such as services, tasks, or artifacts as a whole. You need to use Talend Management Console or this API endpoint to create a Data Plane environment.

    Example

    • Talend Management Console: see Adding a Data Plane environment.
    • API: send the following request:
      curl -i -X POST \
                           -H "Content-Type:application/json" \ 
                           -H "Authorization:Bearer <personal_access_token_or_service_access_token>" \ 
                           -d \ 
                           '{ 
                           "name": "a_meaningful_data_plane_environment_name", 
                           "description": "A meaningful description.",
                           "workspaceName": 'the_name_of_the_workspace_to_be_created_and_associtated'"
                           }' \
                           'https://api.<ENV>.cloud.talend.com/orchestration/dataplane-environments'

    The Environments - Manage permission (ID: TMC_ENVIRONMENT_MANAGEMENT) is required.

  6. Connect the demo Data Plane entity to the Data Plane environment to be used for task or plan executions.

    Use the following request to create the connection. This request also deploys that Data Plane environment to the demo Kubernetes cluster.

    The Engines - Manage permission (ID: TMC_CLUSTER_MANAGEMENT) is required.

    curl -i -X POST \
       -H "Accept:application/json" \
       -H "Authorization:Bearer <personal_access_token_or_service_access_token>" \
     'https://api.<env>.cloud.talend.com/orchestration/dataplane-environments/<environmentId>/connection?dataPlaneId=<data_plane_id_you_obtained>'
    For further information about this endpoint, see Create a connection between an environment and a Data Plane.

Results

The demo Talend Data Plane has been successfully deployed in your VM and connected with Talend Control Plane, ready to be used to run your Job tasks as you usually do on the standard engines. For details about how to add the task to the Data Plane environment for a run, see Adding tasks to a Data Plane environment.

If you want to discover what has been installed in your Kubernetes cluster, run the commands outlined in this official Kubernetes documentation on your VM.

What to do next

When you have tested the demo Control Plane and Data Plane solution successfully, proceed as explained in the following section to register your real-world Kubernetes cluster for establishing an actual Talend Data Plane.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!