Prerequisites to use Dynamic Engines
Access permissions to Talend Management Console and Dynamic Engines
-
Generate access tokens:
- For users, generate a personal access token by following Generating a Personal Access Token.
- For service accounts, generate a service access token by following Generating a service account token.
Once generated, a service account token expires after 30 minutes. If it expires, generate a new token using the POST method at the endpoint https://api.<env>.cloud.talend.com/security/oauth/token. For more information about generating a token, see Generating a service account token.
- You must have the Infrastructure Administrator (containing the TMC_CLUSTER_MANAGEMENT permission) role to access the Dynamic Engine tab in Talend Management Console.
- You must have the Environment Administrator (contains the TMC_ENVIRONMENT_MANAGEMENT permission) role to access the Dynamic Engine Environments tab.
Kubernetes specifications
- If you do not have any Kubernetes cluster yet, create one either locally or on a Cloud platform such as Azure or AWS.
- The Kubernetes cluster must be one of the versions from v1.27 to v1.30.
- Minimum requirements for the Kubernetes cluster:
- Instance size: 16 GB of memory and 4 vCPU
- Number of nodes: 3
- Disk size: 20 GB (including 2 GB minimum for running the engine plus space for
deployed Job artifacts)
This requires the availability of CSI provider and a corresponding storage class configured. This CSI provider and storage class allow your Kubernetes cluster to have persistent storage that is accessible from any node.
The supported CSI storage providers are
- Amazon EFS. For further information about mounting this persistent storage, see Amazon EFS CSI Driver.
- Azure Files. For further information, see Create and use a volume with Azure Files in Azure Kubernetes Service (AKS).
- If you cannot use either of the above providers, install Longhorn, a distributed block storage system for Kubernetes, in your cluster. With Longhorn, the required minimum disk size is 100 GB. For further information about installing Longhorn, see (Optional) Setting up a Longhorn storage solution for your Kubernetes cluster.
Note that for any CSI storage providers, when configuring the storage class, the following annotation must be added:annotations: storageclass.kubernetes.io/is-default-class: "true"Information noteTip: Although not tested, other CSI storage providers, such as Google Cloud, Digital Ocean, or Open Shift, should work as well as the supported ones, provided these providers use standard Kubernetes distributions.
DNS URLs to be added in your proxy or firewall allowlist
Region: AWS EU data center
The port of all these URLs is 443. Their traffic is outbound.
URL Used by Purpose Requirement level https://pair.eu.cloud.talend.com Dynamic Engine Send the initial pairing request, and heartbeats
Required https://engine.eu.cloud.talend.com or wss://engine.eu.cloud.talend.com
Dynamic Engine Pipeline task executions Required https://vault-gateway.eu.cloud.talend.com Dynamic Engine Decrypt secrets used in pipeline task executions Required https://msg.eu.cloud.talend.com Dynamic Engine Data Integration Job task executions Required https://minio-artifacts.eu.cloud.talend.com Dynamic Engine Data Integration Job task executions Required https://minio-customerlogs.eu.cloud.talend.com Dynamic Engine Data Integration Job task executions Required ghcr.io
and
docker.io
Dynamic Engine Provide Docker images Required *.talend.github.io
and
charts.rancher.io
Dynamic Engine Provide Helm charts Required Region: AWS US data center
The port of all these URLs is 443. Their traffic is outbound.
URL Used by Purpose Requirement level https://pair.us.cloud.talend.com Dynamic Engine Send the initial pairing request, and heartbeats
Required https://engine.us.cloud.talend.com or wss://engine.us.cloud.talend.com
Dynamic Engine Pipeline task executions Required https://vault-gateway.us.cloud.talend.com Dynamic Engine Decrypt secrets used in pipeline task executions Required https://msg.us.cloud.talend.com Dynamic Engine Data Integration Job task executions Required https://minio-artifacts.us.cloud.talend.com Dynamic Engine Data Integration Job task executions Required https://minio-customerlogs.us.cloud.talend.com Dynamic Engine Data Integration Job task executions Required ghcr.io
and
docker.io
Dynamic Engine Provide Docker images Required *.talend.github.io
and
charts.rancher.io
Dynamic Engine Provide Helm charts Required Region: Azure US West data center
The port of all these URLs is 443. Their traffic is outbound.
URL Used by Purpose Requirement level https://pair.us-west.cloud.talend.com Dynamic Engine Send the initial pairing request, and heartbeats
Required https://engine.us-west.cloud.talend.com
or
wss://engine.us-west.cloud.talend.com
Dynamic Engine Pipeline task executions Required https://vault-gateway.us-west.cloud.talend.com Dynamic Engine Decrypt secrets used in pipeline task executions Required https://msg.us-west.cloud.talend.com Dynamic Engine Data Integration Job task executions Required https://minio-artifacts.us-west.cloud.talend.com Dynamic Engine Data Integration Job task executions Required https://minio-customerlogs.us-west.cloud.talend.com Dynamic Engine Data Integration Job task executions Required ghcr.io
and
docker.io
Dynamic Engine Provide Docker images Required *.talend.github.io
and
charts.rancher.io
Dynamic Engine Provide Helm charts Required
Did this page help you?
If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!