Setting up a Dynamic Engine and a Dynamic Engine environment in Kubernetes

Provide one of your own Kubernetes clusters to deploy a Dynamic Engine and its environment on it.

Adding a Dynamic Engine

Add a Dynamic Engine logical entity to help deploy a Dynamic Engine in your Kubernetes cluster.

Before you begin

You must have the Infrastructure Administrator (containing the TMC_CLUSTER_MANAGEMENT permission) role in Talend Management Console.

Procedure

Go to the Processing page.
Open the Dynamic Engines tab.
Click Add Dynamic Engine.
Enter a name and a description.
The name must contain only alphanumeric, -, . and _ characters. Spaces and other special characters are not allowed.
Click Add to validate your changes and open the dialog for Dynamic Engine deployment.
A zip containing two manifest files is generated to apply on your Kubernetes cluster to deploy the Dynamic Engine.
Download the manifest zip.
You can always find the link to download this file by hovering the cursor over the Dynamic Engine to be deployed on the Dynamic Engines list, until you have deployed the engine.
Close the dialog box.

Results

A zip file is generated. You need to download it and then sequentially apply the two manifest files in it on your Kubernetes cluster to register the cluster and deploy your Dynamic Engine.

Important: Do not change the content of this manifest file. Only the default configuration pre-defined in this file is supported.

The status of your Dynamic Engine is set to New.

Setting up your Dynamic Engine in Kubernetes using the manifest files

Beta

Apply the generated Dynamic Engine manifest files in the Kubernetes cluster to deploy the Dynamic Engine.

Procedure

Unzip the file previously downloaded to obtain two manifest files: custom-resource-definitions.yaml and resources.yaml.

In your Kubernetes terminal, run the following commands to apply the manifest files in the order presented here:

```
kubectl apply -f custom-resource-definitions.yaml
```
This manifest file creates two custom Kubernetes resource types in your cluster: QlikRuntimeInfra for Dynamic Engine resources and QlikRuntime for Dynamic Engine environment resources.
```
kubectl apply -f resources.yaml
```
This manifest file creates the Dynamic Engine resources to registers your cluster, deploys your Dynamic Engine, and assign the engine to its logical entity in Talend Management Console.

At the end of the second execution, the message returned in your terminal reads like this, confirming that the engine has been successfully deployed:

namespace/talend-infra created
secret/talend-registry created
secret/artifactory-helm-repo-credentials created
serviceaccount/qlik-installer created
role.rbac.authorization.k8s.io/qlik-installer created
rolebinding.rbac.authorization.k8s.io/qlik-installer created
role.rbac.authorization.k8s.io/engine-operator created
rolebinding.rbac.authorization.k8s.io/engine-operator created
clusterrole.rbac.authorization.k8s.io/qlik-installer created
clusterrolebinding.rbac.authorization.k8s.io/qlik-installer created
customresourcedefinition.apiextensions.k8s.io/qlikruntimeinfras.qlik.com created
customresourcedefinition.apiextensions.k8s.io/qlikruntimes.qlik.com created
serviceaccount/engine-operator-sa created
role.rbac.authorization.k8s.io/engine-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/engine-operator-role created
rolebinding.rbac.authorization.k8s.io/engine-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/engine-operator-rolebinding created
deployment.apps/engine-operator-dp created
qlikruntimeinfra.qlik.com/data-plane-c-m-3qiao6iq created
secret/qlik-oidc-client configured

Results

Once done successfully, the status of the Dynamic Engine logic entity in Talend Management Console becomes active.

If you need to know what has been installed in your Kubernetes cluster, go to your Kubernetes machine and run the commands outlined in this official Kubernetes documentation.

Adding a Dynamic Engine environment

Beta

Create a new environment dedicated to a Dynamic Engine.

This environment can be used with Dynamic Engines only and can be assigned to a single Dynamic Engine at a time.

Before you begin

You must have the Environment Administrator role in Talend Management Console.

Procedure

Go to the Environments page.
Open the Dynamic Engine Environments tab.
Click Add environment.
Enter the name of the environment.
The name must contain only alphanumeric, -, # and _ characters. Leading or trailing spaces and other special characters are not allowed.
Enter the name of the first workspace in the new environment.
The allowed characters are the same as the environment name.
Optional: If you want another user to be the workspace owner, select their username from the drop-down list.
Enter a description.
Click Add to save your changes.

Results

The status of your Dynamic Engine environment is set to Not assigned.

The environment is created with the workspace you specified.

To share the workspace, click the environment name then click the Share workspace icon next to the workspace name from the Workspaces tab.

To add other workspaces to this new environment, click the environment name then Add workspace from the Workspaces tab.

What to do next

You are ready to assign your environment to a Dynamic Engine.

Assigning an environment to a Dynamic Engine

Beta

Assign one or more environments to an active Dynamic Engine. The next task runs from the environments will occur in the Dynamic Engine.

Assigning an environment to a Dynamic Engine overrides the existing settings of engine and run profile for that environment.

Before you begin

You must have the Environment Administrator role in Talend Management Console.
The Dynamic Engine to be associated must have be deployed in a Kubernetes cluster (Active status).

Procedure

Go to the Processing page.
Open the Dynamic Engines tab.
Hover the cursor in the Name column on the row of the Dynamic Engine to which you need to assign an environment.
A button appears next to the engine's name.
Click this bugton to open the page for environment assignment.
Alternatively, you can click the Dynamic Engine name to open the details panel, then click the Manage button to open the page for environment assignment.
Click Assign environment.
In the dialog that is displayed, select one environment from the drop-down list.
Click Assign.
Example
Click Download the manifest to download this manifest file.

In the terminal of your Kubernetes cluster, execute the following command to apply this manifest file.

kubectl apply -f <your_environment_manifest_file>.yaml

Upon the successful execution of this command, the following message is returned:

qlikruntime.qlik.com/remote-engine-0f719636-6d65-48f0-92c3-a3e987447ab1 created
namespace/remote-engine-0f719636-6d65-48f0-92c3-a3e987447ab1 created
secret/talend-registry created
secret/artifactory-helm-repo-credentials created
role.rbac.authorization.k8s.io/qlik-installer created
rolebinding.rbac.authorization.k8s.io/qlik-installer created
role.rbac.authorization.k8s.io/engine-operator created
rolebinding.rbac.authorization.k8s.io/engine-operator created
configmap/runtime-specs-latest created
configmap/global-configuration created
configmap/pipeline-configuration created

In this message, you can find the name of the Dynamic Engine environment as a resource. In the example above, this resource name is remote-engine-0f719636-6d65-48f0-92c3-a3e987447ab1. The name is also used as the namespace name, grouping all the services associated with the Dynamic Engine environment resource.

Verify that all services of the Dynamic Engine environment are correctly deployed and running by executing this command:

kubectl get qlikruntime remote-engine-0f719636-6d65-48f0-92c3-a3e987447ab1 -o json | jq -r '.status.conditions'

In this command, qlikruntime represents the custom resource type dedicated to Dynamic Engine environments, and as explained in the example above, remote-engine-0f719636-6d65-48f0-92c3-a3e987447ab1 is the resource and the namespace name of the environment.

If you receive only messages reading like this, the environment deployment has been successful:

[
  {
    "lastTransitionTime": "2024-05-30T12:51:29Z",
    "message": "Service is not deployed",
    "reason": "Deployment_not_found",
    "status": "False",
    "type": "NOT_DEPLOYED"
  },
  {
    "lastTransitionTime": "2024-05-30T12:51:39Z",
    "message": "Service is starting",
    "reason": "Pod_pending",
    "status": "False",
    "type": "STARTING"
  },
  {
    "lastTransitionTime": "2024-05-30T12:54:30Z",
    "message": "Service is running",
    "reason": "Pod_ready",
    "status": "True",
    "type": "RUNNING"
  }
]

If the messages contain statues such as "NOT_DEPLOYED" or "IN_ERROR", the deployment has issues. You need to run this command to obtain and inspect the "serviceStatus" object of the environment resource:

kubectl get qlikruntime remote-engine-92d6e127-e070-4264-a9e8-00adbbbfbc03 -o json | jq -r '.status'

The following message shows an example of the error on the "di-job-deployer" service in the "serviceStatus" object. Its "jobErrorMessage" field gives more details about the error observed during the deployment, indicating that the expected helm chart has not been found.

Note that some lines are skipped in the following example.

{
  "conditions": [
    {
      "lastTransitionTime": "2024-05-30T13:05:25Z",
      "message": "Service is not deployed",
      "reason": "Deployment_not_found",
      "status": "True",
      "type": "NOT_DEPLOYED"
    }
  ],
  "serviceStatus": {
    ...
    "di-job-deployer": {
      "conditions": [
        ...
        {
          "lastTransitionTime": "2024-05-30T13:06:39Z",
          "message": "Service is in error",
          "reason": "Pod_failed",
          "status": "False",
          "type": "IN_ERROR"
        }
      ],
      "creationDate": "2024-05-30T13:05:25Z",
      "jobErrorMessage": "Error: release: not found\nUpgrading...\nupgrade 
      result: Release \"di-job-deployer-92d6e127-e070-4264-a9e8-00adbbbfbc03\" does not exist. 
      Installing it now.\nError: failed to download 
      ...
      \nError: failed to download 
      \"https://artifactory.datapwn.com/artifactory/tlnd-helm-ce-dev/dpe-di-job-deployer-2.0.0.tgz\
      "\nexit_code: 1\n",
      "jobStatus": "FAILED",
      "lastUpdateDate": "2024-05-30T13:01:52Z"
    },
      ...
    }
  }
}

In this situation, contact the Support team to investigate the Dynamic Engine environment manifest file. When informed, unassign the environment and assign it again to generate a new manifest file.

Results

Once done successfully, the Dynamic Engine environment services are installed in your cluster and connect to the Dynamic Engine.

In Talend Management Console, the status of this environment becomes Ready, confirming that it is ready to run tasks or plans.

If the manifest implementation fails or the Dynamic Engine services are disassociated from the Dynamic Engine, the status become Not ready.

What to do next

Since the Dynamic Engine has been successfully deployed in your Kubernetes, connected to Talend Management Console, and associated with a Dynamic Engine environment, it is ready to run Job tasks or plans as you usually do on the standard engines. For details about how to add the task to the Dynamic Engine environment for a run, see Adding tasks to a Dynamic Engine environment.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!

Leave your feedback here

Setting up a Dynamic Engine and a Dynamic Engine environment in Kubernetes

Adding a Dynamic Engine

Before you begin

Procedure

Results

Setting up your Dynamic Engine in Kubernetes using the manifest files

Procedure

Results

Adding a Dynamic Engine environment

Before you begin

Procedure

Results

What to do next

Assigning an environment to a Dynamic Engine

Before you begin

Procedure

Example

Results

What to do next

Did this page help you?