Creating Job tasks - Cloud

Talend Cloud Management Console User Guide
Version
Cloud
Language
English
Product
Talend Cloud
Module
Talend Management Console
Content
Administration and Monitoring > Managing projects
Administration and Monitoring > Managing users
Deployment > Deploying > Executing Pipelines
Deployment > Deploying > Executing Tasks
Deployment > Scheduling > Scheduling Tasks
Last publication date
2023-09-20

Job tasks are executable integration processes that contain a Job artifact published from Talend Studio.

If contexts have been defined for this Job artifact in the Studio, the default context is automatically published with this Job artifact. For recommended rules about defining these contexts in the Studio, see Using contexts and variables.

The values in this context configuration are cleared in the exported files for security reasons. You need to enter the context values again in Talend Cloud Management Console.

Before you begin

  • You must have the Author permission on the workspace where the task is located to create or edit a Job task and manage its configuration.

    The Author permission allows you to configure the Artifact and Parameters steps when creating or editing a task.

  • You must have the Execute permission on the workspace where the task is located to manage the run configuration and schedule, such as setting engine, log level or execution timeout.

    The Execute permission allows you to configure the Engine and Schedule steps when creating or editing a task.

Procedure

  1. Open the Management page.
  2. Go to the Tasks tab.
  3. Select the environment and the workspace using the faceted search at the top of the page.
  4. Click Add task.
  5. In the Artifact step, select the Job artifact type.
  6. Select the artifact you want to execute from the drop-down list.
    The details and description of the artifact appears.
  7. Select the version of the artifact to be used in the task.

    You can select Always use the latest available artifact version if you want your task to be updated automatically each time a new version of the artifact is published. You can also select a specific version and update it manually as needed.

    Note: This option overrides the Update corresponding job task option in Talend Studio.
    The details and description of the artifact are refreshed automatically.
  8. Optional: Select the Override parameter values with artifact defaults check box.
    This option is only available if you selected Always use the latest available artifact version in the Artifact version field. If you select it, the parameters defined in the task are replaced with the default artifact parameters each time a new version of the artifact is published.
  9. Select the workspace for the task.
  10. Give a name to the new task.
    By default, the task name is the name of the artifact from which it is created.
    Note: Tasks must have a unique name within a workspace.
  11. Optional: Add a tag to the task.
    1. Select an existing tag from the drop-down list.
    2. Create a new tag by typing in the name and clicking on it in the drop-down list.
  12. Click Continue.
  13. In the Parameters step, enter the value for the parameters of the Job.
    In this step, when the Show order as in Studio toggle is disabled, you can only set the values to parameters you have defined with the parameter_ prefix in Studio.
    Note: Use \\ instead of \ when the value could be interpreted as a special character, for example, \t.

    When being displayed in Talend Cloud Management Console, the first letters of each word in the parameter names are automatically changed to upper case and underscores (_) to whitespace, for example, city_name becomes City Name.

    This is for display purpose only, that is to say, when you need to use a parameter name in your programs, for example, an API request, do not use the display name but use the name from Talend Studio.

  14. Optional: In the Advanced parameters section, enter the values for the parameters you have defined in Studio without prefixes, and connection_ parameters that are not defined in the Job. This section is available when the Show order as in Studio toggle is disabled. Otherwise, all parameters are in the Parameters step.
    Note: Use \\ instead of \ when the value could be interpreted as a special character, for example, \t.
  15. Optional: In the Connections step, select the appropriate connection from the drop-down list or create a new connection based on the parameters already defined in the artifact.
  16. Optional: In the Resources step, select the appropriate resource to use in the task.
  17. In the Engine step, select the engine or engine cluster on which you want to run your task from the Engine drop-down list.
    • Cloud
    • Cloud exclusive
    • Remote Engine
    • Remote Engine cluster
  18. Optional: Select a run profile.
    You can choose a previously created run profile for the selected Remote Engine or cluster to set the JVM parameters for the task run.

    If you are using a Remote Engine or cluster, this feature is supported from Remote Engine 2.9.1 onwards.

  19. Set the log level.

    Select the lowest log level to be taken into account.

    If you are using a Remote Engine or cluster, this feature is supported from Remote Engine 2.8.3 onwards.

    Example

    If the log level is set to Warning, only logs flagged as Warning and Error will be displayed on the Task run log page.

    For security reasons, set the log level to Warning for Jobs deployed to production.

  20. Optional: Select the Allow parallel run of this task option so that the task can be run in parallel when using a Cloud Engine, Cloud exclusive, Remote Engine, or cluster.

    If you are using a Remote Engine to run identical tasks simultaneously, this feature is supported from Remote Engine 2.12.0 onwards.

    By default, you can run three different or identical tasks in parallel on a single Remote Engine. For further information about the related configuration, see Running tasks in parallel on a single Remote Engine.

    Before you disable this option, stop the task or wait for the end of the run. Otherwise, the parallel run is still allowed.

  21. Optional: If you have selected a Remote Engine or cluster, you can run your Job as another user by entering their username in the Run as impersonated user field.
    This feature is supported only if the Remote Engine is installed on Unix or similar environments.
    For more information about user impersonation, see the Talend Remote Engine User Guide.
  22. Optional: Set the task execution timeout by selecting one of the following options:
    • Custom then entering a value.

      The default value is 5 minutes and the minimum is 1.

    • Smart Timeout.

      For more information on smart timeout, see Enabling smart task timeout.

  23. Optional: In the Schedule step, select the execution frequency and if necessary the trigger timeout.

    You can add one or more triggers to schedule your task or you can manually start the task without a trigger.

    By default, the trigger timeout value is the same as the task timeout.

    Trigger timeout overrides task timeout.

    For more information, see Scheduling Job tasks.
  24. Click Save and close.

Results

Your task is ready to run on Talend Cloud Management Console.

Note:

Exponential back-off retry pattern has been adopted to provide not only faster first and second retries (1 minute and 3 minutes respectively after the initially failed attempt), but also better global behaviour on peak workload situation.

For example, when you try to execute a task that is already running, the execution fails. Talend Cloud Management Console retries the execution with incrementally increased intervals, that is to say, the first retry takes place at the end of 1 minute only, the second one 2 minutes, then subsequently 4 minutes and 8 minutes.

If the fourth retry fails, you will get the error message Exceeded the limit of deployment attempts: another instance of the same task was running on the engine at same time. Try to run the task later.

Troubleshooting: If the task execution schedule is too frequent, it may result in the parallel run of the same task on two different Cloud Engines or Remote Engine cluster. To avoid this issue, run your task in an environment with only a single Cloud Engine allocated to it or schedule the task on a Remote Engine.

Example