Talend Administration Center API

How Jobs are handled via Talend Administration Center

Talend Jobs are scheduled from Talend Administration Center, which provides a browser management interface, and are parameterized with Context variables.

But often you may prefer to have programmatic control of Jobs via an API. This post covers how to expose Talend Jobs via the Talend Administration Center API. It provides sample Jobs, some useful browser utilities, and an example of wrapping the API in a RESTful service layer using Data Services.

The Job Conductor page of Talend Administration Center is easy to use and powerful: its allows you to schedule Jobs with simple, more complex Cron triggers, or file triggers. You also have the possibility to launch Jobs manually from the browser. With exception of the file trigger, the Job is still running at a pre-determined time or with explicit human intervention. Using file triggers as the mechanism for inter-process communication may require access privileges which are not allowed in a secure environment, which is why it may be preferable to invoke a Talend Job via a real API.

Jobs can also be parametrized with Context variables. Context variables can be overridden by system administrators on the Job Conductor page to provide additional flexibility. But when the Job is running, it always runs with the same set of pre-configured Context variables, whether they are the default values or the overridden values, they cannot be changed without human intervention. It is preferable to be able to pass parameters via an API.

One option is to build Jobs as self-contained .zip files. The generated archives will include launching scripts and all necessary .jar files. However, the resulting Jobs are running in isolation and lack the monitoring, management, and control parts provided by Talend Administration Center. No centralized logging is provided, and there is no concept of Job Servers or a Job grid. Instead, these responsibilities fall on the developer. As individual solutions proliferate, the management of the broader system becomes more difficult and the maintenance tail becomes more unwieldy.

So while exported Jobs provide flexibility, they sacrifice manageability. The Talend Administration Center API provides a very simple and powerful alternative.

Talend Administration Center MetaServlet API

The Talend Administration Center MetaServlet API is an RPC style HTTP API, (not restful) that it is very easy to use and can be easily wrapped with a RESTful interface if desired.

MetaServlet API

All MetaServlet operations can be invoked via an HTTP POST or GET request. All parameters to the operation are encoded as a single, unnamed base-64 encoded parameter to the request.

The Talend Administration Center MetaServlet command-line tool is available in the following folder of the Talend Administration Center installation directory:

<tomcat_path>/webapps/org.talend.administrator/WEB-INF/classes/MetaServletCaller.bat on Windows

<tomcat_path>/webapps/org.talend.administrator/WEB-INF/classes/MetaServletCaller.sh on Linux

Running the MetaServletCaller with no arguments shows the top level help message:

<tomcat_path>\webapps\tac\WEB-INF\classes>MetaServletCaller.bat
usage: Missing required option: url
 -f,--format-output          format Json output
 -h,--help                   print this help message
 -json,--json-params <arg>   Required params in a Json object
 -url,--tac-url <arg>        TAC's http url
 -v,--verbose                display more informations

--tac-url

In order to get the full detailed help message, the Talend Administration Center service must be up and running and you must pass the --tac-url parameter.

--help all, -h all

Use --help all to get the full help, and the -h all for an abbreviated version. The examples below capture the output to a text file for subsequent reference.

<tomcat_path>\webapps\org.talend.administrator\WEB-INF\classes>MetaServletCaller.bat 
--tac-url=http://localhost:8080/org.talend.administrator/ -help all > tac-help.txt

<tomcat_path>\webapps\org.talend.administrator\WEB-INF\classes>MetaServletCaller.bat 
--tac-url=http://localhost:8080/org.talend.administrator/ -h > tac-help-short.txt

runTask

Runs a task based on its ID.

<tomcat_path>\webapps\org.talend.administrator\WEB-INF\classes>MetaServletCaller.bat 
--tac-url=http://localhost:8080/org.talend.administrator/ -help runTask
----------------------------------------------------------
  Command: runTask
----------------------------------------------------------
Description             : Allows to run a task defined in Job conductor by its id. 
Mode can be 'asynchronous' or 'synchronous'
Requires authentication : true
Since                   : 4.2
Sample                  :
{
  "actionName": "runTask",
  "authPass": "admin",
  "authUser": "admin@company.com",
  "jvmParams": [
    "-Xmx256m",
    "-Xms64m"
  ],
  "mode": "synchronous",
  "taskId": 1
}

In order to run a task, you need to know its system generated taskId. This information can be retrieved by executing the getTaskIdByName command.

getTaskIdByName

Gets the corresponding ID of the task by looking for its taskName.

<tomcat_path>\webapps\org.talend.administrator\WEB-INF\classes>MetaServletCaller.bat --tac
-url=http://localhost:8080/org.talend.administrator/ -help getTaskIdByName
----------------------------------------------------------
  Command: getTaskIdByName
----------------------------------------------------------
Description             : Get task id by given taskName
Requires authentication : true
Since                   : 5.1
Sample                  :
{
  "actionName": "getTaskIdByName",
  "authPass": "admin",
  "authUser": "admin@company.com",
  "taskName": "task1"
}

Invoking the Talend Administration Center API interactively

When developers work with the MetaServlet API, it can be useful for them to interactively invoke the Talend Administration Center API.

Procedure

Go to the Job Conductor page of Talend Administration Center, select the arrow icon next to any of the columns and make sure the Id column is checked so that it will be displayed.

Note that you can also retrieve the ID of the task using MetaServlet by executing the getTaskIdByName command, see Talend Administration Center MetaServlet API commands.
Go to https://www.base64encode.org/ to encode the MetaServlet JSON arguments in base64.
Send the encoded result with the POST or GET operation to the http://<host>:<port>/<TalendAdministrationCenter_name>/metaServlet URL. It is recommended to use the POST operation with the parameters in the body instead of GET for additional security.
The result of the HTTP message is returned as JSON to the object. It includes the execRequestId which is the handle for your new Job instance.
```
{ execRequestId: "1432855205979_a5zn8", executionTime: { millis: 564, seconds: 0 }, returnCode: 0 }
```
Once the HTTP message is sent, monitor the progress of the execution through the Execution History page of Talend Administration Center.

Invoking the Talend Administration Center API programmatically

Once the JSON arguments are base-64 encoded, they can be passed as the sole parameter to the HTTP request. If you are integrating with Talend, your application might be written in regular Java, or possibly some other language, which should not cause any issues since the HTTP and base-64 are interoperable standards.

Before you begin

To invoke the Talend Administration Center API, the JSON objects must be base-64 encoded.

Procedure

If you happen to be integrating with a Java application, use the following Apache Commons Base64 class method: org.apache.commons.codec.binary.Base64.encodeBase64()
If you are using Talend, use the tLibraryLoad component to add the Apache Commons library.
You can retrieve the MetaServlet Job archive attached to the Downloads tab on the left panel of this page to invoke the Talend Administration Center API from the Talend Job.

Retrieve the MetaServlet Job attached to the Downloads tab of the online version of this page at https://help.talend.com to invoke the Talend Administration Center API from the Talend Job.
It uses the encodeBase64() method within a tMap prior to the tRESTclient invocation of the Talend Administration Center API operations. Three operations are invoked, and each operation is invoked within its own SubJob. Each SubJob starts by initializing the request from the Context Parameters:
- The first invocation looks up the taskId based on the human readable Job name.
- The second invocation uses the taskId returned from the first invocation to trigger the Job.
- The third invocation uses the returned execRequestId handle as the argument to the getTaskExecutionStatus operation to monitor the Job status.

Known issues

The result returned by the getTaskIdByName command is not proper JSON. There are no enclosing quotes around the field named task Id. You need to use a regular expression to replace it prior to parsing the JSON. See TDI-32706.
The runTask command does not document the context argument but it is supported. See TDI-32519.
The runTask command supports the context argument, but the Metaservlet API will not parse it correctly if it is the last argument in the JSON payload that is submitted, see TDI-32380. As a workaround, make sure that the context variable is not the last argument (it can be passed as the second to last argument for example).