Fetching run logs periodically for analysis - Cloud

Talend Cloud APIs User Guide
Version
Cloud
Language
English
Product
Talend Cloud
Module
Talend API Designer
Talend API Tester
Talend Data Preparation
Talend Data Stewardship
Talend Management Console
Content
Design and Development > Designing APIs
Design and Development > Testing APIs

Use the /monitoring/executions/{runId}/logs endpoint and the /monitoring/executions/{runId}/logs/status endpoint to fetch run logs of a given period.

  • Use the POST method at /monitoring/executions/{runId}/logs to generate the full log of a given period of a run as well as the token to access this log. This token is valid for one hour after being generated.

    To ensure the completeness of your log, check that the run for which you need the log has completed and then wait for 30 seconds before sending the API call.

  • Use the POST method at /monitoring/executions/{runId}/logs/status to check whether this log is ready for download and generate a download URL accordingly.

Before you begin

  • Generate access tokens:

    Once generated, a service account token expires after 30 minutes. If it expires, generate a new token using the POST method at the endpoint https://api.<env>.cloud.talend.com/security/oauth/token. For more information about generating a token, see Generating a service account token.

  • The user or the service account to be used must have the Author, the Viewer and the Executor permissions for the workspace to which the task to be monitored belongs.
  • You must know the ID of the task run you need to monitor.

    This ID is available on the Task execution log page, reading as Task execution ID. Or you can obtain this ID using a GET call from the /processing/executables/tasks/{taskId}/executions endpoint.

About this task

In this example, Talend API Tester is used to issue API requests. For further information about Talend API Tester, see Talend Cloud API Tester User Guide.

Procedure

  1. Open Talend API Tester in your browser and select POST from the Method list.
  2. In the field next to the Method drop-down list, enter the endpoint to be used: https://api.<env>.cloud.talend.com/monitoring/executions/<runId>/logs
  3. Click the Add query parameter button to add the startTime and the endTime parameters. These parameters allow you to limit the query to a specific time range of your logs.
    Note that the date value must follow this format:
    yyyy-mm-ddThh:mm:ssZ
    For example, 2021-09-01T07:01:45Z.
  4. Optional: Add the fileFormat parameter to determine the format of the log file to be generated.
    The available formats are
    • json

      This is also the default format.

    • text

      This is a human readable format.

  5. Optional: Add the newFileName parameter to indicate whether you need to include task IDs and execution IDs in the filename of the log to be generated.
    This is a Boolean type parameter. Put true to include those IDs or false to use the old filename format. It is recommended to set this value to be true as the old filename format is going to be deprecated.
  6. In the HEADERS area, click Add header and in the name field that is displayed, enter Authorization and in the value field, enterBearer and your personal access token or the service account access token. Enter a whitespace to separate Bearer and the token.
  7. Verify that the run of which the log is needed has terminated.
  8. Wait for at least 30 seconds.
    Skip this step if the end time of this run was at least 30 seconds ago.
  9. Click Send to issue your request.
    The response contains the size of the generated full log and the token to access it. For example,
    {
"size": the_full_log_size,
"token": "the_generated_token"
}
  10. Click the Save button to save this request in your repository on Talend API Tester. When you save the request, you need to give it a name. In this example, this request is named as generate-full-log-and-token.
  11. Create a new request in Talend API Tester to issue a POST call.
  12. Select POST from the Method list.
  13. In the field next to the Method drop-down list, enter the endpoint to be used: https://api.<env>.cloud.talend.com/monitoring/executions/<runId>/logs/status
  14. In the Body field, enter the one-time token you obtained from the previous request.

    Example

    On Talend API Tester, you can use the expression builder to enter an expression, instead of the static token, to dynamically get the latest token generated by the previous request. Note that this request must have been saved in your repository on Talend API Tester. In this example, this saved request is generate-full-log-and-token.

    Here is an example of what the expression builder may look like:

  15. Click Send to issue your request.
    The response contains the download URL of the generated full log and the status about whether this log is ready to be downloaded.

    If the status shows that the log is not READY yet, issue this call again. When the log is ready, use the returned URL to download it.

    For example,
    {
"presignedURL": "URL_to_download_logs",
"status": "READY"
}

Results

You can then integrate these API requests in your own monitoring system to periodically generate and download the full log of a given time range for analytic purpose.

Only one run is used in this scenario for demonstration purposes. If you want to analyze all the runs of a task, you can use a GET method at the following endpoint to get the IDs of these runs:
 https://api.<env>.cloud.talend.com/processing/executables/tasks/{taskId}/executions

Then pass each of the IDs dynamically to the /monitoring/executions/{runId}/logs and the /monitoring/executions/{runId}/logs/status endpoints to prepare the logs to be used.

After all, you can develop a program, such as a Talend Job, to automate the whole process in your monitoring system.