Saving execution logs to an external Amazon S3 bucket - Cloud

Talend Cloud Management Console User Guide
Version
Cloud
Language
English
Product
Talend Cloud
Module
Talend Management Console
Content
Administration and Monitoring > Managing projects
Administration and Monitoring > Managing users
Deployment > Deploying > Executing Pipelines
Deployment > Deploying > Executing Tasks
Deployment > Scheduling > Scheduling Tasks
Last publication date
2023-09-20
AWSDeprecated
Logs are saved to separate folders, whether the tasks are run live or invoked through a plan.

Before you begin

Before starting this procedure, you must create an Amazon S3 bucket, configure cross-account roles (see https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html) and provide your configuration in Talend Cloud Management Console.

About this task

The external Amazon S3 bucket transfer of log data is not supported for Data Service and Route logs.

Procedure

  1. Go to Configurations > Management Console.
  2. Toggle the Export logs switch on.
  3. On the Management Console export logs page click the Cloud Formation template link to download the Talend Cloud AWS CloudFormation template.
  4. Open your AWS account in a new tab and start the Create Stack wizard on the AWS CloudFormation Console.
  5. In the Select Template step, select Upload a template to Amazon S3 and pick the template provided by Talend Cloud.
  6. In the Specify Details section, define the External ID, S3BucketName, and S3 prefix parameters.
  7. In the Review step, select I acknowledge that AWS CloudFormation might create IAM resources.
  8. Click Create.
    The stack is created. If you select the stack, you can and find the RoleARN key value in the Outputs tab.
  9. Go back to Talend Cloud Management Console.
  10. Supply the required parameters:

    Example

    Parameter Function
    Role ARN Amazon Resource Name uniquely identifying the target S3 bucket.
    External ID External ID of the target S3 bucket.
    Bucket Name Name of the target S3 bucket.
    Prefix (Optional) Prefix of the log files to be exported. In the target S3 bucket, the log files are created under a folder named after the prefix.
  11. Optional: Click Test export.
    A message indicates the timestamped log test file is exported to your bucket. Check that the file has correctly been exported before going further.

    In case of an error, review your AWS configuration or the parameters you have supplied on the Management Console export logs page.

  12. Click Save.

Results

Log files will be exported to your S3 bucket any time you run a task, under the folder named after the prefix. Files are generated for each execution and named with the Execution ID.

Tasks run individually generate logs under [S3Bucket_name]/[prefix]/flows.

Tasks invoked through a plan generate logs under [S3Bucket_name]/[prefix]/ep.
Execution plans generate a log file that contains the end status of the execution (successful or failed).