Enabling runtime lineage for Jobs - Cloud - 8.0

Talend Studio User Guide

Version
Cloud
8.0
Language
English
Product
Talend Big Data
Talend Big Data Platform
Talend Cloud
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Real-Time Big Data Platform
Module
Talend Studio
Content
Design and Development
Last publication date
2024-02-13
Available in...

Big Data

Big Data Platform

Data Fabric

Data Integration

Data Management Platform

Data Services Platform

ESB

MDM Platform

Real-Time Big Data Platform

Talend Studio allows you to enable runtime lineage for Standard Jobs, which can be leveraged in a future release by the analysis capability of Talend Data Catalog for the runtime metadata, for example, the query with variables, the schema with dynamic columns, etc.

When executing a Standard Job for which runtime lineage is enabled, the information needed by Talend Data Catalog, for example, the Job name, the component name, the schema, the query, etc., will be written into a JSON file.

Note: To fully use this feature, you must install Talend Data Catalog.

About this task

To enable runtime lineage for Standard Jobs, complete the following:

Procedure

  1. Go to the Talend Studio installation directory.
  2. Add the -Druntime.lineage=true attribute in the corresponding .ini file according to your operating system to enable the runtime lineage feature in Talend Studio.
  3. Save the file and start Talend Studio.
  4. Click Project settings on the toolbar of the Talend Studio main window or click File > Edit Project properties from the menu bar to open the Project Settings dialog box.
  5. In the tree view of the dialog box, expand the Job Settings node and then click Runtime lineage to display the corresponding view.
    Runtime Lineage configuration in the Project Settings dialog box.
  6. Enable runtime lineage for Standard Jobs via either of the following two ways:
    • To enable runtime lineage for all Standard Jobs, select the Use runtime lineage for all Jobs check box.
    • To enable runtime lineage for specific Standard Jobs, select the check boxes corresponding to the Jobs in the Use runtime lineage for selected Jobs area.
  7. In the Output path field, specify the path where you want to save the JSON files used by Talend Data Catalog.
    Later, each time you execute a Standard Job for which runtime lineage is enabled, a JSON file will be saved under a directory with the format <output_path>/<project>/<jobname>/<version>/runtime_log_<timestamp>.json, where
    • <output_path> is the path specified in the Output path field,
    • <project> is the name of the project,
    • <jobname> is the name of the Job,
    • <version> is the version of the Job, and
    • <timestamp> is the timestamp when the JSON file is generated.

    You can also set the output path by adding a JVM parameter -Druntime.lineage.outputpath=<output_path> for the Job via one of the following ways:

    Note: The output path must be specified for saving the JSON files. If the output path value is specified in multiple places, one of them will take effect according to the following precedence: 1) the value of the JVM parameter for specific Job, 2) the value of the Output path field, 3) the value of the JVM parameter for all Jobs, 4) the value of the JVM parameter in the shell script.
  8. Click Apply and Close to apply your changes and close the dialog box.