Creating a pipeline - Cloud

Talend Cloud Pipeline Designer User Guide

EnrichVersion
Cloud
EnrichProdName
Talend Cloud
EnrichPlatform
Talend Pipeline Designer
task
Administration and Monitoring > Monitoring executions
Administration and Monitoring > Monitoring logs
Data Governance > Filtering data
Data Quality and Preparation > Filtering data
Data Quality and Preparation > Managing datasets
Deployment > Deploying > Executing Pipelines
Design and Development > Designing Pipelines
How to create a pipeline from scratch.

Procedure

  1. On the Home page, click Pipelines > ADD PIPELINE.
  2. On the top toolbar, give a name to your pipeline.
  3. To add a source, click the ADD SOURCE placeholder on the canvas.
  4. Select the dataset you want to use in your pipeline:
    • If you have already created a dataset, select it from the list in the [Select a source] panel and click SELECT.
    • If not, add a new dataset by clicking ADD DATASET as described in Creating a dataset.
  5. Click the icon to select one or more processing component according to your needs: filtering, cleansing, aggregating, etc.
    From the [Add a processor] panel, you can either select a processor in the main list or enter its name or description in the text box.
  6. To add a destination, which is a target component that will consume your data and send it to the system of your choice, click the ADD DESTINATION placeholder on the canvas.
  7. Select the destination dataset:
    • If you have already created a dataset, select it from the list in the [Select a destination] panel and click SELECT.
    • If not, add a new dataset by clicking ADD DATASET as described in Creating a dataset.
    Example of pipeline consuming data from an S3 Input, normalizing and filtering data to send some selected data to an HDFS cluster while the rejected data is sent to another HDFS cluster.

    Note that, before executing your pipeline, you can see a preview of your data at each step of the design process.