Talend Cloud Pipeline Designer architecture - Cloud

Talend Cloud Pipeline Designer User Guide

EnrichVersion
Cloud
EnrichProdName
Talend Cloud
EnrichPlatform
Talend Pipeline Designer
task
Administration and Monitoring > Monitoring executions
Administration and Monitoring > Monitoring logs
Data Governance > Filtering data
Data Quality and Preparation > Filtering data
Data Quality and Preparation > Managing datasets
Deployment > Deploying > Executing Pipelines
Design and Development > Designing Pipelines
Identifies the functional blocks of Talend Cloud Pipeline Designer and shows the interactions between Talend Cloud Management Console, Talend Cloud Pipeline Designer and other components in the cloud service app.

The diagram is divided into two main parts: the customer's local network or Amazon Virtual Private Cloud (VPC) and the cloud infrastructure.

Local network or Amazon Virtual Private Cloud (VPC)

The local network includes a web browser that is used to design pipelines and a Remote Engine Gen2 that is used to execute pipelines.

The Remote Engine Gen2 can be installed either on premises or on the customer's Virtual Private Cloud.

Cloud infrastructure

The cloud infrastructure includes the cloud apps, that rely on the Dataset service, and the engines used to run pipelines.
  • The Dataset service is what provides the unified dataset list within Talend Cloud. Talend Cloud Data Inventory is the central place where you access and maintain your dataset collection. You will be able to quickly search your data, assess its quality, rate, document or share it with other data consumers.

    Talend Cloud Data Preparation and Talend Cloud Pipeline Designer are the two other applications that benefit from the common dataset inventory, and allow you to cleanse or transform your data.

  • In Talend Cloud Management Console, you can administrate roles, users, projects, engines and licenses.

    You have a consistent experience between the two cloud apps, as the pipelines published from Talend Cloud Pipeline Designer and their related information (version, type, etc.) can be visualized from Talend Cloud Management Console and used to create executable tasks.

  • In Talend Cloud Pipeline Designer, you can design complex end-to-end pipelines to process, enrich and transform data at rest or in motion. You can run these pipelines either using the embedded Cloud Engine for Design or using a Remote Engine Gen2 installed on your Virtual Private Cloud or on premises.
  • The Cloud Engine for Design that is embedded by default in Talend Cloud Pipeline Designer helps users quickly get started with the app, but it is recommended to install the secure Remote Engine Gen2 for advanced processing of data.
  • The secure Remote Engine Gen2 and its corresponding run profiles are created from Talend Cloud Management Console before they can be used to execute pipelines in Talend Cloud Pipeline Designer.

    For more information on these engines, see the Remote Engine Gen2 Quick Start Guide.