Skip to main content Skip to complementary content
Announcements

Talend Cloud Pipeline Designer architecture

Identifies the functional blocks of Talend Cloud Pipeline Designer and shows the interactions between Talend Management Console, Talend Cloud Pipeline Designer and other components in the cloud service app.
Talend Cloud Pipeline Designer architecture
The diagram pictures the functional blocks of Talend Cloud Pipeline Designer. On the left side, the customer-controlled environment contains the web browser and the Remote Engine Gen2. On the right side, the cloud environment contains Talend Cloud Pipeline Designer, Talend Cloud Data Preparation, Talend Cloud Data Inventory, and Talend Management Console, as well as a description of the interactions between them and the local environment.

The diagram is divided into two main parts: the customer's local network or Amazon Virtual Private Cloud (VPC) and the cloud infrastructure.

Local network or Amazon Virtual Private Cloud (VPC)

The local network includes a web browser that is used to design pipelines and a Remote Engine Gen2 that is used to execute pipelines.

The Remote Engine Gen2 can be installed either on premises or on the customer's Virtual Private Cloud.

Cloud infrastructure

The cloud infrastructure includes the cloud apps, that rely on the Dataset service, and the engines used to run pipelines.
  • The Dataset service is what provides the unified dataset list within Talend Cloud. Talend Cloud Data Inventory is the central place where you access and maintain your dataset collection. You will be able to quickly search your data, assess its quality, rate, document or share it with other data consumers.

    Talend Cloud Data Preparation and Talend Cloud Pipeline Designer are the two other applications that benefit from the common dataset inventory, and allow you to cleanse or transform your data.

  • In Talend Management Console, you can administrate roles, users, projects, engines and licenses.

    You have a consistent experience between the two cloud apps, as the pipelines published from Talend Cloud Pipeline Designer and their related information (version, type, etc.) can be visualized from Talend Management Console and used to create executable tasks.

  • In Talend Cloud Pipeline Designer, you can design complex end-to-end pipelines to process, enrich and transform data at rest or in motion. You can run these pipelines either using the embedded Cloud Engine for Design or using a Remote Engine Gen2 installed on your Virtual Private Cloud or on-premises.
  • The Cloud Engine for Design that is embedded by default in Talend Cloud Pipeline Designer helps users quickly get started with the app, but it is recommended to install the secure Remote Engine Gen2 for advanced processing of data.

  • The secure Remote Engine Gen2 and its corresponding run profiles are created from Talend Management Console before they can be used to execute pipelines in Talend Cloud Pipeline Designer.

    For more information on these engines, see the Talend Remote Engine Gen2 Quick Start Guide.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!

Leave your feedback here