Overview of Talend Cloud
This article describes the internal Talend Cloud structure and which elements serve to manage development and day-to-day operations.
The article consists of the following sections:
- Development in Talend Cloud
- Operations in Talend Cloud
- Simple Talend Cloud architecture
The diagram below represents the different elements that exist in Talend Cloud and how they relate to one
another. They have been grouped into development and operational areas.
Relationships are shown using crow’s foot notation.
Leveraging development and operational elements in Talend Cloud enables you to manage your Software Development Life Cycle within a single cloud application.
Development in Talend Cloud
Talend Cloud consists of elements that centrally define and manage development projects and provide access to those projects.
When a user logs in to Talend Studio
this information is interpreted, and the appropriate projects are available to the user.
The following elements facilitate this operation:
| Element | Description |
|---|---|
| User | Identified by an email address created to access Talend Cloud. |
| Role | Permissions are grouped together to define a role. A list of predefined roles already exists, but custom roles can also be created. At least one role must be assigned to a user to allow access to Talend Cloud. |
| Group | A collection of users. |
| Project | Provides access to a source repository. To give access
to a project, users and groups are assigned as collaborators. The user's
roles determine the project privileges. Note: Projects are not available for Talend Cloud Pipeline Designer.
|
The diagram below provides a conceptual example of a project in Talend Cloud using the elements outlined above.
Once a user has been given Operator and Studio Integration Developer roles and has been added to the project as a collaborator, they will see the project in Talend Studio.
Operations in Talend Cloud
Talend Cloud provides the elements to centrally define, manage, orchestrate, execute, and promote artifacts published from Talend Studio or Talend Cloud Pipeline Designer.
The following elements facilitate the operations:
| Element | Description |
|---|---|
| Environment | A self-contained space with the required resources that allow the execution of tasks in isolation, without any impact on other environments. |
| Cloud Engine | A compute resource managed by Talend in the cloud that executes tasks. A Cloud Engine is available to an environment. |
| Remote Engine | A compute resource managed by Talend in the cloud that executes tasks. Remote Engines are created for an environment. |
| Remote Engine Cluster | Remote Engines can be grouped together. Any number of clusters can be
created in an environment. Note: Remote Engines added to a cluster cannot be used
to execute tasks directly from Talend Studio.
|
| Remote Engine Gen2 | A compute resource managed by Talend in the cloud that executes tasks. To use a Remote Engine Gen2, you must use the default environment. |
| Run profile | Run profiles contain the set of configurations for executing pipelines on each Remote Engine Gen2. You can define the right resource allocation and advanced properties used at runtime in the run profiles. |
| Workspace | Enables selected users to work together preparing tasks and resources for execution. Any number of workspaces can be created in an environment. Workspaces are used to share tasks, connections, and resources. Users can be added to a workspace with predefined permissions. A user's roles have no impact in a workspace. Groups cannot be added to a workspace. |
| Artifact | A Job, Route, Data Service created in Talend Studio or pipeline created in Talend Cloud Pipeline Designer and published to Talend Cloud. Artifacts are published to a workspace. |
| Connection | Predefined parameter values to access external systems. Connections are created in a workspace. |
| Resource | A cloud data source to store files and directories. Resources are created in a workspace. |
| Task | An artifact with a specific execution configuration created in a workspace. For example, Artifact-1 using Connection-1 executing on RemoteEngine-1. Properly named context variables indicate that an appropriate connection and/or resource must be specified during task configuration. |
| Plan | Scheduled tasks organized into a combination of sequential and parallel
steps. Plans are created in a workspace, but may reference tasks from other
workspaces in the same environment. Note: Plans
can only be created from Job tasks.
|
| Promotion | Provides the mechanism to copy elements from one environment to another.
Once tasks and plans have been verified, the promotion is used to advance the
elements to the next logical environment. Note: Pipelines and personal workspaces
cannot be promoted.
|
The diagram below provides a conceptual example of an environment in Talend Cloud using Talend Studio and the elements outlined
above.
Simple Talend Cloud architecture using Talend Studio
The following diagram outlines a basic Talend Cloud development architecture:
Key points:
- A project in Talend Cloud must be linked to a source repository.
- At least one Cloud Engine or Remote Engine is required.
The next diagram shows a sample architecture with a Development, Test, and Production environment.
Key points:
- Talend Cloud is used to manage operations in all environments.
- A source repository (Talend project) is only relevant to the development environment.
- As the artifacts have already been published toTalend Cloud in the development environment, there is no need for Talend Studio to access any other environments.
- At least one Cloud Engine or Remote Engine is required per environment.
For a more detailed architecture diagram, see the Talend Cloud Getting Started Guide.
Simple Talend Cloud architecture using Talend Cloud Pipeline Designer
The following diagram outlines a basic Talend Cloud development architecture with Talend Cloud Pipeline Designer:
Key points:
- Pipelines are published from Talend Cloud Pipeline Designer to Talend Cloud Management Console.
- Pipelines are available only in the default environment.
- At least one Remote Engine Gen2 is required.
For a more detailed architecture diagram, see the Talend Cloud Getting Started Guide.