Overview of Talend Cloud
This article describes the internal Talend Cloud structure and which elements serve to manage development and day-to-day operations.
- Development in Talend Cloud
- Operations in Talend Cloud
- Simple Talend Cloud architecture
Leveraging development and operational elements in Talend Cloud enables you to manage your Software Development Life Cycle within a single cloud application.
Development in Talend Cloud
Talend Cloud consists of elements that centrally define and manage development projects and provide access to those projects.
Element | Description |
---|---|
User | Identified by an email address created to access Talend Cloud. |
Role | Permissions are grouped together to define a role. A list of predefined roles already exists, but custom roles can also be created. At least one role must be assigned to a user to allow access to Talend Cloud. |
Group | A collection of users. |
Project | Provides access to a source repository. To give access
to a project, users and groups are assigned as collaborators. The user's
roles determine the project privileges. Note: Projects are not available for Talend Cloud Pipeline Designer.
|
Once a user has been given Operator and Studio Integration Developer roles and has been added to the project as a collaborator, they will see the project in Talend Studio.
Operations in Talend Cloud
Talend Cloud provides the elements to centrally define, manage, orchestrate, execute, and promote artifacts published from Talend Studio or Talend Cloud Pipeline Designer.
The following elements facilitate the operations:
Element | Description |
---|---|
Environment | A self-contained space with the required resources that allow the execution of tasks in isolation, without any impact on other environments. |
Cloud Engine | A compute resource managed by Talend in the cloud that executes tasks. A Cloud Engine is available to an environment. |
Remote Engine | A compute resource managed by Talend in the cloud that executes tasks. Remote Engines are created for an environment. |
Remote Engine Cluster | Remote Engines can be grouped together. Any number of clusters can be
created in an environment. Note: Remote Engines added to a cluster cannot be used
to execute tasks directly from Talend Studio.
|
Remote Engine Gen2 | A compute resource managed by Talend in the cloud that executes tasks. To use a Remote Engine Gen2, you must use the default environment. |
Run profile | Run profiles contain the set of configurations for executing pipelines on each Remote Engine Gen2. You can define the right resource allocation and advanced properties used at runtime in the run profiles. |
Workspace | Enables selected users to work together preparing tasks and resources for execution. Any number of workspaces can be created in an environment. Workspaces are used to share tasks, connections, and resources. Users can be added to a workspace with predefined permissions. A user's roles have no impact in a workspace. Groups cannot be added to a workspace. |
Artifact | A Job, Route, Data Service created in Talend Studio or pipeline created in Talend Cloud Pipeline Designer and published to Talend Cloud. Artifacts are published to a workspace. |
Connection | Predefined parameter values to access external systems. Connections are created in a workspace. |
Resource | A cloud data source to store files and directories. Resources are created in a workspace. |
Task | An artifact with a specific execution configuration created in a workspace. For example, Artifact-1 using Connection-1 executing on RemoteEngine-1. Properly named context variables indicate that an appropriate connection and/or resource must be specified during task configuration. |
Plan | Scheduled tasks organized into a combination of sequential and parallel
steps. Plans are created in a workspace, but may reference tasks from other
workspaces in the same environment. Note: Plans
can only be created from Job tasks.
|
Promotion | Provides the mechanism to copy elements from one environment to another.
Once tasks and plans have been verified, the promotion is used to advance the
elements to the next logical environment. Note: Pipelines and personal workspaces
cannot be promoted.
|
Simple Talend Cloud architecture using Talend Studio
- A project in Talend Cloud must be linked to a source repository.
- At least one Cloud Engine or Remote Engine is required.
- Talend Cloud is used to manage operations in all environments.
- A source repository (Talend project) is only relevant to the development environment.
- As the artifacts have already been published toTalend Cloud in the development environment, there is no need for Talend Studio to access any other environments.
- At least one Cloud Engine or Remote Engine is required per environment.
For a more detailed architecture diagram, see the Talend Cloud Getting Started Guide.
Simple Talend Cloud architecture using Talend Cloud Pipeline Designer
- Pipelines are published from Talend Cloud Pipeline Designer to Talend Cloud Management Console.
- Pipelines are available only in the default environment.
- At least one Remote Engine Gen2 is required.
For a more detailed architecture diagram, see the Talend Cloud Getting Started Guide.