Development and Operational Management in Talend Cloud

author
Talend Documentation Team
EnrichVersion
Cloud
EnrichProdName
Talend Cloud
task
Design and Development
Administration and Monitoring > Managing projects
Administration and Monitoring > Managing users
EnrichPlatform
Talend Studio
Talend Management Console

Overview of Talend Cloud

This article describes the internal Talend Cloud structure and which elements serve to manage development and day-to-day operations.

The article consists of the following sections:
  1. Development in Talend Cloud
  2. Operations in Talend Cloud
  3. Simple Talend Cloud architecture
The diagram below represents the different elements that exist in Talend Cloud and how they relate to one another. They have been grouped into development and operational areas.
Relationships are shown using crow’s foot notation.

Leveraging development and operational elements in Talend Cloud enables you to manage your Software Development Life Cycle within a single cloud application.

Development in Talend Cloud

Talend Cloud consists of elements that centrally define and manage development projects and provide access to those projects.

When a user logs in to Talend Studio this information is interpreted, and the appropriate projects are available to the user. The elements that facilitate this include:
Element Description
User Identified by an email address created to access Talend Cloud.
Role Permissions are grouped together to define a role. A list of predefined roles already exists, but custom roles can also be created. At least one role must be assigned to a user to allow access to Talend Cloud.
Group A collection of users.
Project Provides access to a source repository. To give access to a project, users and groups are assigned as collaborators. The user's roles determine the project privileges.
Note: Projects are not available for Talend Cloud Pipeline Designer.
The diagram below provides a conceptual example of a project in Talend Cloud using the elements outlined above.

Once a user has been given Operator and Studio Integration Developer roles and has been added to the project as a collaborator, they will see the project in Talend Studio.

Operations in Talend Cloud

Talend Cloud provides the elements to centrally define, manage, orchestrate, execute, and promote artifacts published from Talend Studio or Talend Cloud Pipeline Designer.

The elements that facilitate operations include:

Element Description
Environment A self-contained space with the required resources that allow the execution of tasks in isolation, without any impact on other environments.
Cloud Engine A compute resource managed by Talend in the cloud that executes tasks. A Cloud Engine is available to an environment.
Remote Engine A compute resource managed by Talend in the cloud that executes tasks. Remote Engines are created for an environment.
Remote Engine Cluster Remote Engines can be grouped together. Any number of clusters can be created in an environment.
Note: Remote Engines added to a cluster cannot be used to execute tasks directly from Talend Studio.
Remote Engine for Pipelines A compute resource managed by Talend in the cloud that executes tasks. Remote Engines for Pipelines are available only in the default environment.
Run profiles Run profiles contain the set of configurations for executing data streams pipelines on Remote Engine for Pipelines. You can define the right resource allocation and advanced properties used at runtime in the run profiles.
Workspace Enables selected users to work together preparing tasks and resources for execution. Any number of workspaces can be created in an environment. Workspaces are used to share tasks, connections, and resources. Users can be added to a workspace with predefined permissions. A user's roles have no impact in a workspace. Groups cannot be added to a workspace.
Artifact A Job, Route, Data Service created in Talend Studio or Pipeline created in Talend Cloud Pipeline Designer and published to Talend Cloud. Artifacts are published to a workspace.
Connection Predefined parameter values to access external systems. Connections are created in a workspace.
Resource A cloud data source to store files and directories. Resources are created in a workspace.
Task An artifact with a specific execution configuration created in a workspace. For example, ArtifactA using Connection1 executing on RemoteEngine1. Properly named context variables indicate that an appropriate connection and/or resource must be specified during task configuration.
Plan Scheduled tasks organized into a combination of sequential and parallel steps. Plans are created in a workspace, but may reference tasks from other workspaces in the same environment.
Promotion Provides the mechanism to copy elements from one environment to another. Once tasks and plans have been verified, the promotion is used to advance the elements to the next logical environment.
Note: Pipelines and personal workspaces cannot be promoted.
The diagram below provides a conceptual example of an environment in Talend Cloud using Talend Studio and the elements outlined above.

Simple Talend Cloud architecture using Talend Studio

The following diagram outlines a basic Talend Cloud development architecture:
Key points:
  • A project in Talend Cloud must be linked to a source repository.
  • At least one Cloud Engine or Remote Engine is required.
The next diagram shows a sample architecture with a Development, Test, and Production environment.
Key points:
  • Talend Cloud is used to manage operations in all environments.
  • A source repository (Talend project) is only relevant to the development environment.
  • As the artifacts have already been published toTalend Cloud in the development environment, there is no need for Talend Studio to access any other environments.
  • At least one Cloud Engine or Remote Engine is required per environment.

For a more detailed architecture diagram, see the Talend Cloud Getting Started Guide.

Simple Talend Cloud architecture using Talend Cloud Pipeline Designer

The following diagram outlines a basic Talend Cloud development architecture with Talend Cloud Pipeline Designer:
Key points:
  • Pipelines are published from Talend Cloud Pipeline Designer to Talend Cloud Management Console.
  • Pipelines are available only in the default environment.
  • At least one Remote Engine for Pipelines is required.

For a more detailed architecture diagram, see the Talend Cloud Getting Started Guide.