AWS PrivateLink with Talend Cloud Data Catalog

Talend provides support for AWS PrivateLink to protect the communication between your AWS VPCs (Virtual Private Clouds) and Talend Cloud Data Catalog.

If you need more details about AWS PrivateLink, see https://aws.amazon.com/privatelink on the AWS official site.

Support for PrivateLink allows you to import or export metadata without going through the public Internet, thus to avoid exposing your own services and the services on Talend Cloud Data Catalog publicly. This significantly reduces the exposure risk.

Architecture of the support for PrivateLink with Talend Cloud Data Catalog

Beta

PrivateLink enables a highly secured network between Talend Cloud Data Catalog and your AWS VPCs.

Support for AWS PrivateLink is available in all Talend Cloud Data Catalog regions.

Remote metadata harvesting servers deployed on premises

The metadata harvesting servers are installed on a remote machine behind the firewall of your organization. Your private network or VPC uses PrivateLink to connect to Talend Cloud Data Catalog while the remote harvesting servers use AWS Direct Connect links or VPNs to connect to this private network or VPC.

Orange arrow: PrivateLink connections.
Red arrow: AWS Direct Connect or VPN.
Blue arrow: metadata in transit.

Remote metadata harvesting servers deployed on Cloud

The remote metadata harvesting servers are installed on the client private network or client VPC. These harvesting servers are connected to Talend Cloud Data Catalog and the supported metadata source technologies via AWS PrivateLink.

Orange arrow: PrivateLink connections.
Blue arrow: metadata in transit.

Requirements to use AWS PrivateLink with Talend Cloud Data Catalog

Support for AWS PrivateLink on Talend Cloud Data Catalog is available only on request. If you wish to use this feature, reach out to your support contact. Before sending the request, ensure that support for PrivateLink is available on your AWS platform.

Working with Talend Cloud Data Catalog and PrivateLink across AWS regions

Beta

While AWS PrivateLink is applicable to VPCs in a same AWS region only, you can enable multi-regional use case by implementing cross-regional VPC-Peering.

This implementation empowers you to leverage Talend services even from regions not yet covered while still keeping a strong security posture.

Procedure

As described in this AWS documentation, enable VPC Peering to a region where the remote harvesting servers operate.
Example
Use either of the following ways to configure DNS for VPC peering.
- In Amazon Route 53, create a private hosted zone overlapping Talend cloud domains, <env>.cloud.talend.com. Associate this zone to your VPC, then in this private hosted zone, create a wildcard (*) record of type A (meaning an Alias record) to match all the hostnames of a given Talend environment, for example, the record name could be *.us.cloud.talend.com and in the field for the resource you want to route traffic to, specify the private IP address for PrivateLink.
  
  For further information about a Amazon private hosted zone, see this AWS documentation.
- Configure the DNS on the EC2 cluster that hosts the VPC with PrivateLink, so that this VPC uses the DNS Forwarder to properly respond DNS queries to direct flows over the PrivateLink connections.
For technical details of this configuration, contact the network administration team of your organization.

Activating or deactivating a PrivateLink connection with Talend Cloud Data Catalog

Beta

Talend team and your organization need to work together to activate or deactivate a PrivateLink connection.

Activating AWS PrivateLink with Talend Cloud Data Catalog

Beta

Procedure

In your AWS VPC, create the endpoint to be used for PrivateLink and ensure to enable private DNS names for this endpoint. If you need assistance to do this, contact the administrator of your AWS system.

Example

Once done, in the Details tab of this PrivateLink endpoint, the following Talend specific DNS names appear:

<env>. cloud.talend.com
*.<env>. cloud.talend.com

Depending on the region of your Talend service, the value of this <env> varies, for example, it could be us.

The service name distributed to this PrivateLink endpoint is also Talend specific, depending on the region of Talend Cloud Data Catalog to be used:

Talend Cloud Data Catalog AWS regions	Talend specific PrivateLink service names	Disaster Recovery URLs
EU	com.amazonaws.vpce.eu-central-1.vpce-svc-0c634141c378efbe1	Not available for beta
US	com.amazonaws.vpce.us-east-1.vpce-svc-0318a52bd8dd3fa7d	Not available for beta
AP	com.amazonaws.vpce.ap-northeast-1.vpce-svc-06f41393a31a38a16	Not available for beta

Send to Talend a request for PrivateLink pairing with Talend Cloud Data Catalog.
Note that you need to provide Talend with the following information:
- The Endpoint ID of the VPC running the PrivateLink connections to be activated.
- Your AWS account ID.
- The region in which you want to establish PrivateLink connections to Talend Cloud Data Catalog.
Wait for Talend to accept the PrivateLink pairing.

Once receiving your request, Talend sends this request to a verification workflow and eventually accepts the PrivateLink pairing from your VPCs. Then Talend informs you of this update.
Deploy your remote harvesting servers as usual. If they have been already deployed, restart them.
All remote harvesting servers on a same VPC must use PrivateLink. If you want some remote harvesting servers to use PrivateLink and some others not to, use multiple VPCs.

Results

Starting from the date your request is received, the entire process takes up to five business days.

Deactivating AWS PrivateLink with Talend Cloud Data Catalog

Beta

Procedure

Remove PrivateLink pairing settings from your VPCs.
Restart your remote harvesting servers used with PrivateLink.
All remote harvesting servers on a same VPC must use PrivateLink. If you want some remote harvesting servers to use PrivateLink and some others not to, use multiple VPCs.