Configuring the Docker container DNS - Cloud

Talend Remote Engine Gen2 Quick Start Guide

Version
Cloud
Language
English
Product
Talend Cloud
Module
Talend Management Console
Talend Pipeline Designer
Content
Deployment > Deploying > Executing Pipelines
Installation and Upgrade
Last publication date
2024-01-24

If Docker runs on a virtual machine (Docker for Mac, Docker for Windows), the default container DNS configuration differs from the one of the physical machine. In that case you need to configure the DNS in the Docker configuration file provided with the Remote Engine Gen2.

Procedure

  1. Go to the Remote Engine Gen2 installation directory and open the following file to edit it:
    default/docker-compose.yml if you are using the engine in the AWS USA, AWS Europe, AWS Asia-Pacific or Azure regions.

    eap/docker-compose.yml if you are using the engine as part of the Early Adopter Program.

  2. To configure:
    1. dynamic DNS resolution, add name servers in a dns section, or add suffixes in a dns_search section. Suffixes will be appended to host names automatically for DNS resolution. The entries listed in the dns_search section will appear in the /etc/resolv.conf file of each container.

      Example

      dns:
        - 8.8.8.8
        - 9.9.9.9
      dns_search:
        - dc1.example.com
        - dc2.example.com
    2. static DNS resolution, add entries in a extra_hosts section. These entries will be automatically added to the /etc/hosts file of each container.
      Note: Two Remote Engine Gen2 containers, livy and component-server, require external connections so the DNS configuration might be needed for both of them.

      Example

      extra_hosts:
       - "somehost:162.242.195.82"
       - "otherhost:50.31.209.229"
  3. Save the file.

Results

Your docker-compose.yml should look like this:
version: '3.6'

networks: ...

volumes:...

services:
 ...
 component-server:
  image: ${COMPONENTS_IMG}
  ...
  labels:...
  dns:
  - 8.8.8.8
  - 9.9.9.9
  dns_search:
  - dc1.example.com
  - dc2.example.com  
  extra_hosts:
  - "somehost:162.242.195.82"
  - "otherhost:50.31.209.229"
...

For more information on this topic, see the Docker documentation.

Learn more:

Services of Remote Engine Gen2 run as Docker containers on a user-defined bridge network called remote-engine_default.
Tip: You can check the presence of this user-defined bridge by executing command docker network ls.
As of Docker version 1.10, user-defined bridge networks embed a DNS server, see the Docker documentation about embedded DNS servers and user-defined networks for more information.

A number of options specified at run time can affect the DNS service resolution for the Docker container, but in absence of such options the default behavior is that Docker uses the /etc/resolv.conf of the host machine. The host machine is where the Docker daemon runs. In some cases Docker may not run on the physical machine but rather on a virtual machine and this is why you might need to configure the Docker container DNS manually in the docker-compose.yml file.