How to call Web Services via a Proxy

Irshad Burtally
Talend Data Fabric
Talend Open Studio for ESB
Talend Real-Time Big Data Platform
Talend ESB
Talend MDM Platform
Talend Data Services Platform
Design and Development > Designing Services
Talend Runtime
Talend Studio

How to call Web Services via a Proxy

A DMZ or demiliterized zone is a physical or logical subnetwork that exposes an organisation private network to external-facing services, generally from a larger, untrusted network, like the Internet. A DMZ or demiliterized zone is a physical or logical subnetwork that exposes an organisation private network to external-facing services, generally from a larger, untrusted network, like the Internet. Most businesses will completely prevent Internet access from their production environments. To enable in-house services to invoke external untrusted services, IT will generally configure these in-house services to run in a DMZ or to go through a set of proxy servers from the internal network.

A proxy is used to communicate to services hosted outside the organisation internal private network. Generally, there are three types of proxies in use:

  • A proxy server that passes requests and responses unmodified is usually called a gateway or sometimes a tunnelling proxy.
  • A forward proxy is an Internet-facing proxy used to retrieve from a wide range of sources (in most cases anywhere on the Internet).
  • A reverse proxy is usually an internal-facing proxy used as a front-end to control and protect access to a server on a private network. A reverse proxy commonly also performs tasks such as load-balancing, authentication, decryption and/or caching.

A Web-Proxy has multiple advantages giving control over all communication going out of the network using that proxy.

Invoking a Service through a Proxy from the Talend Studio

It is quite easy and intuitive to invoke services via proxy using Talend Studio. Components like tRestClient, tSoap and tEsbConsumer have built-in proxy server settings. If these components are being used in a Data Integration Batch Job, then the proxy settings should be set within the components in the job. The screenshot below shows the settings for Proxy in the tRestClient component.

The studio is leveraging the CXF code to apply the proxy settings as shown below. It generates ‘conduit’ code for proxy settings. These settings are only applied to the java thread it is running in, and therefore, other components in the job will remain unaffected by this setting. This setting is localised in the component only.

tSetProxy is universal component to set proxy settings in a Java environment. You can use the tSetProxy when the components you are configuring do not have built-in Proxy Settings.

tSetProxy will generate java code that applies to 'System-Wide' properties. It must not be used where it will have impacts on other Java process running on the same server.

Invoking Service Through Proxy from the Talend RuntimeIcon

It is not advised to use ‘tSetProxy’ in your services that will be deployed in the Talend Runtime. The 'tSetProxy' will affect system-wide properties and will impact the behaviour of all the services within the Talend Runtime container including the container itself. For those components like tRestClient, tESBConsumer and tSoap, the component settings for proxy will be ignored as the code generated only works in non-OSGI environment.

To route traffic inside an OSGI container like Talend Runtime, please modify the org.apache.cxf.http.conduits-common.cfg file. This file is located among all the other configuration files in the “etc” folder of the Talend Runtime.

Below is an example of a configuration file to route https request via a proxy. This will selectively apply the proxy to the URL partterns being matched. Hence, you can deploy many services into the same Talend Runtime, and the proxy will be leveraged for only the services call that requires going through the proxy.

	# #%L
	# Talend :: Assembly
	# %%
	# Copyright (C) 2011 - 2012 Talend Inc.
	# %%
	# Licensed under the Apache License, Version 2.0 (the "License");
	# you may not use this file except in compliance with the License.
	# You may obtain a copy of the License at
	#	#
	# Unless required by applicable law or agreed to in writing, software
	# distributed under the License is distributed on an "AS IS" BASIS,
	# See the License for the specific language governing permissions and
	# limitations under the License.
	# #L%
	#Common SSL conduit configuration
	url = https://st.*
	client.ProxyServer =172.##.##.114
	client.ProxyServerPort = 3125