tHttpRequest - 6.3

Talend Open Studio for Big Data Components Reference Guide

EnrichVersion
6.3
EnrichProdName
Talend Open Studio for Big Data
task
Data Governance
Data Quality and Preparation
Design and Development
EnrichPlatform
Talend Studio

Function

This component sends an HTTP request to the server end and gets the corresponding response information from the server end.

Purpose

The tHttpRequest component allows you to send an HTTP request to the server and output the response information locally.

tHttpRequest properties

Component family

Internet

 

Basic settings

Schema and Edit Schema

A schema is a row description, it defines the number of fields to be processed and passed on to the next component. The schema is either Built-in or stored remotely in the Repository.

Since version 5.6, both the Built-In mode and the Repository mode are available in any of the Talend solutions.

Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

  • View schema: choose this option to view the schema only.

  • Change to built-in property: choose this option to change the schema to Built-in for local changes.

  • Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion. If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the [Repository Content] window.

 

 

Built-in: You create and store the schema locally for this component only. Related topic: see Talend Studio User Guide.

 

 

Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs. Related topic: see Talend Studio User Guide.

 

Sync columns

Click this button to retrieve the schema from the preceding component.

 

URI

Type in the Uniform Resource Identifier (URI) that identifies the data resource on the server. A URI is similar to a URL, but more general.

 

Method

Select an HTTP method to define the action to be performed:

Post: Sends data (for example HTML form data) to the server end.

Get: Retrieves data from the server end.

 

Post parameters from file

Browse to, or enter the path to the file that is used to provide parameters (request body) to the POST method.

 

Write response content to file

Select this check box to save the HTTP response to a local file. You can either type in the file path in the input field or click the three-dot button to browse to the file path.

 

Create directory if not exists

Select this check box to create the directory defined in the Write response content to file field if it does not exist.

This check box appears only when the Write response content to file check box is selected and is cleared by default.

 

Headers

Type in the name-value pair(s) for HTTP headers to define the parameters of the requested HTTP operation.

Key: Fill in the name of the header field of an HTTP header.

Value: Fill in the content of the header field of an HTTP header.

For more information about definition of HTTP headers, please refer to:

en.wikipedia.org/wiki/List_of_HTTP_headers.

 

Need authentication

Select this check box to fill in a user name and a password in the corresponding fields if authentication is needed:

user: Fill in the user name for the authentication.

password: Fill in the password for the authentication.

To enter the password, click the [...] button next to the password field, and then in the pop-up dialog box enter the password between double quotes and click OK to save the settings.

 

Die on error

Select this check box to stop the execution of the Job when an error occurs.

Clear the check box to skip any rows on error and complete the process for error-free rows. When errors are skipped, you can collect the rows on error using a Row > Reject link.

Advanced settings

Set timeout

Select this check box to specify the connect and read timeout values in the following two fields:

  • Connect timeout(s): Enter the connect timeout value in seconds. An exception will occur if the timeout expires before the connection can be established. The value of 0 indicates an infinite time out. By default, the connect timeout value is 30.

  • Read timeout(s): Enter the read timeout value in seconds. An exception will occur if the timeout expires before there is data available for read. By default, the read timeout value is 0, which indicates an infinite time out.

 

tStatCatcher Statistics

Select this check box to gather the Job processing metadata at a Job level and at each component level.

Global Variables

ERROR_MESSAGE: the error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is cleared, if the component has this check box.

CONNECTED: the result of whether a connection to the server established. This is an After variable and it returns a boolean.

RESPONSE_CODE: the response code returned by the remote HTTP server. This is an After variable and it returns an integer.

A Flow variable functions during the execution of a component while an After variable functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl + Space to access the variable list and choose the variable to use from it.

For further information about variables, see Talend Studio User Guide.

Usage

This component can be used in sending HTTP requests to server and saving the response information. This component can be used as a standalone component.

Log4j

If you are using a subscription-based version of the Studio, the activity of this component can be logged using the log4j feature. For more information on this feature, see Talend Studio User Guide.

For more information on the log4j logging levels, see the Apache documentation at http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/Level.html.

Limitation

N/A

Scenario 1: Sending a HTTP request to the server and saving the response information to a local file

This scenario describes a two-component Job that uses the GET method to retrieve information from the server end and writes the response to a local file as well as to the console.

Linking the components

  1. In the Integration perspective of the Studio, create a Job from the Job Designs node in the Repository tree view.

    For further information about how to create a Job, see the Talend Studio User Guide.

  2. Drop the following components from the Palette onto the design workspace: tHttpRequest and tLogRow.

  3. Connect the tHttpRequest component to the tLogRow component using a Row > Main connection.

Configuring the GET request

  1. Double-click the tHttpRequest component to open its Basic settings view and define the component properties.

  2. Fill in the URI field with "http://192.168.0.63:8081/testHttpRequest/build.xml". Note that this URI is for demonstration purposes only and it is not a live address.

  3. From the Method list, select GET.

  4. Select the Write response content to file check box and fill in the input field on the right with the file path by manual entry, D:/test.txt for this use case.

  5. Select the Need authentication check box and fill in the user and password, both tomcat in this use case.

Executing the Job

Then you can run this Job.

The tLogRow component is used to present the execution result of the Job.

  1. If you want to configure how the result is presented by tLogRow, double-click the component to open its Component view and in the Mode area, select the Table (print values in cells of a table) check box.

  2. Press F6 to run this Job.

Once done, the response information from the server is saved and displayed.

Scenario 2: Sending a POST request from a local JSON file

In this scenario, a four-component Job is used to read parameters from a given JSON file and send it in a POST request to a web site.

The JSON file to be used reads as follows:

{"echo":
      [
		{
			"data":"e=hello"
		}
	  ]
}

From that file, tFileInputJSON reads the e parameter and its value hello and tHttpRequest sends the pair to http://echo.itcuties.com/, an URL provided for demonstration by an online programming community, www.itcuties.com.

Note that the e parameter is required by http://echo.itcuties.com/.

Linking the components

  1. In the Integration perspective of the Studio, create an empty Job, named httpRequestPostDemo for example, from the Job Designs node in the Repository tree view.

    For further information about how to create a Job, see the Talend Studio User Guide.

  2. Drop tFileInputJSON, tFileOutputDelimited, tHttpRequest and tLogRow onto the workspace.

  3. Connect tFileInputJSON to tHttpRequest using the Trigger > On Subjob Ok link.

  4. Connect the other components using the Row > Main link.

Reading the JSON file

  1. Double-click tFileInputJSON to open its Component view.

  2. Select JsonPath without loop from the Read By drop-down list.

  3. Click the [...] button next to Edit schema to open the schema editor.

  4. Click the [+] button to add one row and name it, for example, to data.

  5. Click OK to validate these changes and accept the propagation prompted by the pop-up dialog box.

  6. In the Filename field, browse, or enter the path to the source JSON file in which the parameter to be sent is stored.

  7. In the Mapping table, the data column you defined in the previous step in the component schema has been automatically added. In the JSONPath query column of this table, enter the JSON path, in double quotation marks, to extract the parameter to be sent. In this scenario, the path is echo[0].data.

Writing the parameter to a flat file

  1. Double-click tFileOutputDelimited to open its Component view.

  2. In the File name field, browse, or enter the path to the flat file in which you want to write the extracted parameter. This file will be created if it does not exist. In this example, it is C:/tmp/postParamsFile.txt.

Posting the parameter

  1. Double-click tHttpRequest to open its Component view.

  2. In the URI field, enter the server address to which the parameter is to be sent. In this scenario, it is http://echo.itcuties.com/.

  3. From the Method list, select POST.

  4. In the Post parameters from file field, browse, or enter the path to the flat file that contains the parameter to be used. As defined earlier with the tFileOutputDelimited component, this path is C:/tmp/postParamsFile.txt.

Executing the Job

Press F6 to run this Job.

The tLogRow component is used to present the execution result of the Job.

Once done, the Run view is opened automatically, where you can check the execution result.

You can read that the site receiving the parameter returns answers.