tHTTPClient Standard properties

Property type

Guess schema

Click this button to retrieve the schema according to your settings. This button works when Status, headers and body is selected from the Returned content drop-down list or Output key/value pairs is selected and key/value pairs are set in the table under the Output key/value pairs option.

When retrieving data from an HTTP server, you can specify the format and content of the data retrieved using the schema, the Guess schema button, the Response body format option, the Returned content option, the Extract a sub-part of the JSON option, and the Output key/value pairs option. For more information, see tHTTPClient: configuration and output.

Schema and Edit Schema

A schema is a row description, it defines the number of fields to be processed and passed on to the next component. The schema is either Built-in or stored remotely in the Repository.

• Built-in: You create and store the schema locally for this component only. For more information about a component schema in its Basic settings tab, see Basic settings tab.

• Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs. For more information about a component schema in its Basic settings tab, see Basic settings tab.

The component generates the schema automatically according to the related settings of the component. However, you need to define the schema manually if you select Body from the Returned content drop-down list and JSON from the Response body format drop-down list. In this case, you need to add a column for each of the first-level attributes and set the column type to String for attributes with nested JSON objects.

{
    "id": 1,
    "name": "Talend",
    "address": {
       "city": "San Mateo",
       "state": "California",
       "zipcode": "94402"
    }
}

You need to add these columns in the schema: id (type Int), name (type String), and address (type String). The component can then extract zipcode from the address column using Data Shaping Selector Language (DSSL) (for example, input.address.zipcode).

By default, the columns in the schema depend on the settings of the Response body format and the Returned content options as follows.

• When the Status, headers, and body entry is selected from the Returned content drop-down list, the schema contains three columns: body (type String), headers (type String), and status (type Int).
• When the Body entry is selected from the Returned content drop-down list and TEXT is selected from the Response body format drop-down list, the schema contains one column: body (type String).

When retrieving data from an HTTP server, you can specify the format and content of the data retrieved using the schema, the Guess schema button, the Response body format option, the Returned content option, the Extract a sub-part of the JSON option, and the Output key/value pairs option. For more information, see tHTTPClient: configuration and output.

Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

• View schema: choose this option to view the schema only.

• Change to built-in property: choose this option to change the schema to Built-in for local changes.

• Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion.

  If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the Repository Content window.

Sync columns

Click this button to retrieve the schema from the preceding component.

Base URL

Type in the base URL address of the HTTP server you want to access, for example, https://www.example.com/v1.0/. To access a Web service provided by an HTTP server, you need to provide both the base URL address and the path. You can provide the path in the Path field.

You can also define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:

Base URL = "https://{.input.job_url}" and Path = "{.input.job_url_path}"

Authentication type

Select one of the following authentication methods depending on the server security requirements:

• No authentication: no authentication is expected to access the server.
• Basic: A username and a password are expected. For more information, see RFC 2617.
• Digest: A username and a password are expected. For more information, see RFC 2617.
• Bearer token: A raw access token is expected. It will be carried into the HTTP Request header as Authorization: Bearer <your token>.
• NTLM: A username (that may contain a domain name) and a password are expected. For more information, see NT LAN Manager (NTLM) Authentication Protocol.
• API key uses a flexible way to pass an API key token to the server, including the ability to select where to pass it, its name and a prefix:
  • Destination: Select where the token will be set: either in an HTTP header with the given name, or in an HTTP query parameter with the given name (not recommended as the token might be visible in the logs).
  • Name: Enter the name of the header or query parameter.
  • Prefix (optional): Enter the prefix to be added in front of the token (only if the Destination is Request header).
  • Token: Enter the authentication token.
• OAuth 2.0 automatically manages the retrieval and renewal of the access token against the OAuth server then passes it to the target endpoint as a bearer token:
  • Flow: The OAuth flow you want to execute. Currently, only the Client Credentials flow is supported.
  • Authentication mode: For all authentication methods supported, the flow and scope parameters are set in the body in the format application/x-www-form-urlencoded, with keys grant_type=xxx&scope=xxxx.
  • Token endpoint: Enter the authentication token using the format oauth2/mydomain.com/token.
  • Client ID and Client secret: Enter the client ID and client secret.
  • Additional parameters: Enter the additional attributes you want to add, like the scope attribute for example.

HTTP method

Select one of these HTTP methods to specify the action to be performed: GET, POST, HEAD, OPTIONS, PUT, PATCH, DELETE, or TRACE.

To select an HTTP method, click the [...] button next to this field and select the desired HTTP method in the dialog box that appears.

Path

Enter the second part of the URL in order to complete the base URL filled in the Base URL field. Base URL and Path are joined with the character / when needed. For example, to search data from the server with the URL of https://jira.talendforge.org/rest/api/2/, type in search in this field.

Refer to the description on the Base URL field for related information.

You can also define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:

Base URL = "https://{.input.job_url}" and Path = "{.input.job_url_path}"

Path parameters

Select this check box to specify extra parameters needed to complete the base URL or the path in the form of name-value pairs.

If the base URL or path contains placeholders, you can add parameters in this table and set the parameter name and parameter value in the following way to replace the placeholders.

• Name: Enter the name of the placeholder to be replaced.

• Value: Enter the value you want to replace the placeholder with. It can be a static value (for example, contactEntity), a value extracted from the incoming record (for example, {.input.entity.id}), or a mixture of both (for example, version{.input.api.version}).

For example, with the base URL https://www.example.com and the path /{api_version}, you can set the path by adding a parameter in this table, setting Name to api_version and Value to v1.0.

Query parameters

Example: Query parameter name = entityId and query parameter value = UUID-1234567

Request headers

Example: Header name = Content-Type and header value = text/html;charset=utf-8

Request body

Select this option if you want to include a message body in the request.

• Text: Enter the body of the message as plain text in the text frame under the drop-down list. You can define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:

  id={.input.user.id}
  name={.input.user.name}

• JSON: Enter the body of the message in the JSON format in the text frame under the drop-down list. You can define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:

  {
   "id": "{.input.user.id}",
   "name": "{.input.user.name}",
   "age": "{.input.user.age}",
   }

• XML: Enter the body of the message in the XML format in the text frame under the drop-down list. You can define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record. Example:

  <user>
   <id>{.input.user.id}</id>
   <name>{.input.user.name}</name>
   <age>{.input.user.age}</age>
  </user>

• Form data and x_www_urlencoded: Create the body using a multipart form with attribute name-value pairs or using a URL encoded form with attribute name-value pairs. Enter the attribute names and values in rows of the table under the drop-down list. You can define a placeholder using the Data Shaping Selector Language syntax to have some parts dynamically filled with the value extracted from the incoming record.

The Body type drop-down list is available when you select the Request body option.

Response body format

Select the response body format from the drop-down list. The right format allows the connector to parse and apply operations to the response. Currently, text and JSON formats are supported.

• Text: Select this format to return a plain text message. In that case the response payload is not parsed and it will not be possible to extract a sub-part from it.
• JSON: Select this format if the response received has a JSON format. In that case the payload is translated into correctly parsed JSON records.

When retrieving data from an HTTP server, you can specify the format and content of the data retrieved using the schema, the Guess schema button, the Response body format option, the Returned content option, the Extract a sub-part of the JSON option, and the Output key/value pairs option. For more information, see tHTTPClient: configuration and output.

Returned content

Select one of the following two options depending on the data returned by the server.

• Body: If Text is selected from the Response body format drop-down list, a record with one string attribute named body is generated and all the payload is copied in it; If JSON is selected from the Response body format drop-down list, a record that represents the hierarchical structure of the parsed document is generated.

• Status, headers, and body: A record is generated with a status attribute (INT type) that contains the HTTP status code, a headers attribute that is a record with all response headers as nested attributes, and a body attribute that is either a simple string containing the full payload as text (if Text is selected from the Response body format drop-down list) or a hierarchical object representing the parsed document (if JSON is selected from the Response body format drop-down list).

When retrieving data from an HTTP server, you can specify the format and content of the data retrieved using the schema, the Guess schema button, the Response body format option, the Returned content option, the Extract a sub-part of the JSON option, and the Output key/value pairs option. For more information, see tHTTPClient: configuration and output.

Extract a sub-part of the JSON

Enter the path of a node to select a sub-element of the response. If the element is an array, each element of the array will be looped over. For more information about the syntax for entering the node name, see Data Shaping Selector Language syntax.

This field is optional and must remain empty to retrieve the entire JSON response.

{
   "description": "List of some famous geologists",
   "content_length": 5,
   "author": "A. Smith",
   "content":[
      {
         "name":"Maurice Krafft",
         "age":35,
         "female":false,
         "address":{
            "city":"Mulhouse",
            "zipcode":68100
         }
      },
      {
         "name":"Katia Krafft",
         "age":49,
         "female":true,
         "address":{
            "city":"Soultz-Haut-Rhin",
            "zipcode":6860
         }
      },
      {
         "name":"Pline l'Ancien",
         "age":55,
         "female":false,
         "address":{
            "city":"Côme",
            "zipcode":22100
         }
      },
      {
         "name":"James Hutton",
         "age":70,
         "female":false,
         "address":{
            "city":"Édimbourg",
            "zipcode":0
         }
      },
      {
         "name":"Aline Peltier",
         "age":40,
         "female":false,
         "address":{
            "city":"Saint-Pierre",
            "zipcode":97410
         }
      }
   ]
}

{"name":"Katia Krafft","age":49.0,"female":true, ...
{"name":"Pline l'Ancien","age":55.0,"female":false, ...
{"name":"James Hutton","age":70.0,"female":false, ...

|=-----------------=|
|field              |
|=-----------------=|
|Soultz-Haut-Rhin   |
|Côme               |
|Édimbourg          |
'-------------------'

When retrieving data from an HTTP server, you can specify the format and content of the data retrieved using the schema, the Guess schema button, the Response body format option, the Returned content option, the Extract a sub-part of the JSON option, and the Output key/value pairs option. For more information, see tHTTPClient: configuration and output.

This field is available if you select JSON from the Response body format drop-down list.

Output key/value pairs

Select this option to return key/value pairs instead of the raw body of the HTTP response. To enter a value for a node, add a row in the table by clicking the plus button under the table, enter the name of the node in the Name field, and enter the value in the Value field.

The value can be from the component input or from the HTTP response. Enter "{.input.<dssl_path>}" in the Value field if the value is from the component input or "{.response.<dssl_path>}" if the value is from the HTTP response.

When retrieving data from an HTTP server, you can specify the format and content of the data retrieved using the schema, the Guess schema button, the Response body format option, the Returned content option, the Extract a sub-part of the JSON option, and the Output key/value pairs option. For more information, see tHTTPClient: configuration and output.

This option is available when Output key/value pairs is selected.

Select this option to save the attachments of the response as a file in a specified directory. This option works when the response body is a MIME multipart with attachments. You need to type in the path to the directory in the Directory to save field. If the directory specified does not exist, the component will try to create it. Make sure you have the permission to write the directory specified.

For a MIME multipart response with attachments, the first part that is not an attachment in the body will be treated as the output record body; the attachment part is marked with Content-Disposition=attachment;... header and the attachment filename is taken from its `name=...` value. If the name field is not provided in header metadata, the file will be named using the service name (that is, the Path value). If no part has the Content-Disposition=attachment header, the first part of the multipart is treated as the body for the output record and other parts are treated as the attachments.

tStatCatcher Statistics

Select this check box to gather the Job processing metadata at a Job level and at each component level.

Connection timeout (ms)

Set the timeout period (in milliseconds) for the connector to establish a connection to the server. An error occurs if an attempt to establish a connection fails.

Receive timeout (ms)

Set the timeout period (in milliseconds) for receiving response data. An error occurs if no data is received when the read timeout period expires.

Bypass server certificate validation

Select this option to prevent the client from validating the server certificate.

This option is not recommended in production environments.

Use a proxy

Select this option to use an HTTP or a SOCKS proxy.

• Proxy type: Select the type of proxy you want to use: HTTP or SOCKS. The HTTP proxy support basic authentication.
• Proxy host and Proxy port: Enter the address and port of the proxy.
• Proxy login and Proxy password: Enter the credentials needed to authenticate to the proxy. These two fields are available when HTTP is selected from the Proxy type drop-down list.

Accept redirections

Select this option to apply the HTTP Redirection rules on your resources.

• Maximum number of redirects: Set the maximum number of redirects that the connector should follow. If there are more redirections than configured, the last one is returned as the result of the query.
• Redirect only on same host: Enable this option if you want redirections to be performed only when using the same host.

Note that pagination is only compliant with JSON payload and the desired elements must be in an array in the JSON payload.

You need to set the following options for pagination to work properly.

• Preset: Select preset configuration for specific service that supports pagination strategy by selecting the corresponding service. To select a service, click the [...] button next to this field and select the service. You can select one of the following services.
  • BOX: see Offset-based Pagination for more information.
  • JIRA: see JIRA Server platform REST API reference for more information.
  • OData: see OData Version 4.01. Part 1: Protocol for more information.
• Load selected preset: Click this button to load the preset configuration selected in the Preset field.
• Pagination strategy: Select the pagination strategy to be used. Currently, only Offset/limit strategy is available. The offset/limit strategy retrieves the elements starting at a specific position in pages of a specific size. The offset parameter specifies the beginning element in terms of the offset from the first element; the limit parameter specifies the maximum size of a page in terms of the number of the elements. With Pagination strategy applied, a query stops when the last call returns 0 elements.
• Location: Specify the way to set the offset and the limit parameters, which can be Query parameters or In headers.
• Name of the offset: Set the name of the parameter that will be used as the offset.
• Value of the offset: Set the offset. To start from the first element, set it to 0. You can obtain the offset of a query from the incoming records using the DSSL expression '{.input.last_page_info.offset}'.
• Name of the limit: Set the name of the parameter that defines the maximum number of elements allowed in a page.
• Value of the limit: Set the limit, the maximum number of elements that can be returned in a page. The pagination will stop when a page is returned with 0 elements.
• Path to elements: Set the path to the JSON array that contains the desired elements. It must be a JSON array. With offset/limit strategy applied, the element specified by this parameter becomes the root regardless of the setting in Extract a sub-part of the response option. For instance, with JIRA in the Preset field, setting Path to elements to .issue retrieves the issues even if Extract a sub-part of the response is set to .key (assuming that key is an attribute of the elements present in the issues).

• Content type: Set the content type of the file defined by HTTP. See https://www.w3.org/Protocols/rfc1341/4_Content-Type.html for related information.
• Encoding: Set the file encoding.
• Path to file: Enter the full path to the file to be uploaded, including the file name.
• Attachment name: Set the HTTP-formed ID of the attachment.

The Add attachments option is available when you select the Request body option and select Form data from the Body type drop-down list in the Basic settings view.

Force JSON numbers to "Double" type in response body

Select this option if you want to force all JSON numbers to be converted to Double type. Differentiating between Integer and Double numbers is not always possible in JSON.

For instance, a geographic location matrix, that should contain only Double, may have one of its item inferred as Integer "[47, -1.567075]". It is error prone, since the connector will try to generate an Integer array, and then will try to add a Double inside. Forcing all numbers to Double will prevent these errors.

This option is available when JSON is selected from the Response body format drop-down list in the Basic settings view.

Global Variables

ERROR_MESSAGE: the error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is cleared, if the component has this check box.

NB_LINE: the number of rows read by an input component or transferred to an output component. This is an After variable and it returns an integer.

Usage rule

This component can be used as intermediate step in a data flow or as an end object in the Job flowchart.