tSQSInput - 6.3

Talend Open Studio for Big Data Components Reference Guide

EnrichVersion
6.3
EnrichProdName
Talend Open Studio for Big Data
task
Data Governance
Data Quality and Preparation
Design and Development
EnrichPlatform
Talend Studio

tSQSInput retrieves one or more messages, with a maximum limit of ten messages, from an Amazon SQS (Simple Queue Service) queue.

tSQSInput properties in Standard Jobs

These properties are used to configure tSQSInput.

The tSQSInput component belongs to the Cloud/Amazon/SQS family.

Basic settings

Use an existing connection

Select this check box and in the Component List click the relevant connection component to reuse the connection details you already defined.

Access Key and Secret Key

Specify the access keys (the access key ID in the Access Key field and the secret access key in the Secret Key field) required to access the Amazon Web Services. For more information about AWS access keys, see Access keys (access key ID and secret access key).

To enter the secret key, click the [...] button next to the secret key field, and then in the pop-up dialog box enter the password between double quotes and click OK to save the settings.

Inherit credentials from AWS role

Select this check box to leverage the instance profile credentials. These credentials can be used on Amazon EC2 instances, and are delivered through the Amazon EC2 metadata service. To use this option, your Job must be running within Amazon EC2 or other services that can leverage IAM Roles for access to resources. For more information, see Using an IAM Role to Grant Permissions to Applications Running on Amazon EC2 Instances.

Assume role

Select this check box and specify the values for the following parameters used to create a new assumed role session.

  • Role ARN: the Amazon Resource Name (ARN) of the role to assume.

  • Role session name: an identifier for the assumed role session.

  • Session duration (minutes): the duration (in minutes) for which we want to have the assumed role session to be active.

For more information about assuming roles, see AssumeRole.

Region

Specify the AWS region by selecting a region name from the list or entering a region between double quotation marks (for example "us-east-1") in the list. For more information about AWS Regions, see AWS Regions and Endpoints.

Queue URL

Specify the URL of the queue from which the messages will be retrieved.

Schema and Edit schema

A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. The schema is either Built-In or stored remotely in the Repository.

The schema of this component is predefined with the following four String type message attributes:

  • MessageId: the ID of the message.

  • ReceiptHandle: the receipt handle of the message.

  • MD5OfBody: the MD5 digest of the message body.

  • Body: the message body.

You can click the button next to Edit schema to view the schema and edit the schema by removing the existing attributes or adding the standard SQS attributes and the customized user attributes.

Read standard attributes

Select this check box to retrieve the standard Amazon SQS attributes along with each message. For more information about the Amazon SQS attributes, see ReceiveMessage > Request Parameters.

Read custom user attributes

Select this check box to retrieve the customized user attributes along with each message.

Custom visibility timeout

Select this check box and in the field displayed specify the visibility timeout (in seconds) for the received messages. If not specified, the overall visibility timeout for the queue will be used for the received messages. For more information, see Visibility Timeout.

Custom wait time

Select this check box and in the field displayed specify the duration (in seconds) for which the call will wait for the message to arrive in the queue before returning.

Delete the messages while streaming

Select this check box to delete the message while retrieving it from the queue.

Read all messages from the queue

Select this check box to retrieve all messages from the queue.

This check box is not available when the Delete the messages while streaming check box is cleared.

Die on error

Select this check box to stop the execution of the Job when an error occurs.

Advanced settings

Config client

Select this check box and in the table displayed specify the client configuration parameters. Click the button below the table to add as many rows as needed, each row for a client configuration parameter, and set the value of the following attributes for each parameter:

  • Client Parameter: click the cell and select a parameter from the drop-down list.

  • Value: enter the value for the corresponding parameter.

STS Endpoint

Select this check box and in the field displayed, specify the AWS Security Token Service endpoint where session credentials are retrieved from.

This check box is available only when the Assume role check box is selected.

tStatCatcher Statistics

Select this check box to gather the Job processing metadata at the Job level as well as at each component level.

Global variables

NB_LINE

The number of messages processed. This is an After variable and it returns an integer.

ERROR_MESSAGE

The error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is cleared, if the component has this check box.

Usage

Usage Rule

This component is usually used as a start component of a Job or Subjob and it always needs an output link.

Retrieving messages from an Amazon SQS queue

Here's an example of using Talend components to retrieve all messages from an existing Amazon SQS (Simple Queue Service) queue and then delete the empty queue.

Creating a Job for retrieving Amazon SQS messages

Create a Job to open a connection to access Amazon SQS, then retrieve all messages from an existing Amazon SQS queue, finally delete the queue.

Prerequisites: You must deliver the following message creator attribute and message body data to the Amazon SQS queue named talend in order to replicate this scenario.

#Creator;Body
Talend DOC Team;the 1st message from Talend DOC team
Talend DEV Team;the 2nd message from Talend DEV team
Talend QA Team;the 3rd message from Talend QA team
  1. Create a new Job and add a tSQSConnection component, a tSQSInput component, a tLogRow component, and a tSQSQueueDelete component by typing their names in the design workspace or dropping them from the Palette.

  2. Link the tSQSInput component to the tLogRow component using a Row > Main connection.

  3. Link the tSQSConnection component to the tSQSInput component using a Trigger > OnSubjobOk connection.

  4. Link the tSQSInput component to the tSQSQueueDelete component using a Trigger > OnSubjobOk connection.

Opening a connection to access Amazon SQS

Configure the tSQSConnection component to open a connection to access Amazon SQS.

  1. Double-click the tSQSConnection component to open its Basic settings view.

  2. In the Access Key and Secret Key fields, specify the authentication information required to access Amazon SQS.

  3. Select an AWS region available for Amazon SQS from the Region drop-down list. In this example, it is Asia Pacific (Tokyo).

Retrieving messages from an Amazon SQS queue

Configure the tSQSInput component and the tLogRow component to retrieve all messages from an existing Amazon SQS queue and display the retrieved messages on the console of Talend Studio.

  1. Double-click the tSQSInput component to open its Basic settings view.

  2. Specify the connection details required to access Amazon SQS. In this example, select the Use an existing connection check box and from the Component List drop-down list displayed, select the connection component to reuse its connection details you have already defined.

  3. In the Queue URL field, enter the name for the queue from which the messages will be retrieved. In this example, it is talend.

  4. Click the button next to Edit schema to open the schema dialog box.

    You can see the schema of the tSQSInput component is predefined with four columns of String type: MessageId, ReceiptHandle, MD5OfBody and Body.

  5. Click the button to add five columns of String type, including four Amazon SQS attributes ApproximateFirstReceiveTimestamp, ApproximateReceiveCount, SenderId and SentTimestamp and one customized attribute Creator.

  6. Click OK to save the changes and close the schema dialog box.

  7. Select the Read standard attributes and Read custom user attributes check boxes to retrieve the value of four Amazon SQS attributes and one customized user attribute along with each message.

  8. Double-click the tLogRow component to open its Basic settings view, and then select Table (print values in cells of a table) in the Mode area for better readability of the result.

Deleting an Amazon SQS queue

Configure the tSQSQueueDelete component to create an Amazon SQS queue.

  1. Double-click the tSQSQueueDelete component to open its Basic settings view.

  2. Specify the connection details required to access Amazon SQS. In this example, select the Use an existing connection check box and from the Component List drop-down list displayed, select the connection component to reuse its connection details you have already defined.

  3. In the Queue field, enter the name for the queue to be deleted. In this example, it is talend.

Executing the Job to retrieve Amazon SQS messages

After setting up the Job and configuring the components used in the Job for retrieving Amazon SQS messages, you can then execute the Job and verify the Job execution result.

  1. Press Ctrl + S to save the Job and then F6 to execute the Job.