tS3Put Standard properties - Cloud - 8.0

Amazon S3

Version
Cloud
8.0
Language
English
Product
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Real-Time Big Data Platform
Module
Talend Studio
Content
Data Governance > Third-party systems > Amazon services (Integration) > Amazon S3 components
Data Quality and Preparation > Third-party systems > Amazon services (Integration) > Amazon S3 components
Design and Development > Third-party systems > Amazon services (Integration) > Amazon S3 components
Last publication date
2024-02-20

These properties are used to configure tS3Put running in the Standard Job framework.

The Standard tS3Put component belongs to the Cloud family.

The component in this framework is available in all Talend products.

Basic settings

Use existing connection

Select this check box and in the Component List drop-down list, select the desired connection component to reuse the connection details you already defined.

Credential Provider

Specify the way to obtain AWS security credentials.

  • Static Credentials: Use access key and secret key as the AWS security credentials.
  • Inherit credentials from AWS role: Obtain AWS security credentials from your EMR instance metadata. To use this option, the Amazon EMR cluster must be started and your Job must be running on this cluster. For more information, see Using an IAM Role to Grant Permissions to Applications Running on Amazon EC2 Instances. This option enables you to develop your Job without having to put any AWS keys in the Job, thus easily comply with the security policy of your organization.
  • Web Identity Token: Use Web token for establishing the Amazon S3 connection.
  • Profile credentials or inherit from AWS role: Use credentials stored in a file (known as profile configuration file) as the AWS security credentials or inherit AWS security credentials from the AWS role.

    AWS security credentials can be grouped in profiles in profile configuration files. A profile configuration file can contain one or multiple profiles. You can optionally specify the profile configuration file in the Profile config file path field and specify the profile to be used in the Profile name field.

    If the Profile config file path field and the Profile name filed are left empty, <user folder>/.aws/credentials will be used as the profile configuration file and the profile named default will be used.

    In cases of EC2 instances, if the profile (the specified or the default profile) does not exist, the component will try to inherit the AWS security credentials from the AWS role.

Note:
  • The Profile config file path field and the Profile name field are available when Profile credentials or inherit from AWS role is selected.
  • tRedshiftBulkExec, tRedshiftOutputBulk, and tRedshiftOutputBulkExec support S3 connections that are established with Credential Provider being Static Credentials or Inherit credentials from AWS role only.

Access Key

The Access Key ID that uniquely identifies an AWS Account. For more information, see Managing access keys for IAM users.

Secret Key

The Secret Access Key, constituting the security credentials in combination with the access Key.

To enter the secret key, click the [...] button next to the secret key field, and then in the pop-up dialog box enter the password between double quotes and click OK to save the settings.

Assume Role

If you temporarily need some access permissions associated to an AWS IAM role that is not granted to your user account, select this check box to assume that role. Then specify the values for the following parameters to create a new assumed role session.

Ensure that access to this role has been granted to your user account by the trust policy associated to this role. If you are not certain about this, ask the owner of this role or your AWS administrator.

  • Role ARN: the Amazon Resource Name (ARN) of the role to assume. You can find this ARN name on the Summary page of the role to be used on your AWS portal, for example, this role ARN could read like am:aws:iam::[aws_account_number]:role/[role_name].

    Role session name: enter the name you want to use to uniquely identify your assumed role session. This name can contain upper- and lower-case alphanumeric characters with no spaces. You can also include underscores or any of the following characters: =,.@-.
  • Session duration (minutes): the duration (in minutes) for which you want the assumed role session to be active. This duration cannot exceed the maximum duration which your AWS administrator has set. The duration defaults to 3600 seconds if you give it no value.

For an example about an IAM role and its related policy types, see Create and Manage AWS IAM Roles from the AWS documentation.

Region

Specify the AWS region by selecting a region name from the list or entering a region between double quotation marks (e.g. "us-east-1") in the list. For more information about the AWS Region, see Regions and Endpoints.

Client-Side Encrypt

Select this check box and from the Key type drop-down list displayed, select one of the following three options for encrypting the data on the client-side before sending to Amazon S3. For more information, see Protecting Data Using Client-Side Encryption.
  • KMS-managed customer master key: use a KMS-managed customer master key (CMK) for the client-side data encryption. In the Key field, you need to specify the AWS KMS customer master key ID (CMK ID).

  • Symmetric Master Key: use a symmetric master key (256-bit AES secret key) for the client-side data encryption.

    • Algorithm: select the algorithm associated with the key from the list. By default, there is only one algorithm named AES.

    • Encoding: select the encoding type associated with the key from the list, either Base64 or X509.

    • Key or Key file: specify the key or the path to the file that stores the key.

  • Asymmetric Master Key: use an asymmetric master key (a 1024-bit RSA key pair) for the client-side data encryption.

    • Algorithm: select the algorithm associated with the key from the list. By default, there is only one algorithm named RSA.

    • Public key file: specify the path to the public key file.

    • Private key file: specify the path to the private key file.

Bucket

Specify the name of the bucket, namely the top level folder, on the S3 server.

Key

Specify the path to the file that will hold the uploaded data on the S3 server, using slash (/) to separate folders in the path.

  • If you only specify a file name in this field, the file will be in the bucket.
  • If the path contains folders that do not exist, the folders will be created.

  • If the file already exists, it will be overwritten.

Warning: This field must not start with a slash (/).

File or Stream

The path to the local file to be uploaded or the input stream from which data will be uploaded.

For an example of how to use stream, see Reading data from a remote file in streaming mode.

Server-Side Encryption

Select this check box to enable server-side encryption with Amazon S3-Managed Encryption Keys (SSE-S3) and use the 256-bit Advanced Encryption Standard (AES-256) cipher to encrypt your data.

For more information about the server-side encryption, see Protecting Data Using Server-Side Encryption.

Use KMS

Select this check box to enable server-side encryption with AWS KMS-Managed Keys (SSE-KMS) instead of Amazon S3-Managed Encryption Keys (SSE-S3).

This property is available when the Server-Side Encryption check box is selected.

Customer Master Key

Specify your own customer master key (CMK) that is created in the IAM console using AWS Key Management Service for SSE-KMS encryption. If not specified, the default CMK, which is created the first time you add an SSE-KMS encrypted object to the defined bucket in your region, will be used for SSE-KMS encryption.

This property is available when the Use KMS check box is selected.

Die on error

This check box is cleared by default, meaning to skip the row on error and to complete the process for error-free rows.

Advanced settings

Config client

Select this check box to configure client parameters.

  • Value: Enter the client parameter value.

  • Client Parameter: Select the client parameter from the list.

This check box is not available when Use existing connection is selected in the Basic settings.

STS Endpoint

Select this check box to specify the AWS Security Token Service (STS) endpoint from which to retrieve the session credentials. For example, enter sts.amazonaws.com.

This check box is available only when the Assume Role check box is selected.

Signing region

Select the AWS region of the STS service. If the region is not in the list, you can enter its name between double quotation marks. The default value is us-east-1.

This drop-down list is available only when the Assume Role check box is selected.

External Id

If the administrator of the account to which the role belongs provided you with an external ID, enter its value here. The External Id is a unique identifier that allows a limited set of users to assume the role.

This field is available only when the Assume Role check box is selected.

Serial number

When you assume a role, the trust policy of this role might require Multi-Factor Authentication (MFA). In this case, you must indicate the identification number of the hardware or virtual MFA device that is associated with the user who assumes the role.

This field is available only when the Assume Role check box is selected.

Token code

When you assume a role, the trust policy of this role might require Multi-Factor Authentication (MFA). In this case, you must indicate a token code. This token code is a time-based one-time password produced by the MFA device.

This field is available only when the Assume Role check box is selected.

Tags

List session tags in the form of key-value pairs. You can then use these session tags in policies to allow or deny access to requests.

Transitive: select this check box to indicate that a tag will persist to the next role in a role chain.

For more information about tags, see Passing Session Tags in AWS STS

This field is available only when the Assume Role check box is selected.

IAM Policy ARNs

Enter the Amazon Resource Names (ARNs) of the IAM managed policies that you want to use as managed session policies. Use managed session policies to limit the permissions of the session. The policies must exist in the same account as the role. The resulting session's permissions are the intersection of the role's identity-based policy and the session policies.

For more information about session policies, see the corresponding section in Policies and Permissions

This field is available only when the Assume Role check box is selected.

Policy

Enter an IAM policy in JSON format that you want to use as a session policy. Use session policies to limit the permissions of the session. The resulting session's permissions are the intersection of the role's identity-based policy and the session policies.

For more information about session policies, see the corresponding section in Policies and Permissions

This field is available only when the Assume Role check box is selected.

Multipart threshold (MB)

Specify the multipart threshold (in MB). A file will be uploaded in parts if its size is larger than the multipart threshold. Note that this setting has no effect in the streaming mode.

Part size (Mb)

Specify the part size (in MB) when uploading an object in parts.

Part size is in the range 5MB to 5GB. A value less than 5 uploads the object with the part size of 5MB. For an object that is less than 5GB in size, you can have it uploaded without being divided into parts by setting this option to a large enough value.

For information about S3 multipart upload, see Multipart Upload Overview.

Enable Accelerate Mode

Select this check box to enable fast, easy and secure transfers of files over long distances between your client and an S3 bucket. To take it into account, you should enable this acceleration mode on the S3 bucket in advance.

This option can work as a dynamic parameter and be assigned a value using a context variable.

Access control

Select this check box to set the access control list (ACL) for the current object.

With Access control selected, click the plus button under the Access table to add a new line, where you can add an AWS user account and set permissions for it as follows.
  • Enter the canonical user ID or the Email address of the user in double quotation marks in the Account canonical ID / email column;
  • Select the check box in the Read object column to grant the permission to read the data and metadata of the object in the bucket;
  • Select the check box in the Read object permissions column to grant the permission to read the ACLs of the object in the bucket;
  • Select the check box in the Write object permissions column to grant the permission to write the ACLs of the object in the bucket.
Note:
  • The string in the Account canonical ID / email column is considered as an Email address if it contains @.
  • If Access control is not selected, the user who uploads an object becomes the owner of the object and owns all the permissions over the object.
  • If Access control is selected, the user who uploads an object becomes the owner of the object but owns no permission over the object. You can grant permissions to the owner in the Access table in the way mentioned above.
  • AWS group is not supported.

Canned Access Control

Select an option from the drop-down list to grant a predefined permission for the current resource. See Canned ACL for information about canned ACL.

Enable object lock

Select this option to enable the Amazon S3 Object lock feature.

With this option enabled, you need also to select a retention mode (Governance or Compliance) from the Retention mode drop-down list and set a retention period in the Retention period field.

For information about the Amazon S3 Object lock feature, retention mode, and retention period, see Protecting data with Amazon S3 Object Lock.

Set object tags

Select this check box to set object tags in the form of key-value pairs. These tags will be added to S3 objects when uploading them to the predefined bucket. For more information, see Categorizing your storage using tags.

tStatCatcher Statistics

Select this check box to collect log data at the component level.

Global Variables

ERROR_MESSAGE

The error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is selected.

Usage

Usage rule

This component can be used alone or with other S3 components, e.g. tS3Connection.

Dynamic settings

Click the [+] button to add a row in the table and fill the Code field with a context variable to choose your database connection dynamically from multiple connections planned in your Job. This feature is useful when you need to access database tables having the same data structure but in different databases, especially when you are working in an environment where you cannot change your Job settings, for example, when your Job has to be deployed and executed independent of Talend Studio.

Once a dynamic parameter is defined, the Component List box in the Basic settings view becomes unusable.

For examples on using dynamic parameters, see Reading data from databases through context-based dynamic connections and Reading data from different MySQL databases using dynamically loaded connection parameters. For more information on Dynamic settings and context variables, see Dynamic schema and Creating a context group and define context variables in it.