tSqoopExport Properties - 6.1

Talend Components Reference Guide

EnrichVersion
6.1
EnrichProdName
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for Big Data
Talend Open Studio for Data Integration
Talend Open Studio for Data Quality
Talend Open Studio for ESB
Talend Open Studio for MDM
Talend Real-Time Big Data Platform
task
Data Governance
Data Quality and Preparation
Design and Development
EnrichPlatform
Talend Studio

Component family

Big Data / Sqoop

 

Function

tSqoopExport calls sqoop to transfer data from the Hadoop Distributed File System (HDFS) to a relational database management system (RDBMS).

Please be aware that some features provided by this component are only supported by the latest Sqoop version. For further information about the availability of each feature, see Apache's documentation about Sqoop.

Note

Sqoop is typically installed in every Hadoop distribution. But if the Hadoop distribution you need to use have no Sqoop installed, you have to install one on your own and ensure to add the Sqoop command line to the PATH variable of that distribution. For further information about how to install Sqoop, see the documentation of Sqoop.

Purpose

tSqoopExport is used to define the arguments required by Sqoop for transferring data to a RDBMS.

Basic settings

Mode

Select the mode in which Sqoop is called in a Job execution.

Use Commandline: the Sqoop shell is used to call Sqoop. In this mode, you have to deploy and run the Job in the host where Sqoop is installed. Therefore, if you are a subscription-based user, we recommend installing and using a Jobserver provided by Talend in that host to run the Job; if you are using one of the Talend solutions with Big Data, you have to ensure that the Studio and the Sqoop to be used are in the same machine. For further information about how to install a Jobserver, see Talend Installation Guide.

Use Java API: the Java API is used to call Sqoop. In this mode, the Job can be run locally in the Studio but you need to configure the connection to the Hadoop distribution to be used. Note that JDK is required to execute the Job in the Java API mode and the versions of the JDK kits installed in both machines must be compatible with each other; for example, the versions are the same or the JDK version of the Hadoop machine is more recent.

 

Hadoop properties

Either Built-in or Repository:

  • Built-in: you enter the configuration information of the Hadoop distribution to be used locally for this component only.

  • Repository: you have already created the Hadoop connection and stored it in the Repository; therefore, you reuse it directly for the component configuration and the Job design. For further information about how to create a centralized Hadoop connection, see Talend Big Data Getting Started Guide.

Version

Distribution

Select the cluster you are using from the drop-down list. The options in the list vary depending on the component you are using. Among these options, the following ones requires specific configuration:

  • If available in this Distribution drop-down list, the Microsoft HD Insight option allows you to use a Microsoft HD Insight cluster. For this purpose, you need to configure the connections to the WebHCat service, the HD Insight service and the Windows Azure Storage service of that cluster in the areas that are displayed. A demonstration video about how to configure this connection is available in the following link: https://www.youtube.com/watch?v=A3QTT6VsNoM.

  • The Custom option allows you to connect to a cluster different from any of the distributions given in this list, that is to say, to connect to a cluster not officially supported by Talend.

In order to connect to a custom distribution, once selecting Custom, click the button to display the dialog box in which you can alternatively:

  1. Select Import from existing version to import an officially supported distribution as base and then add other required jar files which the base distribution does not provide.

  2. Select Import from zip to import the configuration zip for the custom distribution to be used. This zip file should contain the libraries of the different Hadoop elements and the index file of these libraries.

    In Talend Exchange, members of Talend community have shared some ready-for-use configuration zip files which you can download from this Hadoop configuration list and directly use them in your connection accordingly. However, because of the ongoing evolution of the different Hadoop-related projects, you might not be able to find the configuration zip corresponding to your distribution from this list; then it is recommended to use the Import from existing version option to take an existing distribution as base to add the jars required by your distribution.

    Note that custom versions are not officially supported by Talend. Talend and its community provide you with the opportunity to connect to custom versions from the Studio but cannot guarantee that the configuration of whichever version you choose will be easy, due to the wide range of different Hadoop distributions and versions that are available. As such, you should only attempt to set up such a connection if you have sufficient Hadoop experience to handle any issues on your own.

    Note

    In this dialog box, the active check box must be kept selected so as to import the jar files pertinent to the connection to be created between the custom distribution and this component.

    For a step-by-step example about how to connect to a custom distribution and share this connection, see Connecting to a custom Hadoop distribution.

 

Hadoop Version

Select the version of the Hadoop distribution you are using. The available options vary depending on the component you are using. Along with the evolution of Hadoop, please note the following changes:

  • If you use Hortonworks Data Platform V2.2, the configuration files of your cluster might be using environment variables such as ${hdp.version}. If this is your situation, you need to set the mapreduce.application.framework.path property in the Hadoop properties table of this component with the path value explicitly pointing to the MapReduce framework archive of your cluster. For example:

    mapreduce.application.framework.path=/hdp/apps/2.2.0.0-2041/mapreduce/mapreduce.tar.gz#mr-framework
  • If you use Hortonworks Data Platform V2.0.0, the type of the operating system for running the distribution and a Talend Job must be the same, such as Windows or Linux. Otherwise, you have to use Talend Jobserver to execute the Job in the same type of operating system in which the Hortonworks Data Platform V2.0.0 distribution you are using is run. For further information about Talend Jobserver, see Talend Installation Guide.

Configuration

NameNode URI

Select this check box to indicate the location of the NameNode of the Hadoop cluster to be used. The NameNode is the master node of a Hadoop cluster. For example, we assume that you have chosen a machine called masternode as the NameNode of an Apache Hadoop distribution, then the location is hdfs://masternode:portnumber.

For further information about the Hadoop Map/Reduce framework, see the Map/Reduce tutorial in Apache's Hadoop documentation on http://hadoop.apache.org.

 

JobTracker Host

Select this check box to indicate the location of the Jobtracker service within the Hadoop cluster to be used. For example, we assume that you have chosen a machine called machine1 as the JobTracker, then set its location as machine1:portnumber. A Jobtracker is the service that assigns Map/Reduce tasks to specific nodes in a Hadoop cluster. Note that the notion job in this term JobTracker does not designate a Talend Job, but rather a Hadoop job described as MR or MapReduce job in Apache's Hadoop documentation on http://hadoop.apache.org.

This property is required when the query you want to use is executed in Windows and it is a Select query. For example, SELECT your_column_name FROM your_table_name

If you use YARN in your Hadoop cluster such as Hortonworks Data Platform V2.0.0 or Cloudera CDH4.3 + (YARN mode), you need to specify the location of the Resource Manager instead of the Jobtracker. Then you can continue to set the following parameters depending on the configuration of the Hadoop cluster to be used (if you leave the check box of a parameter clear, then at runtime, the configuration about this parameter in the Hadoop cluster to be used will be ignored ):

  1. Select the Set resourcemanager scheduler address check box and enter the Scheduler address in the field that appears.

  2. Select the Set jobhistory address check box and enter the location of the JobHistory server of the Hadoop cluster to be used. This allows the metrics information of the current Job to be stored in that JobHistory server.

  3. Select the Set staging directory check box and enter this directory defined in your Hadoop cluster for temporary files created by running programs. Typically, this directory can be found under the yarn.app.mapreduce.am.staging-dir property in the configuration files such as yarn-site.xml or mapred-site.xml of your distribution.

  4. Allocate proper memory volumes to the Map and the Reduce computations and the ApplicationMaster of YARN by selecting the Set memory check box in the Advanced settings view.

  5. Select the Set Hadoop user check box and enter the user name under which you want to execute the Job. Since a file or a directory in Hadoop has its specific owner with appropriate read or write rights, this field allows you to execute the Job directly under the user name that has the appropriate rights to access the file or directory to be processed.

  6. Select the Use datanode hostname check box to allow the Job to access datanodes via their hostnames. This actually sets the dfs.client.use.datanode.hostname property to true. When connecting to a S3N filesystem, you must select this check box.

For further information about these parameters, see the documentation or contact the administrator of the Hadoop cluster to be used.

For further information about the Hadoop Map/Reduce framework, see the Map/Reduce tutorial in Apache's Hadoop documentation on http://hadoop.apache.org.

 Authentication

Use kerberos authentication

If you are accessing the Hadoop cluster running with Kerberos security, select this check box, then, enter the Kerberos principal name for the NameNode in the field displayed. This enables you to use your user name to authenticate against the credentials stored in Kerberos.

In addition, since this component performs Map/Reduce computations, you also need to authenticate the related services such as the Job history server and the Resource manager or Jobtracker depending on your distribution in the corresponding field. These principals can be found in the configuration files of your distribution. For example, in a CDH4 distribution, the Resource manager principal is set in the yarn-site.xml file and the Job history principal in the mapred-site.xml file.

This check box is available depending on the Hadoop distribution you are connecting to.

  Use a keytab to authenticate

Select the Use a keytab to authenticate check box to log into a Kerberos-enabled Hadoop system using a given keytab file. A keytab file contains pairs of Kerberos principals and encrypted keys. You need to enter the principal to be used in the Principal field and the access path to the keytab file itself in the Keytab field.

Note that the user that executes a keytab-enabled Job is not necessarily the one a principal designates but must have the right to read the keytab file being used. For example, the user name you are using to execute a Job is user1 and the principal to be used is guest; in this situation, ensure that user1 has the right to read the keytab file to be used.

 

Hadoop user name

Enter the user name under which you want to execute the Job. Since a file or a directory in Hadoop has its specific owner with appropriate read or write rights, this field allows you to execute the Job directly under the user name that has the appropriate rights to access the file or directory to be processed. Note that this field is available depending on the distribution you are using.

 

JDBC property

Either Built-in or Repository:

  • Built-in: you enter the connection information of the database to be used locally for this component only.

  • Repository: you have already created the database connection and stored it in the Repository; therefore, you reuse it directly for the component configuration and the Job design. For further information about how to create a centralized database connection, see Talend Studio User Guide.

    Note that only the General JDBC connection stored in the Repository is supported.

 

Connection

Type in the JDBC URL used to connect to the target database.

 

User name and Password

Type in the authentication information to the target database.

To enter the password, click the [...] button next to the password field, and then in the pop-up dialog box enter the password between double quotes and click OK to save the settings.

If your password is stored in a file, select the The password is stored in a file check box and enter the path to that file in the File path field that is displayed.

  • This file can be stored either in the machine where the Job is to be executed or in the HDFS system of the Hadoop cluster to be used.

  • The password stored in this file must not contain \n (the newline escape) at the end, that is to say, you must not insert a new line at the end of the password even though this line is empty.

Note that this feature is available depending on the Sqoop version you are using.

 

Driver JAR

In either the Use Commandline mode or the Java API mode, you must add the driver file of the database to be used to the lib folder of the Hadoop distribution you are using. For that purpose, use this Driver JAR table to add that driver file for the current Job you are designing.

 

Table Name

Type in the name of the target table to which data is transferred from HDFS. This table must already exist in the target database. The input files are read and parsed into a set of records according to the user-specified delimiters.

 

Export Dir

Enter the path to the source data to be transferred in HDFS.

Direct

Select this check box to use the export fast path.

 

Specify Number of Mappers

Select this check box to indicate the number of map tasks (parallel processes) used to perform the data transfer.

If you do not want Sqoop to work in parallel, enter 1 in the displayed field.

 

Call a stored procedure

Select this check box to enable the component to call a specific store procedure to write data into the target database.

You need to enter the name of the stored procedure to be used in the field that is displayed.

Syntax and capabilities of stored procedures vary among different databases; for this reason, you are recommended to consult the documentation of the database you want to use for details about how a given stored procedure could be called.

 

Use batch mode

Select this check box to execute the statements in batch mode instead of running a multi-row INSERT statement to write multiple records in the target database.

 

Clear staging table

If you are using a specific staging table for the desired data transfer, select this check box to ensure that the staging table is empty when the data transfer runs.

 

Define a staging table

Select this check box to create a staging table for the data to be transferred. The transferred data is staged within this table before being written into the target table so as to avoid only a part of the data being committed to the target table when the transfer fails.

For further information about whether a staging table is supported for a given data transfer, see Apache's documentation for Sqoop.

 

Specify how updates are performed when new rows are found with non-match keys in database

Select this check box to determine the action to be taken when a given update key does not have any matching records in the target table. You can then select either of the following options:

  • Update only: this updates only the records that already exist in the target table.

  • Allow insert: this works like the SQL UPSERT statement. It writes new records in the table if they do not exist there.

 

Use column for update

Select this check box and in the table that is displayed, add the columns to be used as the update key.

 

Print Log

Select this check box to activate the Verbose check box.

 

Verbose

Select this check box to print more information while working, for example, the debugging information.

Advanced settings

Use MySQL default delimiters

Select this check box to use MySQL's default delimiter set. This check box is available only to the Commandline mode.

 

Define Java mapping

Sqoop provides default configuration that maps most SQL types to appropriate Java types. If you need to use your custom map to overwrite the default ones at runtime, select this check box and define the map(s) you want to use in the table that appears.

 

Additional arguments

Complete this table to use additional arguments if needs be.

By adding additional arguments, you are able to perform multiple operations in one single transaction. For example, you can use --hive-import and --hive-table in the Commandline mode or hive.import and hive.table.name in the Java API mode to create Hive table and write data in at the runtime of the transaction writing data in HDFS. For further information about the available Sqoop arguments in the Commandline mode and the Java API mode, respectively, see Additional arguments.

 

Use speed parallel data transfers

Select this check box to enable quick parallel data transfers between the Teradata database and the Hortonworks Hadoop distribution. Then the Specific params table and the Use additional params check box appear to allow you to specify the Teradata parameters required by parallel transfers.

  • In the Specific params table, two columns are available:

    • Argument: select the parameters as needed from the drop-down list. They are the most common parameters for the parallel transfer.

    • Value: type in the value of the parameters.

  • By selecting the Additional params check box, you make the Specific additional params field displayed. In this field, you can enter the Teradata parameters that you need to use but are not provided in the Specific params table. The syntax for a parameter is -Dparameter=value and when you put more than one parameter in this field, separate them using whitespace.

You must ensure that the Hortonworks Connector for Teradata has been installed in your Hortonworks cluster. The latest connector can be downloaded from the website of Hortonworks and installed by following the explanations from http://hortonworks.com/wp-content/uploads/2014/02/bk_HortonworksConnectorForTeradata.pdf. In the same document, you can as well find the detailed explanations about each parameter that is available for the parallel transfer purpose.

Available in the Use Commandline mode only.

 

Hadoop properties

Talend Studio uses a default configuration for its engine to perform operations in a Hadoop distribution. If you need to use a custom configuration in a specific situation, complete this table with the property or properties to be customized. Then at runtime, the customized property or properties will override those default ones.

  • Note that if you are using the centrally stored metadata from the Repository, this table automatically inherits the properties defined in that metadata and becomes uneditable unless you change the Property type from Repository to Built-in.

For further information about the properties required by Hadoop and its related systems such as HDFS and Hive, see the documentation of the Hadoop distribution you are using or see Apache's Hadoop documentation on http://hadoop.apache.org/docs and then select the version of the documentation you want. For demonstration purposes, the links to some properties are listed below:

 

Mapred job map memory mb and Mapred job reduce memory mb

If the Hadoop distribution to be used is Hortonworks Data Platform V1.2 or Hortonworks Data Platform V1.3, you need to set proper memory allocations for the map and reduce computations to be performed by the Hadoop system.

In that situation, you need to enter the values you need in the Mapred job map memory mb and the Mapred job reduce memory mb fields, respectively. By default, the values are both 1000 which are normally appropriate for running the computations.

If the distribution is YARN, then the memory parameters to be set become Map (in Mb), Reduce (in Mb) and ApplicationMaster (in Mb), accordingly. These fields allow you to dynamically allocate memory to the map and the reduce computations and the ApplicationMaster of YARN.

 

Path separator in server

Leave the default value of the Path separator in server as it is, unless you have changed the separator used by your Hadoop distribution's host machine for its PATH variable or in other words, that separator is not a colon (:). In that situation, you must change this value to the one you are using in that host.

 

tStatCatcher Statistics

Select this check box to collect log data at the component level.

Global Variables

ERROR_MESSAGE: the error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is cleared, if the component has this check box.

EXIT_CODE: the exit code of the remote command. This is an After variable and it returns an integer.

A Flow variable functions during the execution of a component while an After variable functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl + Space to access the variable list and choose the variable to use from it.

For further information about variables, see Talend Studio User Guide.

Usage

This component is used standalone. It respects the Sqoop prerequisites. You need necessary knowledge about Sqoop to use it.

We recommend using the Sqoop of version 1.4+ in order to benefit the full functions of these components.

For further information about Sqoop, see the Sqoop manual on: http://sqoop.apache.org/docs/

Prerequisites

The Hadoop distribution must be properly installed, so as to guarantee the interaction with Talend Studio. The following list presents MapR related information for example.

  • Ensure that you have installed the MapR client in the machine where the Studio is, and added the MapR client library to the PATH variable of that machine. According to MapR's documentation, the library or libraries of a MapR client corresponding to each OS version can be found under MAPR_INSTALL\ hadoop\hadoop-VERSION\lib\native. For example, the library for Windows is \lib\native\MapRClient.dll in the MapR client jar file. For further information, see the following link from MapR: http://www.mapr.com/blog/basic-notes-on-configuring-eclipse-as-a-hadoop-development-environment-for-mapr.

    Without adding the specified library or libraries, you may encounter the following error: no MapRClient in java.library.path.

  • Set the -Djava.library.path argument, for example, in the Job Run VM arguments area of the Run/Debug view in the [Preferences] dialog box. This argument provides to the Studio the path to the native library of that MapR client. This allows the subscription-based users to make full use of the Data viewer to view locally in the Studio the data stored in MapR. For further information about how to set this argument, see the section describing how to view data of Talend Big Data Getting Started Guide.

For further information about how to install a Hadoop distribution, see the manuals corresponding to the Hadoop distribution you are using.

Log4j

If you are using a subscription-based version of the Studio, the activity of this component can be logged using the log4j feature. For more information on this feature, see Talend Studio User Guide.

For more information on the log4j logging levels, see the Apache documentation at http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/Level.html.

Limitation

If you have selected the Use Commandline mode, you need to use the host where Sqoop is installed to run the Job using this component.

Connections

Outgoing links (from this component to another):

Trigger: Run if; On Subjob Ok; On Subjob Error.

Incoming links (from one component to this one):

Row: Iterate;

Trigger: Run if; On Subjob Ok; On Subjob Error; On Component Ok; On Component Error

For further information regarding connections, see Talend Studio User Guide.