tSqoopMerge Standard properties
These properties are used to configure tSqoopMerge running in the Standard Job framework.
The Standard tSqoopMerge component belongs to the Big Data and the File families.
The component in this framework is available in all Talend products with Big Data and in Talend Data Fabric.
Basic settings
Mode |
Select the mode in which Sqoop is called in a Job execution. Use Commandline: the Sqoop shell is used to call Sqoop. You can read data from either HDFS or HCatalog. In this mode, you have to deploy and run the Job in the host where Sqoop is installed. Therefore, if you are a subscription-based user, we recommend installing and using a Talend JobServer provided by Talend in that host to run the Job; if you are using one of the Talend solutions with Big Data, you have to ensure that Talend Studio and the Sqoop to be used are in the same machine. Use Java API: the Java API is used to call Sqoop. In this mode, the Job can be run locally in Talend Studio but you need to configure the connection to the Hadoop distribution to be used. Note that JDK is required to execute the Job in the Java API mode and the versions of the JDK kits installed in both machines must be compatible with each other; for example, the versions are the same or the JDK version of the Hadoop machine is more recent. |
Hadoop properties |
Either Built-in or Repository:
|
Distribution |
Select the cluster you are using from the drop-down list. The options in the
list vary depending on the component you are using. Among these options, the following
ones requires specific configuration:
|
Hadoop Version |
Select the version of the Hadoop distribution you are using. The available options vary depending on the component you are using. |
NameNode URI |
Type in the URI of the Hadoop NameNode, the master node of a Hadoop system. For example, we assume that you have chosen a machine called masternode as the NameNode, then the location is hdfs://masternode:portnumber. If you are using WebHDFS, the location should be webhdfs://masternode:portnumber; WebHDFS with SSL is not supported yet. |
JobTracker Host |
Select this check box and in the displayed field, enter the location of the ResourceManager of your distribution. For example, tal-qa114.talend.lan:8050. This property is required when the query you want to use is executed in Windows and it is a Select query. For example, SELECT your_column_name FROM your_table_name Then you can continue to set the following parameters depending on the
configuration of the Hadoop cluster to be used (if you leave the check box of a
parameter clear, then at runtime, the configuration about this parameter in the
Hadoop cluster to be used will be ignored ):
For further information about the Hadoop Map/Reduce framework, see the Map/Reduce tutorial in Apache's Hadoop documentation on http://hadoop.apache.org. |
Use kerberos authentication |
If you are accessing the Hadoop cluster running with Kerberos security, select this check box, then, enter the Kerberos principal name for the NameNode in the field displayed. This enables you to use your username to authenticate against the credentials stored in Kerberos. In addition, since this component performs Map/Reduce computations, you also need to authenticate the related services such as the Job history server and the Resource manager or JobTracker depending on your distribution in the corresponding field. These principals can be found in the configuration files of your distribution. For example, in a CDH4 distribution, the Resource manager principal is set in the yarn-site.xml file and the Job history principal in the mapred-site.xml file. This check box is available depending on the Hadoop distribution you are connecting to. |
Use a keytab to authenticate |
Select the Use a keytab to authenticate check box to log into a Kerberos-enabled system using a given keytab file. A keytab file contains pairs of Kerberos principals and encrypted keys. You need to enter the principal to be used in the Principal field and the access path to the keytab file itself in the Keytab field. This keytab file must be stored in the machine in which your Job actually runs, for example, on a Talend JobServer. Note that the user that executes a keytab-enabled Job is not necessarily the one a principal designates but must have the right to read the keytab file being used. For example, the username you are using to execute a Job is user1 and the principal to be used is guest; in this situation, ensure that user1 has the right to read the keytab file to be used. |
Hadoop user name |
Enter the user name under which you want to execute the Job. Since a file or a directory in Hadoop has its specific owner with appropriate read or write rights, this field allows you to execute the Job directly under the user name that has the appropriate rights to access the file or directory to be processed. Note that this field is available depending on the distribution you are using. |
Old data directory |
Enter the path to the older dataset to be merged. |
New data directory |
Enter the path to the newer dataset to be merged. |
Target directory |
Enter the directory where you need to put the output of the merging. |
Merge key |
Enter the name of the column used as the key of each record for the merging. This primary key must be unique. |
Need to generate the JAR file |
Select this check box to generate the merge jar file and the merge class required to parse the datasets to be merged. The default name of the jar file and the class is SqoopMerge_component_ID. This compnent_ID is the ID of the tSqoopMerge component that generates the jar file and the class, such as tSqoopMerge_1, or tSqoopMerge_2. As this jar file is generated from the source table of the imported data, selecting this check box displays the corresponding parameters to be set for connecting to that table. |
JDBC property |
Either Built-in or Repository:
|
Connection |
Enter the JDBC URL used to connect to the database where the source data is stored. |
User name and Password |
Enter the authentication information used to connect to the source database. To enter the password, click the [...] button next to the password field, enter the password in double quotes in the pop-up dialog box, and click OK to save the settings. |
Table Name |
Type in the name of the source table. This name is used to name the generated jar file. |
Driver JAR |
In either the Use Commandline mode or the Java API mode, you must add the driver file of the database to be used to the lib folder of the Hadoop distribution you are using. For that purpose, use this Driver JAR table to add that driver file for the current Job you are designing. This driver jar is required only when you need to connect to the database to be used to generate the merge jar file; therefore this Driver JAR table is available only when you have selected the Need to generate the JAR file check box. |
JAR file |
If a required merge class already exists and is available, specify the access path to the jar file that contains that class for reusing the class. In this situation, you need to enter the name of the class in the Class name field in the Advanced settings tab. |
Print Log |
Select this check box to activate the Verbose check box. |
Verbose |
Select this check box to print more information while working, for example, the debugging information. |
Advanced settings
Custom class name |
Select this check box to display the Class name field and enter the name of the merge class you need to use. This check box must be clear if you use Generate the JAR file in the Basic settings tab. |
Additional arguments |
Complete this table to use additional arguments if needs be. By adding additional arguments, you are able to perform multiple operations in one single transaction. For example, you can use --hive-import and --hive-table in the Talend CommandLine mode or hive.import and hive.table.name in the Java API mode to create Hive table and write data in at the runtime of the transaction writing data in HDFS. For further information about the available Sqoop arguments in the Talend CommandLine mode and the Java API mode, respectively, see Additional arguments. |
Hadoop properties |
Talend Studio
uses a default configuration for its engine to perform
operations in a Hadoop distribution. If you need to use a custom configuration in a specific
situation, complete this table with the property or properties to be customized. Then at
runtime, the customized property or properties will override those default ones.
For further information about the properties required by Hadoop and its related systems such
as HDFS and Hive, see the documentation of the Hadoop distribution you
are using or see Apache's Hadoop documentation on http://hadoop.apache.org/docs and then select the version of the documentation you want. For demonstration purposes, the links to some properties are listed below:
|
Mapred job map memory mb and Mapred job reduce memory mb |
You can tune the map and reduce computations by selecting the Set memory check box to set proper memory allocations for the computations to be performed by the Hadoop system. In that situation, you need to enter the values you need in the Mapred job map memory mb and the Mapred job reduce memory mb fields, respectively. By default, the values are both 1024 which are normally appropriate for running the computations. The memory parameters to be set are Map (in Mb), Reduce (in Mb) and ApplicationMaster (in Mb). These fields allow you to dynamically allocate memory to the map and the reduce computations and the ApplicationMaster of YARN. |
Path separator in server |
Leave the default value of the Path separator in server as it is, unless you have changed the separator used by your Hadoop distribution's host machine for its PATH variable or in other words, that separator is not a colon (:). In that situation, you must change this value to the one you are using in that host. |
tStatCatcher Statistics |
Select this check box to collect log data at the component level. |
Global Variables
Global Variables |
ERROR_MESSAGE: the error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is cleared, if the component has this check box. EXIT_CODE: the exit code of the remote command. This is an After variable and it returns an integer. A Flow variable functions during the execution of a component while an After variable functions after the execution of the component. To fill up a field or expression with a variable, press Ctrl+Space to access the variable list and choose the variable to use from it. For more information about variables, see Using contexts and variables. |
Usage
Usage rule |
This component is used standalone. It respects the Sqoop prerequisites. You need necessary knowledge about Sqoop to use it. Talend recommends you to use the Sqoop of version 1.4+ in order to benefit the full functions of these components. For further information about Sqoop, see the Sqoop manual on: http://sqoop.apache.org/docs/ |
Prerequisites |
The Hadoop distribution must be properly installed, so as to guarantee the interaction with Talend Studio . The following list presents MapR related information for example.
For further information about how to install a Hadoop distribution, see the manuals corresponding to the Hadoop distribution you are using. |
Limitation |
If you have selected the Use Commandline mode, you need to use the host where Sqoop is installed to run the Job using this component. |
Connections |
Outgoing links (from this component to another): Trigger: Run if; On Subjob Ok; On Subjob Error; On Component Ok; On Component Error.
Incoming links (from one component to this one): Row: Iterate; Trigger: Run if; On Subjob Ok; On Subjob Error; On Component Ok; On Component Error
For further information regarding connections, see Using connections in a Job. |