tMongoDBBulkLoad properties - 6.3

Talend Open Studio for Big Data Components Reference Guide

EnrichVersion
6.3
EnrichProdName
Talend Open Studio for Big Data
task
Data Governance
Data Quality and Preparation
Design and Development
EnrichPlatform
Talend Studio

Component family

Big Data / MongoDB

 

Basic settings

Schema and Edit schema

A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. The schema is either Built-In or stored remotely in the Repository.

Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

  • View schema: choose this option to view the schema only.

  • Change to built-in property: choose this option to change the schema to Built-in for local changes.

  • Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion. If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the [Repository Content] window.

 

MongoDB directory

Fill in this field with the MongoDB home directory.

 

Use local DB path

Select this check box to provide the information of the local database that you want to use. MongoDB V3.0 and onward versions do not support this feature.

  • Local DB path: type in the path to the local database specified when starting the MongoDB server.

 

Use replica set address

Select this check box to define a replica set to be connected.

  • Replica set name: specify the name of the replica set.

  • Replica address: specify multiple MongoDB database servers for failover as needed. Note that if you leave the replica host or replica port unspecified, their default values localhost and 27017 will be used.

 

Server

Hostname or IP address of the database server. Note that the default value localhost will be used if the server is not specified.

Warning

This field is available only when the Use replica set address check box is not selected.

 

Port

Listening port of the database server. Note that the default value 27017 will be used if the port is not specified.

Warning

This field is available only when the Use replica set address check box is not selected.

 

Database

Type in the name of the database to import data to.

 

Collection

Type in the name of the collection to import data to.

 

Use SSL connection

Select this check box to enable the SSL or TLS encrypted connection.

Then you need to use the tSetKeystore component in the same Job to specify the encryption information.

For further information about tSetKeystore, see tSetKeystore.

Note that the SSL connection is available only for the version 2.4 + of MongoDB.

 

Drop collection if exist

Select this check box to remove the collection if it already exists.

 

Required authentication

Select this check box to enable the database authentication.

The Authentication mechanism drop-down list becomes available. Among the listed mechanisms, the NEGOTIATE one is recommended if you are not using Kerberos, because it automatically select the authentication mechanism the most adapted to the MongoDB version you are using.

For details about the other mechanisms in this list, see MongoDB Authentication from the MongoDB documentation.

 

Set Authentication database

If the username to be used to connect to MongoDB has been created in a specific Authentication database of MongoDB, select this check box to enter the name of this Authentication database in the Authentication database field that is displayed.

For further information about the MongoDB Authentication database, see User Authentication database.

Username and Password

DB user authentication data.

To enter the password, click the [...] button next to the password field, and then in the pop-up dialog box enter the password between double quotes and click OK to save the settings.

Available when the Required authentication check box is selected.

If the security system you have selected from the Authentication mechanism drop-down list is Kerberos, you need to enter the User principal, the Realm and the KDC server fields instead of the Username and the Password fields.

 

Data file

Type in the full path of the file from which the data will be imported or click the [...] button to browse to the desired data file.

Warning

Make sure that the data file is in standard format. For example, the fields in CSV files should be separated with commas.

 

File type

Select the proper file type from the list. CSV, TSV and JSON are supported.

 

The JSON file starts with an array

Select this check box to allow tMongoDBBulkload to read the JSON files starting with an array.

This check box appears when the File type you have selected is JSON.

 

Action on data

Select the action that you want to perform on the data.

  • Insert: Insert the data into the database.

    Warning

    When inserting data from CSV or TSV files into the MongoDB database, you need to specify fields either by selecting the First line is header check box or defining them in the schema.

  • Upsert: Insert the data if they do not exist or update the existing data.

    Warning

    When upserting data into the MongoDB database, you need to specify a list of fields for the query portion of the upsert operation.

 

Upsert fields

Customize the fields that you want to upsert as needed.

Warning

This table is available when you select Upsert from the Action on data list.

 

First line is header

Select this check box to use the first line in CSV or TSV files as a header.

Warning

This check box is available only when you select CSV or TSV from the File type list.

 

Ignore blanks

Select this check box to ignore the empty fields in CSV or TSV files.

Warning

This check box is available only when you select CSV or TSV from the File type list.

 

Print log

Select this check box to print logs.

Advanced settings

Additional arguments

Complete this table to use the additional arguments as required.

For example, you can use the argument "--jsonArray" to accept the import of data expressed with multiple MongoDB documents within a single JSON array. For more information about the additional arguments, go to http://docs.mongodb.org/manual/reference/program/mongoimport/ and read the description of options.

 

tStatCatcher Statistics

Select this check box to collect the log data at a component level.

Global Variables

NB_LINE: the number of rows read by an input component or transferred to an output component. This is an After variable and it returns an integer.

ERROR_MESSAGE: the error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl + Space to access the variable list and choose the variable to use from it.

For further information about variables, see Talend Studio User Guide.

Usage

This component can be used together with the tMongoDBInput component to check if the data is imported as expected.

Log4j

If you are using a subscription-based version of the Studio, the activity of this component can be logged using the log4j feature. For more information on this feature, see Talend Studio User Guide.

For more information on the log4j logging levels, see the Apache documentation at http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/Level.html.

Limitation

The MongoDB client tool needs to be installed on the machine where Jobs using this component are executed.