tCouchDBOutput - 6.3

Talend Components Reference Guide

EnrichVersion
6.3
EnrichProdName
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for Big Data
Talend Open Studio for Data Integration
Talend Open Studio for Data Quality
Talend Open Studio for ESB
Talend Open Studio for MDM
Talend Real-Time Big Data Platform
task
Data Governance
Data Quality and Preparation
Design and Development
EnrichPlatform
Talend Studio

Warning

This component is available in the Palette of the studio only if you have subscribed to one of the Talend solutions with Big Data.

Function

tCouchDBOutput receives data from the preceding component, and writes data into CouchDB.

Purpose

tCouchDBOutput allows you to load JSON documents, write data into or remove data from them and then save the documents back to the database on a CouchDB server.

tCouchDBOutput properties

Component family

Big Data / CouchDB

 

Basic settings

Schema and Edit schema

A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. The schema is either Built-In or stored remotely in the Repository.

Since version 5.6, both the Built-In mode and the Repository mode are available in any of the Talend solutions.

Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

  • View schema: choose this option to view the schema only.

  • Change to built-in property: choose this option to change the schema to Built-in for local changes.

  • Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion. If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the [Repository Content] window.

 

 

Built-In: You create and store the schema locally for this component only. Related topic: see Talend Studio User Guide.

 

 

Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs. Related topic: see Talend Studio User Guide.

When the schema to be reused has default values that are integers or functions, ensure that these default values are not enclosed within quotation marks. If they are, you must remove the quotation marks manually.

For more details, see the article Verifying default values in a retrieved schema on Talend Help Center (https://help.talend.com).

 

Sync columns

Click this button to retrieve schema from the previous component connected in the Job.

Connection

Use existing connection

Select this check box and in the Component List click the relevant connection component to reuse the connection details you already defined.

 DB Version

Select the CouchDB version that you are using.

 

Server

Hostname or IP address of the CouchDB server.

 

Port

Listening port number of the CouchDB server.

 

Database

Name of the database that you want to use.

Authentication

Required authentication

Select this check box to provide credentials for the CouchDB authentication.

  • Username: Type in the user name for the CouchDB authentication.

  • Password: Type in the password for the CouchDB authentication.

To enter the password, click the [...] button next to the password field, and then in the pop-up dialog box enter the password between double quotes and click OK to save the settings.

Replication

Use trigger replication

Select this check box to trigger replication between databases.

Warning

This check box appears only if you do not select the Use an existing connection check box.

Replicate target database

Target DB name

Specify the target database to which the documents will be copied. It can be a local database name or a remote database URL.

 

Continuous

Select this check box to continue a replication upon the server restart.

 

Create target DB

Select this check box to create the target database if it does not exist.

 

IS Canceled

Select this check box to cancel the existing replication between the specified source database and target database at the end of the Job.

 

Action on data

On the data of interest, you can perform:

  • Insert: Insert data into the database.

  • Update: Update data in the database.

  • Upsert: Insert the data if they do not exist or update the existing data.

  • Delete: Remove the data from the database.

JSON Configuration

Generate JSON Document

Select this check box to generate a JSON document and configure the desired data structure for it.

 

Key

Select the key that you want to use from the list.

 

Configure JSON Tree

Click the [...] button to open the window for JSON tree configuration. For more information, see Configuring a JSON Tree.

 

Group by

Customize the input columns based on which you want to group the data.

 

Remove root node

Select this check box to remove the root node.

 

Die on error

This check box is cleared by default, meaning to skip the row on error and to complete the process for error-free rows.

Advanced settings

tStatCatcher Statistics

Select this check box to gather the Job processing metadata at the Job level as well as at each component level.

Global Variables

NB_LINE: the number of rows read by an input component or transferred to an output component. This is an After variable and it returns an integer.

NB_LINE_INSERTED: the number of rows inserted. This is an After variable and it returns an integer.

ERROR_MESSAGE: the error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl + Space to access the variable list and choose the variable to use from it.

For further information about variables, see Talend Studio User Guide.

Usage

This component is used as an output component and it always needs an incoming link.

Log4j

If you are using a subscription-based version of the Studio, the activity of this component can be logged using the log4j feature. For more information on this feature, see Talend Studio User Guide.

For more information on the log4j logging levels, see the Apache documentation at http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/Level.html.

Limitation

n/a

Scenario: Replicating data from the source database to the target database

This scenario describes a Job that writes the information of books into a CouchDB database, replicates the data to the target database, and then displays some replicated information of interest on the console.

Linking the components

  1. Drop the following components from the Palette onto the design workspace: tCouchDBConnection, tFileInputDelimited, tCouchDBOutput, tCouchDBInput, tLogRow and tCouchDBClose.

  2. Connect tFileInputDelimited to tCouchDBOutput using a Row > Main link.

  3. Do the same to connect tCouchDBInput to tLogRow.

  4. Connect tCouchDBConnection to tFileInputDelimited using a Trigger > OnSubjobOk link.

  5. Do the same to connect tFileInputDelimited to tCouchDBInput and tCouchDBInput to tCouchDBClose.

  6. Label the components to better identify their functions.

Configuring the components

Opening a CouchDB connection and triggering replication

  1. Double-click the tCouchDBConnection component to open its Basic settings view in theComponent tab.

  2. In the Server and Port fields, type in the connection details.

  3. In the Database field, type in the name of the database you want to use: bookstore_old in this example.

  4. Select the CouchDB version that you are using from the DB Version list.

  5. Select the Use trigger replication check box.

  6. In the Replicate target database area, click [+] to add one line for database replication settings.

  7. Enter the name of the target database name: bookstore_new in this example.

  8. Select the Continuous check box to continue the replication upon the server restart.

  9. In this example, the target database does not exist. Select the Create target DB check box to create the target database.

  10. Select the Is Canceled check box to cancel the replication between bookstore_old and bookstore_new at the end of the Job.

Reading the input data

  1. Double-click the tFileInputDelimited component to open its Component view.

  2. Click the [...] button next to the File name/Stream field to browse to the file that you want to read data from. In this scenario, it is D:/Input/bookstore.txt. The file contains six columns: _id, title, author, category, ISBN, and abstract.

    _id;title;author;category;ISBN;abstract
    001;Computer Networks: A Systems Approach;Larry L. Peterson, Bruce S. Davie;Computer Science;0123850606;This best-selling and classic book teaches you the key principles of computer networks with examples drawn from the real world of network and protocol design.
    002;David Copperfield;Charles Dickens;Language&Literature;1555763227;This adaptation of the original story is presented in the format of a novel study, complete with exercises and vocabulary lists, and is geared to the language arts classes of grades 4 and 5.
    003;Life of Pi;Yann Martel;Language&Literature;0547350651;The son of a zookeeper, Pi Patel has an encyclopedic knowledge of animal behavior and a fervent love of stories.
    004;Les Miserables: Easyread Comfort Edition;Victor Hugo;Language&Literature;1425048250;Expressing the author's ideas about society, religion and politics, it is in the backdrop of Napoleonic Wars and ensuing years that the story unravels. Grace, moral philosophy, law and history of France are discussed.
    005;Computer Security;Dieter Gollmann;Computer Science;0470741155;This text moves away from the 'multi-level' security approach to compare and evaluate design alternatives in computer security.
    006;Advanced Database Systems;Carlo Zaniolo;Database;155860443X;This book, written by a team of leading specialists in their fields, introduces the research issues at the forefront of database technology and supports them with a variety of examples.
  3. In the Header field, type in 1 so that the header of the file will be skipped.

  4. Click Edit schema to define the data to pass on to the tCouchDBOutput component.

Writing data into one database and replicating the data

  1. Double-click the tCouchDBOutput component to open its Basic settings view in the Component tab.

  2. Click Sync columns to retrieve the schema from the preceding component.

  3. Select the Use an existing connection check box. In this example, the replication is triggered when opening the CouchDB connection.

  4. Select Upsert from the Action on data drop-down list.

Extracting the replicated data of interest

  1. Double-click the tCouchDBInput component to open its Component view.

  2. Click Edit schema to define the data structure to be read from the CouchDB database.

    By default, the Include docs check box is selected, so the id, key, value and jsonDoc columns are available in the schema.

    In this example, we define four columns to be extracted: id, title, author and category.

  3. Enter the Server and Port information.

  4. In the Database field, enter the name of the database from which the replicated data will be read. In this example, it is bookstore_new.

  5. In the Querying options area, type in the start key and end key to set the range of the data to be read: "001" and "006" in this example.

  6. Select the Extract JSON field check box to extract the desired data.

  7. Select jsonDoc from the JSON field list.

  8. In the Mapping area, click [+] to add items. Select the schema output column from the list and then type in the proper XPath query.

Displaying the extracted data

  1. Double-click the tLogRow component to open its Component view.

  2. Click Edit schema to define the data structure to be displayed on the console. In this example, we need to remove the jsonDoc column.

  3. In the Mode area, select Table (print values in cells of a table).

Closing the CouchDB connection

  1. Double-click the tCouchDBClose component to open its Component view.

  2. Select the connection to be closed from the Component List.

Saving and executing the Job

  1. Press Ctrl+S to save your Job.

  2. Execute the Job by pressing F6 or clicking Run on the Run tab.