Scenario: Replicating data from the source database to the target database - 6.1

Talend Components Reference Guide

Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for Big Data
Talend Open Studio for Data Integration
Talend Open Studio for Data Quality
Talend Open Studio for ESB
Talend Open Studio for MDM
Talend Real-Time Big Data Platform
Data Governance
Data Quality and Preparation
Design and Development
Talend Studio

This scenario describes a Job that writes the information of books into a CouchDB database, replicates the data to the target database, and then displays some replicated information of interest on the console.

Linking the components

  1. Drop the following components from the Palette onto the design workspace: tCouchDBConnection, tFileInputDelimited, tCouchDBOutput, tCouchDBInput, tLogRow and tCouchDBClose.

  2. Connect tFileInputDelimited to tCouchDBOutput using a Row > Main link.

  3. Do the same to connect tCouchDBInput to tLogRow.

  4. Connect tCouchDBConnection to tFileInputDelimited using a Trigger > OnSubjobOk link.

  5. Do the same to connect tFileInputDelimited to tCouchDBInput and tCouchDBInput to tCouchDBClose.

  6. Label the components to better identify their functions.

Configuring the components

Opening a CouchDB connection and triggering replication

  1. Double-click the tCouchDBConnection component to open its Basic settings view in theComponent tab.

  2. In the Server and Port fields, type in the connection details.

  3. In the Database field, type in the name of the database you want to use: bookstore_old in this example.

  4. Select the CouchDB version that you are using from the DB Version list.

  5. Select the Use trigger replication check box.

  6. In the Replicate target database area, click [+] to add one line for database replication settings.

  7. Enter the name of the target database name: bookstore_new in this example.

  8. Select the Continuous check box to continue the replication upon the server restart.

  9. In this example, the target database does not exist. Select the Create target DB check box to create the target database.

  10. Select the Is Canceled check box to cancel the replication between bookstore_old and bookstore_new at the end of the Job.

Reading the input data

  1. Double-click the tFileInputDelimited component to open its Component view.

  2. Click the [...] button next to the File name/Stream field to browse to the file that you want to read data from. In this scenario, it is D:/Input/bookstore.txt. The file contains six columns: _id, title, author, category, ISBN, and abstract.

    001;Computer Networks: A Systems Approach;Larry L. Peterson, Bruce S. Davie;Computer Science;0123850606;This best-selling and classic book teaches you the key principles of computer networks with examples drawn from the real world of network and protocol design.
    002;David Copperfield;Charles Dickens;Language&Literature;1555763227;This adaptation of the original story is presented in the format of a novel study, complete with exercises and vocabulary lists, and is geared to the language arts classes of grades 4 and 5.
    003;Life of Pi;Yann Martel;Language&Literature;0547350651;The son of a zookeeper, Pi Patel has an encyclopedic knowledge of animal behavior and a fervent love of stories.
    004;Les Miserables: Easyread Comfort Edition;Victor Hugo;Language&Literature;1425048250;Expressing the author's ideas about society, religion and politics, it is in the backdrop of Napoleonic Wars and ensuing years that the story unravels. Grace, moral philosophy, law and history of France are discussed.
    005;Computer Security;Dieter Gollmann;Computer Science;0470741155;This text moves away from the 'multi-level' security approach to compare and evaluate design alternatives in computer security.
    006;Advanced Database Systems;Carlo Zaniolo;Database;155860443X;This book, written by a team of leading specialists in their fields, introduces the research issues at the forefront of database technology and supports them with a variety of examples.
  3. In the Header field, type in 1 so that the header of the file will be skipped.

  4. Click Edit schema to define the data to pass on to the tCouchDBOutput component.

Writing data into one database and replicating the data

  1. Double-click the tCouchDBOutput component to open its Basic settings view in the Component tab.

  2. Click Sync columns to retrieve the schema from the preceding component.

  3. Select the Use an existing connection check box. In this example, the replication is triggered when opening the CouchDB connection.

  4. Select Upsert from the Action on data drop-down list.

Extracting the replicated data of interest

  1. Double-click the tCouchDBInput component to open its Component view.

  2. Click Edit schema to define the data structure to be read from the CouchDB database.

    By default, the Include docs check box is selected, so the id, key, value and jsonDoc columns are available in the schema.

    In this example, we define four columns to be extracted: id, title, author and category.

  3. Enter the Server and Port information.

  4. In the Database field, enter the name of the database from which the replicated data will be read. In this example, it is bookstore_new.

  5. In the Querying options area, type in the start key and end key to set the range of the data to be read: "001" and "006" in this example.

  6. Select the Extract JSON field check box to extract the desired data.

  7. Select jsonDoc from the JSON field list.

  8. In the Mapping area, click [+] to add items. Select the schema output column from the list and then type in the proper XPath query.

Displaying the extracted data

  1. Double-click the tLogRow component to open its Component view.

  2. Click Edit schema to define the data structure to be displayed on the console. In this example, we need to remove the jsonDoc column.

  3. In the Mode area, select Table (print values in cells of a table).

Closing the CouchDB connection

  1. Double-click the tCouchDBClose component to open its Component view.

  2. Select the connection to be closed from the Component List.

Saving and executing the Job

  1. Press Ctrl+S to save your Job.

  2. Execute the Job by pressing F6 or clicking Run on the Run tab.

    The book information read from the replicated database is shown on the console.