tHBaseLookupInput - 6.1

Talend Components Reference Guide

EnrichVersion
6.1
EnrichProdName
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for Big Data
Talend Open Studio for Data Integration
Talend Open Studio for Data Quality
Talend Open Studio for ESB
Talend Open Studio for MDM
Talend Real-Time Big Data Platform
task
Data Governance
Data Quality and Preparation
Design and Development
EnrichPlatform
Talend Studio

Warning

The streaming version of this component is available in the Palette of the studio on the condition that you have subscribed to Talend Real-time Big Data Platform or Talend Data Fabric.

Function

tHBaseLookupInput extracts columns corresponding to schema definition. Then it passes the extracted data to the next component.

Purpose

tHBaseLookupInput provides lookup data to the main flow of a streaming Job.

It passes on the extracted data to tMap in order to provide the lookup data to the main flow. It must be directly connected to a tMap component and requires this tMap to use Reload at each row or Reload at each row (cache) for the lookup flow.

HBase filters

This table presents the HBase filters available in Talend Studio and the parameters required by those filters.

Filter type

Filter column

Filter familyFilter operationFilter valueFilter comparator typeObjective

Single Column Value Filter

Yes

Yes

Yes

Yes

Yes

It compares the values of a given column against the value defined for the Filter value parameter. If the filtering condition is met, all columns of the row will be returned.

Family filter

 

Yes

Yes

 

Yes

It returns the columns of the family that meets the filtering condition.

Qualifier filter

Yes

 

Yes

 

Yes

It returns the columns whose column qualifiers match the filtering condition.

Column prefix filter

Yes

Yes

   

It returns all columns of which the qualifiers have the prefix defined for the Filter column parameter.

Multiple column prefix filter

Yes (Multiple prefixes are separated by coma, for example, id,id_1,id_2)

Yes

   

It works the same way as a Column prefix filter does but allows specifying multiple prefixes.

Column range filter

Yes (The ends of a range are separated by coma. )

Yes

   It allows intra row scanning and returns all matching columns of a scanned row.

Row filter

  

Yes

Yes

YesIt filters on row keys and returns all rows that matches the filtering condition.

Value filter

  

Yes

Yes

Yes

It returns only columns that have a specific value.

The use explained above of the listed HBase filters is subject to revisions made by Apache in its Apache HBase project; therefore, in order to fully understand how to use these HBase filters, we recommend reading Apache's HBase documentation.

tHBaseLookupInput properties in Spark Streaming Jobs

If you have subscribed to Talend Real-time Big Data Platform or Talend Data Fabric, this component is available in the following types of Jobs:

Component family

Databases/HBase

 

Basic settings

Storage configuration

Select the tHBaseConfiguration component from which the Spark system to be used reads the configuration information to connect to HBase.

Property type

Either Built-In or Repository.

Built-In: No property data stored centrally.

Repository: Select the repository file where the properties are stored.

 

Click this icon to open a database connection wizard and store the database connection parameters you set in the component Basic settings view.

For more information about setting up and storing database connection parameters, see Talend Studio User Guide.

 

Schema et Edit schema

A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. The schema is either Built-In or stored remotely in the Repository.

Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

  • View schema: choose this option to view the schema only.

  • Change to built-in property: choose this option to change the schema to Built-in for local changes.

  • Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion. If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the [Repository Content] window.

 

 

Built-In: You create and store the schema locally for this component only. Related topic: see Talend Studio User Guide.

 

 

Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs. Related topic: see Talend Studio User Guide.

 

Table name

Type in the name of the HBase table from which you need to extract columns.

 

Set table Namespace mappings

Select this check box and in the field that is displayed, enter the string to be used to construct the mapping between an Apache HBase table and a MapR table.

For the valid syntax you can use, see http://doc.mapr.com/display/MapR40x/Mapping+Table+Namespace+Between+Apache+HBase+Tables+and+MapR+Tables.

 

Define a row selection

Select this check box and then in the Start row and the End row fields, enter the corresponding row keys to specify the range of the rows you want the current component to extract.

Different from the filters you can set using Is by filter requiring the loading of all records before filtering the ones to be used, this feature allows you to directly select only the rows to be used.

 Mapping

Complete this table to map the columns of the HBase table to be used with the schema columns you have defined for the data flow to be processed.

FilterIs by filter

Select this check box to use HBase filters to perform fine-grained data selection from HBase, such as selection of keys, or values, based on regular expressions.

Once selecting it, the Filter table that is used to define filtering conditions becomes available.

These filters are advanced features provided by HBase and subject to constraints explained in Apache's HBase documentation. Therefore, advanced knowledge of HBase is required to make full use of these filters.

 Logical operation

Select the operator you need to use to define the logical relation between filters. This available operators are:

  • And: every defined filtering conditions must be satisfied. It represents the relationship FilterList.Operator.MUST_PASS_ALL

  • Or: at least one of the defined filtering conditions must be satisfied. It represents the relationship: FilterList.Operator.MUST_PASS_ONE

 Filter

Click the button under this table to add as many rows as required, each row representing a filter. The parameters you may need to set for a filter are:

  • Filter type: the drop-down list presents pre-existing filter types that are already defined by HBase. Select the type of the filter you need to use.

  • Filter column: enter the column qualifier on which you need to apply the active filter. This parameter becomes mandatory depending on the type of the filter and of the comparator you are using. For example, it is not used by the Row Filter type but is required by the Single Column Value Filter type.

  • Filter family: enter the column family on which you need to apply the active filter. This parameter becomes mandatory depending on the type of the filter and of the comparator you are using. For example, it is not used by the Row Filter type but is required by the Single Column Value Filter type.

  • Filter operation: select from the drop-down list the operation to be used for the active filter.

  • Filter Value: enter the value on which you want to use the operator selected from the Filter operation drop-down list.

  • Filter comparator type: select the type of the comparator to be combined with the filter you are using.

Depending on the Filter type you are using, some or each of the parameters become mandatory. For further information, see HBase filters

Usage in Spark Streaming Jobs

In a Talend Spark Streaming Job, it is used as a start component and requires an output link. The other components used along with it must be Spark Streaming components, too. They generate native Spark code that can be executed directly in a Spark cluster.

This component should use a tHBaseConfiguration component present in the same Job to connect to HBase.

This component, along with the Spark Streaming component Palette it belongs to, appears only when you are creating a Spark Streaming Job.

Note that in this documentation, unless otherwise explicitly stated, a scenario presents only Standard Jobs, that is to say traditional Talend data integration Jobs.

Spark Connection

You need to use the Spark Configuration tab in the Run view to define the connection to a given Spark cluster for the whole Job. In addition, since the Job expects its dependent jar files for execution, one and only one file system related component from the Storage family is required in the same Job so that Spark can use this component to connect to the file system to which the jar files dependent on the Job are transferred:

This connection is effective on a per-Job basis.

Log4j

If you are using a subscription-based version of the Studio, the activity of this component can be logged using the log4j feature. For more information on this feature, see Talend Studio User Guide.

For more information on the log4j logging levels, see the Apache documentation at http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/Level.html.

Related scenarios

No scenario is available for the Spark Streaming version of this component yet.