Google BigQuery Database - Import - 7.1

Talend Data Catalog Bridges

author
Talend Documentation Team
EnrichVersion
7.1
EnrichProdName
Talend Big Data Platform
Talend Data Fabric
Talend Data Management Platform
Talend Data Services Platform
Talend MDM Platform
Talend Real-Time Big Data Platform
EnrichPlatform
Talend Data Catalog

Bridge Requirements

This bridge:
  • requires Internet access to https://repo.maven.apache.org/maven2/ and/or other tool sites to download drivers into <TDC_HOME>/data/download/MIMB/. For more information on how to retrieve third-party drivers when the TDC server cannot access the Internet, see this article.

Bridge Specifications

Vendor Google
Tool Name BigQuery Database
Tool Version 2.0.0
Tool Web Site http://developers.google.com/bigquery/
Supported Methodology [Relational Database] Data Store (Physical Data Model) via Java API
Data Profiling
Multi-Model Harvesting
Incremental Harvesting
Remote Repository Browsing for Model Selection

BRIDGE INFORMATION
Import tool: Google BigQuery Database 2.0.0 (http://developers.google.com/bigquery/)
Import interface: [Relational Database] Data Store (Physical Data Model) via Java API from Google BigQuery Database
Import bridge: 'GoogleBigQuery' 10.1.0

BRIDGE DISCLAIMER
This bridge requires internet access to https://repo.maven.apache.org/maven2/ (and exceptionally a few other tool sites)
in order to download the necessary third party software libraries into $HOME/data/download/MIMB/
(such directory can be copied from another MIMB server with internet access).
By running this bridge, you hereby acknowledge responsibility for the license terms and any potential security vulnerabilities from these downloaded third party software libraries.

BRIDGE DOCUMENTATION
IMPORTING FROM GOOGLE BIGQUERY USING Google's Client APIs for Java.

This bridge establishes a connection to Google's BigQuery servers in order to extract the physical metadata.
It is critical that the parameters are filled correctly to satisfy the local connection requirements on the client workstation that runs the bridge. Please refer to the individual parameter's tool tips for more detailed examples.

This bridge uses Oauth 2.0 protocol for accessing Google's BigQuery service. Currently, it supports authorization using the 'service accounts'.
Please refer to the documentation provided with the link below to create a new service account. This will provide you with the client ID and the key file that is required to run the bridge. https://developers.google.com/bigquery/docs/authorization#service-accounts-server


Bridge Parameters

Parameter Name Description Type Values Default Scope
Service Client Id The client ID for the service account created for the bridge. It is usually of the form xyzz@@developer.gserviceaccount.com

STRING     Mandatory
Private Key File The private key file for the service account created for the bridge.
This file should be in the PKCS #12 format (Public Key Cryptography Standard #12).
FILE *.*   Mandatory
Project Id The fully qualified project id. Usually this is of the form: mydomain.com:myProject. REPOSITORY_SUBSET     Mandatory
Dataset You can specify a list of datasets to import and skip.
When the list is empty all available datasets are imported.
The list can have one or more dataset names separated by semicolons (e.g. dataset1; dataset2).
You can specify dataset name patterns using SQL LIKE expression syntax.
Patterns support inclusions and exclusions.

Here is an example of inclusion syntax, "A%; %B; %C%; D" that tries to get dataset names that:
- start with A or
- end with B or
- contain C or
- equal D

Note: when a pattern has special characters, like spaces, enclose it in single quote marks (e.g. "'two wo%'; onewo%;").

To exclude a pattern, prefix it with "NOT". Here is an example of exclusion syntax:
"A%; %B; NOT %C; "
REPOSITORY_SUBSET      
Miscellaneous Specify miscellaneous options identified with a -letter and value.

For example, -m 4G -f 100 -j -Dname=value -Xms1G

-m the maximum Java memory size whole number (e.g. -m 4G or -m 2500M ).
-s import system objects that are skipped by default (e.g. -s)
-v set environment variable(s) (e.g. -v var1=value -v var2="value with spaces").
-j the last option that is followed by Java command line options (e.g. -j -Dname=value -Xms1G).

STRING      

 

Bridge Mapping

Meta Integration Repository (MIR)
Metamodel
(based on the OMG CWM standard)
"Google BigQuery Database"
Metamodel
Google BigQuery (Project)
Mapping Comments
     
Attribute Column  
Description Description  
ExtraConstraint Constraint  
InitialValue Initial Value  
Name Name  
NativeId Native Id  
Optional Nullable  
PhysicalName Physical Name  
Position Position  
Class Table  
Description Description  
Name Name  
NativeId Native Id  
PhysicalName Physical Name  
ClassDiagram Diagram  
Description Description  
Name Name  
DatabaseSchema Dataset  
Description Description  
Name Name  
NativeId Native Id  
PhysicalName Physical Name  
DesignPackage Subject Area  
Description Description  
Name Name  
Join Logical Relationship  
Description Description  
Name Name  
SQLViewAttribute ViewColumn  
Description Description  
InitialValue Initial Value  
Name Name  
NativeId Native Id  
Optional Nullable  
PhysicalName Physical Name  
Position Position  
SQLViewEntity View  
Description Description  
Name Name  
NativeId Native Id  
PhysicalName Physical Name  
StoreModel Project  
Author Author  
CreationTime Creation Time  
Description Description  
ModificationTime Modification Time  
Modifier Modifier  
Name Name  
NativeId Native Id  
PhysicalName Physical Name  
StoreType Store Type  
SystemMajorVersion System Major Version  
SystemMinorVersion System Minor Version  
SystemReleaseVersion System Release Version  
SystemType System Type  
SystemTypeOld System Type Old