Snowflake Database (via JDBC) - Import - 7.1

Talend Data Catalog Bridges

author
Talend Documentation Team
EnrichVersion
7.1
EnrichProdName
Talend Big Data Platform
Talend Data Fabric
Talend Data Management Platform
Talend Data Services Platform
Talend MDM Platform
Talend Real-Time Big Data Platform
EnrichPlatform
Talend Data Catalog

Bridge Specifications

Vendor Snowflake
Tool Name Snowflake Database
Tool Version 2.x
Tool Web Site https://www.snowflake.com/
Supported Methodology [Relational Database] Multi-Model, Data Store (Physical Data Model, Stored Procedure Expression Parsing) via JDBC API
Multi-Model Harvesting
Incremental Harvesting
Data Profiling
Remote Repository Browsing for Model Selection

SPECIFICATIONS
Tool: Snowflake Snowflake Database version 2.x via JDBC API
See https://www.snowflake.com/
Metadata: [Relational Database] Multi-Model, Data Store (Physical Data Model, Stored Procedure Expression Parsing)
Bridge: JdbcSnowflake version 11.0.0

OVERVIEW
This bridge uses a JDBC connection to a SNOWFLAKE database server to extract all its metadata (e.g. Schemas, Tables, Columns, Data Types.) and parse any SQL (e.g. Views, Stored Procedures)to produce the data flow lineage.
This bridge imports the database server metadata as multi models with one model per schema. Further imports (incremental harvesting) will detect changes in schema to only import changed schemas (reusing already harvested unchanged schemas).

REQUIREMENTS
Minimal supported database version is 8.4.

FREQUENTLY ASKED QUESTIONS
n/a

LIMITATIONS
Refer to the current general known limitations at http://metaintegration.com/Products/MIMB/MIMBKnownLimitations.html or bundled in Documentation/ReadMe/MIMBKnownLimitations.html

SUPPORT
Provide a trouble shooting package with debug log. Debug log can be set in the UI or in conf/conf.properties with MIR_LOG_LEVEL=6


Bridge Parameters

Parameter Name Description Type Values Default Scope
Driver location
This parameter is optional.

By default, all necessary JARs will be loaded from:
'${MODEL_BRIDGE_HOME}/java/Jdbc/postgresql'

However, you may use this parameter to specify a different location. All JARs found in the specified folder will be added to the CLASSPATH.

The 'org.postgresql.Driver' JDBC driver with the 'Snowflake Native Driver' name is required.
DIRECTORY      
Host Host name, IP address where the database server is running
or full qualified JDBC connection string for Snowflake JDBC driver.
STRING   localhost Mandatory
Warehouse Warehouse. STRING     Mandatory
Database Database. STRING     Mandatory
User The database user name on whose behalf the connection is being made. STRING      
Password The user password used to connect to the database via JDBC. PASSWORD      
Schema Description: you can specify a list of database schemas to import.
When the list is empty all available schemas are imported.
The list can have one or more schema names separated by semicolons (e.g. schema1; schema2).
You can specify schema name patterns by using SQL LIKE expressions syntax.
Patterns support inclusions and exclusions.
Here is an example of inclusion syntax, "A%; %B; %C%; D" that tries to get schema names that:
- start with A or
- end with B or
- contain C or
- equal D

Note: when a pattern has special characters, like spaces enclose it in single quote marks (e.g. "'two wo%'; onewo%;").

To exclude a pattern prefix it with 'NOT'. Here is an example of exclusion syntax, "A%; %B; NOT %SYS; NOT 'SYS%'"
that contributes to the following SQL filter: "where (name like A% or name like %B) and (name not like %SYS) and (name not like 'SYS%')"
REPOSITORY_SUBSET      
Tables, Views The subset of tables to import expressed as a semicolon separated list of tables, e.g.
table1; table2

All user tables are imported if that list is empty.

All system tables (e.g. SysAdmin) and objects are ignored by default. If system tables are needed, use the Miscellaneous parameter option -s.

Table name patterns using SQL LIKE expressions syntax:

- Inclusion syntax:
using '%, e.g.
A%; %B; %C%; D
- start with A or
- end with B or
- contain C or
- equal D

- Exclusion syntax:
using 'NOT", e.g.
A%; %B; NOT %SYS; NOT 'SYS%'
- where (name like A% or name like %B)
- and (name NOT like %SYS)
- and (name NOT like 'SYS%')

- Special characters:
using single quote, e.g.
'two wo%'; onewo%;
STRING      
Import indexes Controls the import of Indexes:

'False'
Indexes are not imported

'True'
Indexes are imported
BOOLEAN
False
True
False  
Stored procedure details Controls the amount of details imported from stored procedures:

'Signature'
The name and parameters of stored procedures

'Code, signature'
The above plus code

'Lineage, code, signature'
The above plus data lineage derived from the code

'None'
stored procedure details are not included.
ENUMERATED
None
Signature
Code, signature
Lineage, code, signature
Signature  
Miscellaneous Specify miscellaneous options identified with a -option followed by a value if required:

GENERAL OPTIONS
-m <Java Memory's maximum size>
1G by default on 64bits JRE or as set in conf/conf.properties, e.g.
-m 8G
-m 2500M

-j <Java Runtime Environment command line options>
This option must be the last one in the Miscellaneous parameter as all the text after -j is passed "as is" to the JRE, e.g.
-j -Dname=value -Xms1G

-jre <Java Runtime Environment full path name>
It can be an absolute path to javaw.exe on Windows or a link/script path on Linux, e.g.
-jre "c:\Program Files\Java\jre1.8.0_211\bin\javaw.exe"

-v <Environment variable value>
None by default, e.g.
-v var1=value1 -v var2="value2 with spaces"

-model.name <model name>
Override the model name, e.g.
-model.name "My Model Name"

-prescript <script name>
The script must be located in the bin directory, and have .bat or .sh extension.
The script path must not include any parent directory symbol (..).
The script should return exit code 0 to indicate success, or another value to indicate failure.
For example:
-prescript \"script.bat\"

JDBC OPTIONS
-f <number of rows>
The database driver fetch size in number of rows. E.g.
-f 100

-s
Import system objects that are skipped by default.

-synonyms
Disable import of all synonyms.

-server.name <database identification name>
Useful when importing a pluggable database that is being moved between containers or a development database going to a production server.

-backup <directory>
Full path of an empty directory to save the metadata input files for further troubleshooting.
STRING      

 

Bridge Mapping

Meta Integration Repository (MIR)
Metamodel
(based on the OMG CWM standard)
"Snowflake Database (via JDBC)"
Metamodel
Jdbc
Mapping Comments
     
Name Name  
Attribute Table Column  
Comment Comments on the column If JDBC driver supports
InitialValue Default value  
Name Name  
Optional Based on the nullable property For Access databases, set to false only if the attribute is in a Unique Index
PhysicalName Name  
Position Position If position is not provided, the order in which the attributes are retrieved is used.
BaseType Types  
DataType Data Type See datatype conversion array
Length Size  
Name   The name is computed from the datatype
PhysicalName Name  
Scale Maximum scale  
Class Table of type "TABLE"
Comment Comments on the table If JDBC driver supports
CppClassType   Set to ENTITY
CppPersistent   Set to True
Name Name  
PhysicalName Name  
ClassDiagram Schema A class diagram is created for each package and contains all the elements of the package
DerivedType Column Table column, stored procedure column SQL View column or type
DataType Data Type See datatype conversion array
Length Size  
Name   The name is computed from the datatype
PhysicalName Name  
Scale Decimal digits  
UserDefined   True for Type
DesignPackage Schema A Package is created for each retrieved schema. If there is no schema a default package is created.
Name Name Set to "Schema" if there is no schema or the schema has no name.
SQLViewAttribute View Column  
Comment Comments on the column If JDBC driver supports
Name Name  
PhysicalName Name  
Position Ordinal position  
SQLViewEntity Table of type "VIEW"
Comment Comments on the table If JDBC driver supports
Name Name  
PhysicalName Name  
StoreModel Catalog The model is built using the elements contained in the catalog (e.g. the database for MS SQL server)
Name Name Set to "Catalog" if the catalog has no name.
Synonym Table Synonym If JDBC driver supports
Name Name