Oracle Data Integrator (ODI) - Import - Cloud

Talend Cloud Data Catalog Bridges

author
Talend Documentation Team
EnrichVersion
Cloud
EnrichProdName
Talend Cloud
EnrichPlatform
Talend Data Catalog

Note: This bridge is not available in Talend Cloud Data Catalog by default. You need to set up a remote harvesting agent to import metadata from this source. For more information, see Deploying a remote harvesting agent in Talend Cloud Data Catalog Administration guide.

Bridge Requirements

This bridge:
  • requires the tool to be installed to access its SDK.

Bridge Specifications

Vendor Oracle
Tool Name Oracle Data Integrator (ODI)
Tool Version 11.1.1.6 to 12c
Tool Web Site http://www.oracle.com/technetwork/middleware/data-integrator/overview/index.html
Supported Methodology [Data Integration] Multi-Model, Data Store (Physical Data Model, Logical Data Model), ETL (Source and Target Data Stores, Transformation Lineage, Expression Parsing) via Java API
Data Profiling
Remote Repository Browsing for Model Selection
Multi-Model Harvesting
Incremental Harvesting

SPECIFICATIONS
Tool: Oracle / Oracle Data Integrator (ODI) version 11.1.1.6 to 12c via Java API
See http://www.oracle.com/technetwork/middleware/data-integrator/overview/index.html
Metadata: [Data Integration] Multi-Model, Data Store (Physical Data Model, Logical Data Model), ETL (Source and Target Data Stores, Transformation Lineage, Expression Parsing)
Component: OracleDataIntegrator version 11.0.0

OVERVIEW
This bridge imports models and mappings from an Oracle Data Integrator work repository.

REQUIREMENTS
The bridge relies on the Oracle Data Integrator java API to import metadata. Therefore, the Oracle Data Integrator tool must be properly installed and accessible from the machine executing this bridge.
This bridge does not supports odiref for ODI v12c.

FREQUENTLY ASKED QUESTIONS

Q: What are the configuration steps required for External Authentication (against an LDAP server) ?
A: Make sure the External Authentication mode is correctly configured for the ODI tool, then copy the files jps-config.xml and cwallet.sso to folder /MetaIntegration/bin/config/. Templates of the jps-config.xml and cwallet.sso files are located in the ODI installation folder (/Oracle_ODI1/modules/oracle.jps_11.1.1/client_config).
Please refer to the Oracle documentation for details:
http://www.oracle.com/webfolder/technetwork/tutorials/obe/fmw/odi/odi_11g/odi_external_authentication/odi_external_authentication.htm

Q: Why are there "NOT_MAPPED_PHYSICALLY" under Connections?
A: The 'NOT_MAPPED_PHYSICALLY' is a special name for a database schema in the situation when the Logical schema is not mapped to the Physical schema. In this case, we can't create a physical database with the real name and use this default name.

LIMITATIONS
Refer to the current general known limitations at http://metaintegration.com/Products/MIMB/MIMBKnownLimitations.html or bundled in Documentation/ReadMe/MIMBKnownLimitations.html
n/a

SUPPORT
Provide a trouble shooting package with:
- the debug log (can be set in the UI or in conf/conf.properties with MIR_LOG_LEVEL=6)
- the metadata backup if available (can be set in the Miscellaneous parameter with option -backup)

Q: How do I provide metadata to the support team in order to reproduce an issue?
A: In order to reproduce an issue, a backup of both the master and work repositories must be created. Instructions are available as part of the ODI documentation:
- Backup Master Repository: http://docs.oracle.com/cd/E21764_01/integrate.1111/e12643/export_import.htm#autoId10
- Backup Work Repository: http://docs.oracle.com/cd/E21764_01/integrate.1111/e12643/export_import.htm#autoId12

Q: How do I validate the ODI Home directory (Oracle Home path) ?
A: The directory structure depends from ODI version and contains many subfolders and many files, but we can check the directory structure using few files from list below:

For ODI 11 the root folder contains at least that 23 folders and 170 files.
[Oracle Home path]\modules\oracle.jps_11.1.1\jps-api.jar
[Oracle Home path]\modules\oracle.jdbc_11.1.1\ojdbc6dms.jar
[Oracle Home path]\modules\oracle.idm_11.1.1\identitystore.jar
[Oracle Home path]\oracledi.sdk\lib\odi-core.jar
[Oracle Home path]\setup\manual\oracledi-sdk\oracle.odi-sdk-jse_11.1.1.jar

For ODI 12.1.3 the root folder contains at least that 213 folders and 2461 files.
[Oracle Home path]\odi\sdk\lib\oracle.odi-sdk-jse_12.1.3.jar
[Oracle Home path]\oracle_common\modules\oracle.security-api_12.1.3.jar
[Oracle Home path]\oracle_common\modules\oracle.jps_12.1.3\jps-api.jar
[Oracle Home path]\oracle_common\modules\oracle.jdbc_12.1.0\ojdbc6.jar
[Oracle Home path]\oracle_common\modules\oracle.ldap_12.1.3\ldapjclnt11.jar


Bridge Parameters

Parameter Name Description Type Values Default Scope
Driver Name Driver Name to connect to master repository database. The full name (including the package name) of the class should be specified, e.g.

'Oracle'
oracle.jdbc.OracleDriver

'SQLServer'
sun.jdbc.odbc.JdbcOdbcDriver

'IBM DB2 8.1 Net driver'
COM.ibm.db2.jdbc.net.DB2Driver
STRING   oracle.jdbc.OracleDriver Mandatory
Oracle Home path The Oracle home directory where ODI Studio and SDK are installed. The SDK is used to read the ODI metadata. The directory should have 'odi' and 'oracle_common' sub-directories. The \odi\sdk\lib\oracle.odi-sdk-jse_[VERSION].jar file must be present. It is used to identify the ODI version and load dependent libraries. DIRECTORY     Mandatory
URL Driver URL to connect to the master repository's database.
URL is driver and database dependent.

'SQLServer'
jdbc:odbc:YOUR_DSN

'IBM DB2 8.1 Net'
jdbc:db2://COMPUTER_NAME_OR_IP:6789/DATABASE_NAME

'IBM DB2 8.1 App'
jdbc:db2:DATABASE_NAME

'Oracle Thin'
jdbc:oracle:thin:@COMPUTER_NAME_OR_IP:1521:ORACLE_SID

'Oracle Thin'
jdbc:oracle:thin:@(description=(address=(host=DATABASE_HOST)(protocol=tcp)(port=1521))(connect_data=(sid=SID)))

'NCR Teradata'
jdbc:odbc:YOUR_DSN
- When using the TeraDriver (native driver), jdbc:odbc:<DB_Server_Name>

'MS Access'
jdbc:odbc:YOUR_DSN
STRING   jdbc:oracle:thin:@hostname:port:sid Mandatory
Master Repository User User name used to connect to Master Repository's Database. STRING     Mandatory
Master Repository Password Password for specified Master Repository Database User. PASSWORD      
ODI User Oracle Data Integrator Username. STRING     Mandatory
ODI User Password Password for specified Oracle Data Integrator Username. PASSWORD     Mandatory
Work repository Work repository name. REPOSITORY_MODEL     Mandatory
Content Browsing Scope Specify the method used to browse ODI repository content.

'Projects'
Collection of Packages and Interfaces that can be selected individually or in bulk using their Folders and Projects.
Selecting a Package selects all its dependent objects, like Interfaces.
All selected objects are considered to be executable using the specified Context.

'Load Plans and Scenarios'
Collection of Load Plans and Scenarios that can be selected individually or in bulk using their Folders.
Selecting a Load Plans or Scenario selects all its dependent objects, like Packages and Interfaces.
ENUMERATED
Load Plans and Scenarios
Projects
Load Plans and Scenarios  
Content Specify the list of identifiers of objects you would like to import. Identifiers must be separated by the semicolon (;).

Identifiers use TYPE_ID format (e.g. LP_1234) where ID is the Internal ID of the ODI object.

ODI Studio UI shows Internal ID in the Version tab. Different types of ODI objects can have the same Internal ID.
A combination of Type and ID makes the object identifier unique.

The TYPE prefix can be:
'LP'
for Load Plans

'SC'
for Scenarios

'SF'
for Scenario Folders

'PR'
for Projects

'PF'
for Project Folders

'IN'
for Interfaces

'MP'
for ODI 12 Mappings

'PG'
for Packages

When you need to import all available projects but you cannot specify them explicitly (e.g. because projects come and go from the repository) you can specify the PR_ALL identifier that instructs the application to import all projects.

You can specify the list of identifiers using the browsing UI. The UI allows you to browse objects by either 'Projects' or 'Load Plans and Scenarios' type. You should use the Content Browsing Scope parameter to pick the browsing type.

The browsing UI needs to connect to the repository. You must specify all repository connection parameters before invoking the browsing UI.
REPOSITORY_SUBSET     Mandatory
Context Type the Context name (e.g. 'Global') or select a Context using a browser user interface. It is used to resolve Logical Schemas to Physical schemas.
The selected Context should define Physical Schemas for all Logical Schemas involved.
REPOSITORY_MODEL     Mandatory
JavaScript API Specify the ANSI file with ODI Substitution API JavaScript implementation.

This file contains definition of OdiRef class with required methods. You may copy/paste JavaScript example below and start coding in JavaScript to implement the OdiRef class, e.g.

function OdiRef () {
this.getSysDate = function() {
var d = new Date();
var m = d.getMonth() + 1;
var D = d.getDate();
var y = d.getFullYear();
return m + '/' + D + '/' + y;
}

// any others odiRef methods
}

var odiRef = new OdiRef();

You may use in your JavaScript code some predefined variables that represents the ODI expression context:

- isCatalogSupported
- isSchemaSupported
- technologyName
- contextName
- logicalSchemaName
- dataServerInstanceName
- catalogName
- workCatalogName
- databaseSchemaName
- workSchemaName
- errorTablePrefix
- integrationTablePrefix
- journalTablePrefix
- journalTriggerPrefix
- journalViewPrefix
- loadingTablePrefix
- temporaryIndexPrefix
- localObjectMask
- localSequenceMask
- partitionMask
- remoteObjectMask
- remoteSequenceMask
- subPartitionMask
- partitionName
FILE *.*    
Miscellaneous INTRODUCTION
Specify miscellaneous options starting with a dash and optionally followed by parameters, e.g.
-connection.cast MyDatabase1="SQL Server"
Some options can be used multiple times if applicable, e.g.
-connection.rename NewConnection1=OldConnection1 -connection.rename NewConnection2=OldConnection2;
As the list of options can become a long string, it is possible to load it from a file which must be located in ${MODEL_BRIDGE_HOME}\data\MIMB\parameters and have the extension .txt. In such case, all options must be defined within that file as the only value of this parameter, e.g.
ETL/Miscellaneous.txt

JAVA ENVIRONMENT OPTIONS
-m <Java Memory's maximum size>
1G by default on 64bits JRE or as set in conf/conf.properties, e.g.
-m 8G
-m 8000M

-j <Java Runtime Environment command line options>
This option must be the last one in the Miscellaneous parameter as all the text after -j is passed "as is" to the JRE, e.g.
-j -Dname=value -Xms1G
The following option must be set when a proxy is used to access internet (this is critical to access https://repo.maven.apache.org/maven2/ (and exceptionally a few other tool sites) in order to download the necessary third party software libraries.
-j -Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=3128 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=3128 -Dhttp.proxyUser=user -Dhttp.proxyPassword=pass -Dhttps.proxyUser=user -Dhttps.proxyPassword=pass

-jre <Java Runtime Environment full path name>
It can be an absolute path to javaw.exe on Windows or a link/script path on Linux, e.g.
-jre "c:\Program Files\Java\jre1.8.0_211\bin\javaw.exe"

-v <Environment variable value>
None by default, e.g.
-v var1=value1 -v var2="value2 with spaces"

MODEL IMPORT OPTIONS
-model.name <model name>
Override the model name, e.g.
-model.name "My Model Name"

-prescript <script name>
The script must be located in the bin directory, and have .bat or .sh extension.
The script path must not include any parent directory symbol (..).
The script should return exit code 0 to indicate success, or another value to indicate failure.
For example:
-prescript "script.bat arg1 arg2"

-cache.clear
Clears the cache before the import, and therefore will run a full import without incremental harvesting.
Warning: this is a system option managed by the application calling the bridge and should not be set by users.

-backup <directory>
Full path of an empty directory to save the metadata input files for further troubleshooting.

DATA CONNECTION OPTIONS
Data Connections are produced by the import bridges typically from ETL/DI and BI tools to refer to the source and target data stores they use. These data connections are then used by metadata management tools to connect them (metadata stitching) to their actual data stores (e.g. databases, file system, etc.) in order to produce the full end to end data flow lineage and impact analysis. The name of each data connection is unique by import model. The data connection names used within DI/BI design tools are used when possible, otherwise connection names are generated to be short but meaningful such as the database / schema name, the file system path, or Uniform Resource Identifier (URI). The following options allows to manipulate connections. These options replaces the legacy options -c, -cd, and -cs.

-connection.cast ConnectionName=ConnectionType
Casts a generic database connection (e.g. ODBC/JDBC) to a precise database type (e.g. ORACLE) for SQL Parsing, e.g.
-connection.cast "My Database"="SQL SERVER".
The list of supported data store connection types includes:
ACCESS
CASSANDRA
DB2
DENODO
HIVE
MYSQL
NETEZZA
ORACLE
POSTGRESQL
PRESTO
REDSHIFT
SALESFORCE
SAP HANA
SNOWFLAKE
SQL SERVER
SYBASE
TERADATA
VECTORWISE
VERTICA

-connection.rename OldConnection=NewConnection
Renames an existing connection to a new name, e.g.
-connection.rename OldConnectionName=NewConnectionName
Multiple existing database connections can be renamed and merged into one new database connection, e.g.
-connection.rename MySchema1=MyDatabase -connection.rename MySchema2=MyDatabase

-connection.split oldConnection.Schema1=newConnection
Splits a database connection into one or multiple database connections.
A single database connection can be split into one connection per schema, e.g.
-connection.split MyDatabase
All database connections can be split into one connection per schema, e.g.
-connection.split *
A database connection can be explicitly split creating a new database connection by appending a schema name to a database, e.g.
-connection.split MyDatabase.schema1=MySchema1

-connection.map DestinationPath=SourcePath
Maps a source path to destination path. This is useful for file system connections when different paths points to the same object (directory or file).
On Hadoop, a process can write into a CSV file specified with the HDFS full path, but another process reads from a HIVE table implemented (external) by the same file specified using a relative path with default file name and extension, e.g.
-connection.map hdfs://host:8020/users/user1/folder/file.csv=/user1/folder
On Linux, a given directory (or file) like /data can be referred to by multiple symbolic links like /users/john and /users/paul, e.g.
-connection.map /users/John=/data -connection.map /users/paul=/data
On Windows, a given directory like C:\data can be referred to by multiple network drives like M: and N:, e.g.
-connection.map M:\=C:\data -connection.map N:\=C:\data

-pppd
Create the connections and connection data sets in DI/ETL design models. This feature should only be used when intending to export to another DI/ETL tool.
STRING      

 

Bridge Mapping

Meta Integration Repository (MIR)
Metamodel
(based on the OMG CWM standard)
"Oracle Data Integrator (ODI)"
Metamodel
Oracle Data Integrator 12 (Mapping)
Mapping Comments
     
QueryColumn Column  
Description Description  
Name Name  
Optional Mandatory  
Position Order  
QueryFilter Filter  
Description Description  
Name Name  
QueryJoin Join  
Description Description  
JoinType Type  
Name Name  
QueryLookup Lookup  
Description Description  
Name Name  
QueryMapping Mapping, Reusable Mapping  
CreationTime Creation Time  
Description Description  
ModificationTime Modification Time  
Name Name  
NativeId Internal Id  
QueryTable Set Input, Input Signature, Distinct, Set, Datastore, Expression, Output Signature, Aggregate  
Name Name  
SqlQuery Reusable Mapping Component  
Name Name  

Meta Integration Repository (MIR)
Metamodel
(based on the OMG CWM standard)
"Oracle Data Integrator (ODI)"
Metamodel
Oracle Data Integrator 12 (Package)
Mapping Comments
     
ActivityModel Package  
Description Description  
Name Name  
NativeId Internal Id  
EmbeddedCallStep Start Load Plan, Mapping, Start Scenario  
Description Description  
Name Name  
NativeId Internal Id  
GenericStep Unknown  
Name Name  

Meta Integration Repository (MIR)
Metamodel
(based on the OMG CWM standard)
"Oracle Data Integrator (ODI)"
Metamodel
Oracle Data Integrator 12 (Repository Browse)
Mapping Comments
     
DiDesignActivityContent Load Plan, Scenario  
CreationTime Creation Time  
Description Description  
LastModificationTime Last Modification Time  
Name Name  
NativeId Internal Id  
DirectoryStructureModel ODI Repository  
CreationTime Creation Time  
Description Description  
ImportDate Import Date  
ModificationTime Modification Time  
Name Name  
NativeType Native Type  
Type Type  
Folder Scenario Folder  
CreationTime Creation Time  
Description Description  
LastModificationTime Last Modification Time  
Name Name  
NativeId Internal Id  

Meta Integration Repository (MIR)
Metamodel
(based on the OMG CWM standard)
"Oracle Data Integrator (ODI)"
Metamodel
Oracle Data Integrator 12 (Model)
Mapping Comments
     
Attribute Column  
Description Description  
InitialValue Initial Value  
Name Name  
Optional Is Null  
Position Order  
CandidateKey Alternate Key, Primary Key  
Description Description  
Name Name  
NativeId Internal Id  
Class Datastore  
Description Description  
Name Name  
NativeId Internal Id  
PhysicalName Alias  
DatabaseSchema Schema  
Description Description  
Name Name  
NativeId Internal Id  
DesignPackage Sub-Model  
Description Description  
Name Name  
NativeId Internal Id  
FlatTextFile FileDatastore  
Description Description  
Name Name  
NativeId Internal Id  
PhysicalName Alias  
Index Not Unique Index  
Description Description  
Name Name  
NativeId Internal Id  
StoreModel Model  
CreationTime Creation Time  
Description Description  
ModificationTime Modification Time  
Name Name  
StoreType Store Type  
SystemMajorVersion System Major Version  
SystemMinorVersion System Minor Version  
SystemReleaseVersion System Release Version  
SystemType System Type  

Meta Integration Repository (MIR)
Metamodel
(based on the OMG CWM standard)
"Oracle Data Integrator (ODI)"
Metamodel
Oracle Data Integrator 12 (Repository Organization)
Mapping Comments
     
Connection Runtime Connection  
Name Name  
DiDesignActivityContent Executable Activity, Package, Load Plan, Scenario  
CreationTime Creation Time  
Description Description  
LastModificationTime Last Modification Time  
Name Name  
NativeId Internal Id  
DiParameterContent Variables, Sequences, Runtime Context  
CreationTime Creation Time  
Description Description  
LastModificationTime Last Modification Time  
Name Name  
NativeId Internal Id  
DiQueryMappingContent Executable Mapping  
Description Description  
Name Name  
NativeId Internal Id  
DiRuntimeActivityContent Scenario Package Summary Lineage, Executable Summary Lineage, Runtime Package, Scenario Interface Summary Lineage, Load Plan Summary Lineage  
Description Description  
Name Name  
NativeId Internal Id  
DiRuntimeTaskContent Runtime Mapping  
Name Name  
DirectoryStructureModel ODI Repository  
CreationTime Creation Time  
Description Description  
ImportDate Import Date  
ModificationTime Modification Time  
Name Name  
NativeType Native Type  
Type Type  
Folder Context, Physical Architecture, Reusable Mappings, Database Schema, Projects, Topology, Contexts, Project, Mappings, Logical Architecture, Model Folder, Technology, Packages, Scenario Folder, Context Item, Logical Schema, Project Folder, Models, Load Plans and Scenarios, Physical Schema  
CreationTime Creation Time  
Description Description  
LastModificationTime Last Modification Time  
Name Name  
NativeId Internal Id  
LineageSummaryContent LineageSummary  
Description Description  
Name Name  
NativeId Internal Id  
ParameterDefinition Variable, Sequence  
Description Description  
Name Name  
NativeId Internal Id  
ParameterModel Repository Variables, Repository Sequences  
Description Description  
Name Name  
NativeId Internal Id  
StoreContent Connection, Model, Physical Connection  
CreationTime Creation Time  
Description Description  
LastModificationTime Last Modification Time  
Name Name  
NativeId Internal Id  

Meta Integration Repository (MIR)
Metamodel
(based on the OMG CWM standard)
"Oracle Data Integrator (ODI)"
Metamodel
Oracle Data Integrator 12 (Repository Browse Projects)
Mapping Comments
     
Connection Runtime Connection  
Name Name  
DiDesignActivityContent Package, Load Plan, Scenario  
CreationTime Creation Time  
Description Description  
LastModificationTime Last Modification Time  
Name Name  
NativeId Internal Id  
DiParameterContent Runtime Context  
Name Name  
DiQueryMappingContent Executable Mapping  
Description Description  
Name Name  
NativeId Internal Id  
DiRuntimeActivityContent Scenario Package Summary Lineage, Runtime Package, Scenario Interface Summary Lineage, Load Plan Summary Lineage  
Name Name  
DiRuntimeTaskContent Runtime Mapping  
Name Name  
DirectoryStructureModel ODI Repository  
CreationTime Creation Time  
Description Description  
ImportDate Import Date  
ModificationTime Modification Time  
Name Name  
NativeType Native Type  
Type Type  
Folder Context, Project, Reusable Mappings, Mappings, Model Folder, Packages, Scenario Folder, Project Folder, Projects  
CreationTime Creation Time  
Description Description  
LastModificationTime Last Modification Time  
Name Name  
NativeId Internal Id  
StoreContent Model, Physical Connection  
CreationTime Creation Time  
Description Description  
LastModificationTime Last Modification Time  
Name Name  
NativeId Internal Id  

Meta Integration Repository (MIR)
Metamodel
(based on the OMG CWM standard)
"Oracle Data Integrator (ODI)"
Metamodel
Oracle Data Integrator 12 (Mapping)
Mapping Comments
     
QueryColumn Column  
Description Description  
Name Name  
Optional Mandatory  
Position Order  
QueryFilter Filter  
Description Description  
Name Name  
QueryJoin Join  
Description Description  
JoinType Type  
Name Name  
QueryLookup Lookup  
Description Description  
Name Name  
QueryMapping Mapping, Reusable Mapping  
CreationTime Creation Time  
Description Description  
ModificationTime Modification Time  
Name Name  
NativeId Internal Id  
QueryTable Set Input, Input Signature, Distinct, Set, Datastore, Expression, Output Signature, Aggregate  
Name Name  
SqlQuery Reusable Mapping Component  
Name Name