Open Studio for Data Quality
Profiling data from Google BigQuery requires to go through a JDBC connection setting.
For more information, see how to build the Connection URL.
The RECORD data type is not supported.
When you set up a JDBC connection, specify each jar file extracted from the zip file.
- Go to the Hive server configuration.
- Set the
HiveServer2 Java Heap Sizeparameter to at least 1 GB.
In the connection wizard, you must select from the Distribution list the platform that hosts Hive. You must also set the Hive version and model.
For more information, see http://hadoop.apache.org/.
If you decide to change the user name in an embedded mode of a Hive connection, you must restart the Studio before being able to successfully run the profiling analyses that use the connection.
Click the button next to Hadoop Properties and in the
open dialog box click the [+] button to add two lines to
Enter the parameters names as
Set their values to the by-default value 1000.
This value is normally appropriate for running the computations.
|Unsupported indicators||Unsupported functions||Unsupported analyses|
|With SQL engine:
Soundex Low Frequency
Upper Quartile and Lower Quartile
All Date Frequency indicators
The only analysis that is not supported for Hive is Time Correlation Analysis as the Date data type does not exist in Hive. For further information on this analysis type, see Time correlation analyses.
All right-click options on analysis results that generate Jobs to validate, standardize, and deduplicate data are not supported for Hive. For further information about these Jobs, see Validating data.
When you select to connect to a Hive database to create and execute different analyses, then in the connection wizard, you must, as explained above, select from the Distribution contextual menu list the platform that hosts Hive.
- The parameter is yarn.application.classpath
- The value is
Microsoft SQL Server
Microsoft SQL Server 2012 and later are supported.
If you select to connect to the Microsoft SQL Server database with Windows Authentication Mode, you can select Microsoft or JTDS open source from the Db Version list.
When creating a connection to MySQL using JDBC, it is not mandatory to include the database name to the JDBC URL. Regardless of whether the database connection URL specified in the JDBC URL field includes the database name, all databases are retrieved.
For example, if you specify jdbc:mysql://192.168.33.41:3306/tbi?noDatetimeStringSync=true, where
tbi is the database name, or jdbc:mysql://192.168.33.41:3306/?noDatetimeStringSync=true, all databases are retrieved.
To support surrogate pairs in data and metadata, you need to edit the following properties in the MySQL server configuration file:
[client] default-character-set=utf8mb4 [mysql] default-character-set=utf8mb4 character-set-server=utf8mb4
The Netezza database does not support regular expressions. If you want to use regular expressions with this database, you must:
- Install the SQL Extensions Toolkit package on a Netezza system. Use the regex_like function provided in this toolkit in the SQL template as documented in https://www.ibm.com/docs/en/psfa/7.2.1?topic=toolkit-sql-extensions-installation-setup.
- Add the indicator definition for Netezza in the Pattern Matching folder in Talend Studio under .
The query template you need to define for Netezza is as the following:
SELECT COUNT(CASE WHEN REGEXP_LIKE(<%=COLUMN_NAMES%>,<%=PATTERN_EXPR%>) THEN 1 END), COUNT FROM <%=TABLE_NAME%><%=WHERE_CLAUSE%>.
To support surrogate pairs, the
NLS_CHARACTERSET parameter of the database must be set to UTF8 or AL32UTF8.
NLS_CHARACTERSET parameters are:
SQL> SELECT * FROM NLS_DATABASE_PARAMETERS;
When you connect to a PostgreSQL database using a JDBC connection, the
INT8 data types are replaced by a
String data type. As a consequence, if your analysis uses the
T-swoosh algorithm, the survivorship functions are for strings, not for numbers.
- Close the analysis and switch to the Integration perspective.
- Expand Metadata and right-click the database connection .
- Select the check box of the table to update.
- When Creation status is set to Success, click Next.
- If columns with no database type must be of
Integertype, set DB Type to INT.
- Click Finish and close the dialog box.
- Switch to the Profiling perspective and open the analysis.
- In Survivorship Rules for Columns, delete and add back the columns you updated. You can see the survivorship functions for numbers (Largest and Smallest).
Profiling data from SAP HANA is only possible for Table, View, and Calculation View schemas.
The Soundex frequency statistics indicators support the English alphabet only.
Profiling data from Snowflake requires a JDBC connection.
For more information, see https://docs.snowflake.net/manuals/user-guide/jdbc-configure.html.
In the Teradata database, the regular expression function is installed by default only starting from version 14. If you want to use regular expressions with older versions of this database, you must install a User Defined Function in Teradata and add the indicator definition for Teradata in Talend Studio.