Metadata Filtering in a Worksheet

A filter panel can be presented to the left of the results. Talend Data Catalog provides a sophisticated set of post filtering options.

Filter Preferences

You may specify filter Preferences including:

• Apply Filters Automatically – When checked, if you update or define a new filter, the results are updated automatically. Otherwise, the an APPY button appears in the UI and you must click this button to update the results.
• Ignore Connection Objects – not show objects which are part of connection definitions, such as ETL sources and targets, or BI sources. These are not a part of the actual data store models (databases, files, etc.) and are often not important for data analysis.

Apply Filters Automatically or Manually

Depending upon the worksheet Apply Filters Automaticallypreferences settings, worksheets will either:

• Apply Filters Automatically is checked – Every action, such as to update the filter, add a column set preferences, etc., will cause the worksheet to re-run the query
• Apply Filters Automatically is un-checked – You must click the APPLY button (which is only available in this mode) to re-run the query. The button is highlighted when a refresh of the query would produce new results.

Filter by example by column

You may always filter the results by what is already presented. E.g., if one wishes to only see columns for Type View Column, you may simply right-click a cell in the grid that contains that value and select that option.

Using the Filter Panel

You may also expand the Filters panel on the left in order to limit the results based upon advanced matching criteria.

Common Filter Criteria

Use +FILTER to add addition filter criteria, criteria, click the X next to a specific filter to clear that one, or double-click on the name of the filter to clear it.

Special Columns Which May Not be Used in Filters

These are documented in the Developer’s Guide as it is a fundamental feature of Metadata Query Language (MQL). Obviously, they will not appear in the list of possible filters as a result of the +FILTER button.

Remove a Filter Criteria from the Worksheet

Click the X that appears in the upper right corner of the criterion in the Filter panel.

Define the Order of the Filter Criteria

To change the order of a particular filter criterion click on the Configure icon that appears in the upper right corner of the criterion in the Filter panel and select Move Up or Move Down.

Filter by Numeric Attributes

Filtering criteria may be based upon the <,<=, >, >= or BETWEEN (Note: NOT is not supported) operators on all numeric data profiling filters and all integer attribute types.

Steps

1. Open a worksheet by any of the create methods detailed, or go to
   • WORKSHEETS > Manage,
   • WORKSHEETS > RECENT >,
   • WORKSHEETS > FAVORITES >.
2. Click +FILTER and add a numeric column.
3. Specify the criteria, <,<=, >, >= or BETWEEN
4. Click outside the dialog box.

Example

Create a worksheet (search) for Fields in Files.

Click +FILTER and select Data Profiling Distinct, specify less than and specify 6.

Filter by Search Text

You may enter text to filter by, and the text will be matched to the properties specified, as follows.

Steps

1. Open a worksheet by any of the create methods detailed, or go to
   • WORKSHEETS > Manage,
   • WORKSHEETS > RECENT >,
   • WORKSHEETS > FAVORITES >.
2. Click the Search Text and enter the search text.
3. Optionally, specify Search Options, including
   • What attributes to search within for the text
   • Include Semantic Search
   • Obtain Help on the search
4. Click outside the dialog box.

Example

Open a worksheet by clicking the Search box in the upper right corner, entering “Invoice” and pressing ENTER. Click Search Options

Filter by Category

You may specify from a hierarchical pick list which object types to include in the results. Thus, you may include, e.g., both Terms and Database Columns in the search results.

Steps

1. Open a worksheet by any of the create methods detailed, or go to
   • WORKSHEETS > Manage,
   • WORKSHEETS > RECENT >,
   • WORKSHEETS > FAVORITES >.
2. Click the pull-down under CATEGORY and check the box for one or more categories of object types.
3. Click outside the dialog box.

Example

Create a worksheet and click the pull-down under CATEGORY and check the box for Fields in Files.

More Information

The list of categories includes:

• System Objects – E.g., Worksheets, Dashboards, Collections, Groups, Users

• Import Model types – E.g., File, Database, Data Modeling, Tableau, InformaticaPowerCenter, etc.

• Custom Models and Objects – e.g., Glossaries, Data Mappings, Business Rules, etc.

Filter by Model

Use this filter to restrict the results to only those which are contained within particular models (e.g., a specific imported model or glossary).

In addition, if there is a directory structure in a model (e.g., a database import containing multiple schemas or a repository import containing directory structure, models, connections, etc.) then one may also use checkboxes to filter the results.

Steps

1. Open a worksheet by any of the create methods detailed, or go to
   • WORKSHEETS > Manage,
   • WORKSHEETS > RECENT >,
   • WORKSHEETS > FAVORITES >.
2. Click +FILTER and select Model.
3. Click in the dropdown box and select one or more models in which the results will be included.
4. Click outside the dialog box.

Example

Create a worksheet and filter on the text “Invoice”. Then, click +FILTER and select Model.

Click SHOW MORE.

Filter by Updated Date and Updated By

Filter to only those results which have an Updated Date value in a specific date range.

Steps

1. Open a worksheet by any of the create methods detailed, or go to
   • WORKSHEETS > Manage,
   • WORKSHEETS > RECENT >,
   • WORKSHEETS > FAVORITES >.
2. Click +FILTER and select Updated Date.
3. Click on the filter condition, including:
   • equals
   • not equals
   • today
   • yesterday
   • last hour
   • last 7 days
   • last 60 days
   • last 90 days
   • last year
   • before
   • after
   • between
4. Specify a value for the condition.
5. Click outside the dialog box.

Example

Create a worksheet and filter on the text “Invoice”. Then, click +FILTER and select Updated Date.

Click the filter Condition (equals by default).

Select last 7 days.

Dates in Worksheets

See Dates in the Product.

Filter by Object Roles

Filter by Object Role responsibility assignment (to users or groups), including Steward. Results then then include all the matching objects which have the particular responsibility assignment specified in the Filter.

Steps

1. Open a worksheet by any of the create methods detailed, or go to
   • WORKSHEETS > Manage,
   • WORKSHEETS > RECENT >,
   • WORKSHEETS > FAVORITES >.
2. Click +FILTER and scroll down to the OBJECT ROLES section.
3. Select a specific role.
4. Click on the filter condition, including:
   • equals
   • not equals
   • equals any
   • not equals any
   • equals all
   • not equals all
   • exists
   • not exists
5. Specify a value for the condition.
6. Click in the dropdown box and select one or more users and/or groups that the object role is assigned to.
7. Click outside the dialog box.

Example

Create a worksheet and filter on the text “Invoice”. Then, click +FILTER and select Data User.

Select exists as the filter condition.

E.g., click the model Staging DW in which the first result (Invoice) is located.

Go to the Responsibilities tab.

Filter by Stewardship

In order to filter by stewardship, you must filter by object roles based upon responsibility assignment.

Filter by Label

Filter to only those results which have one of the selected labels assigned.

Steps

1. Open a worksheet by any of the create methods detailed, or go to
   • WORKSHEETS > Manage,
   • WORKSHEETS > RECENT >,
   • WORKSHEETS > FAVORITES >.
2. Click +FILTER and select Label.
3. Select one or more LABLES or All.

Example

Create a worksheet and filter on the text “Invoice”. Then, click +FILTER and select Label.

Click QuarterlyReview in the faceted search filter and only three items are displayed.

Filter by Curation Status

You may endorse, certify, warn and comment on any object.

You may then filter to only those results which have any of these or any of these from a specific set of individuals.

Steps

1. Open a worksheet by any of the create methods detailed, or go to
   • WORKSHEETS > Manage,
   • WORKSHEETS > RECENT >,
   • WORKSHEETS > FAVORITES >.
2. Click +FILTER and scroll down to Certified.

3. Click on the filter condition, including:
   • equals
   • not equals
4. Specify a value for the condition.

Example

Create a worksheet and filter on the text “Customer”. Then, click +FILTER and select Certified.

Select equals as the filter condition.

Select True for the value of the condition.

Filter by Lineage Status

Object in a configuration may have (or may not have) data flow and/or semantic lineage. You may then filter to only those results which have any of these or do not have lineage of a particular type.

You may use any of the four filter criteria:

You may specify:

• Exists – Filters to all objects for which the filter can apply (classifier and feature level objects)
• Not Exists - Filters to all objects for which the filter does not apply (classifier and feature level objects)
• Equals (True/False) – Filters to all objects which have or do not have the lineage.

Example

Search for “Invoice” to create a worksheet. Click +FILTER and select Has Data Impact.

Select equals and True for the condition. Also, it could be useful to add the Has Data Impact column.

Detection of Unused Data Elements in the Architecture

Asking the question the other way, i.e., what objects do not have data impact and are also in the data warehouse, will show you all the data elements which are not being used.

Click +FILTER and select Has Data Impact.

Select Equals and False for the condition.

Use the Model filter to only search Staging DW.dbo.

Also, it could be useful to add the Has Data Impact column.

Filter by Sensitivity Label

You may filter by the data classifications which are associated (tagged) on an object. These filters include:

• Sensitivity Label – The current sensitivity label associated with the object
• Sensitivity Label Approved – The current approved (or manually assigned) sensitivity label associated with the object
• Sensitivity Label Lineage Proposed – inferred from other object using data flow analysis
• Sensitivity Label Data Proposed – detected during the data classification process
• Sensitivity Label Rejected – All labels which were rejected and will not be assigned automatically as a part of data classification.

You may filter by the data classifications which are associated (tagged) on an object. These filters include:

• Data Classifications – Data classes associated with the object. The attribute shows both approved (or manually assigned) data classes as well as proposed ones.
• Data Classifications Approved – Data classes approved for (or manually assigned to) the object.
• Data Classifications Matched – Data classes detected during the data classification process and thus associated with the object.
• Data Classifications Proposed – Data Classes proposed during the data classification process but are not yet approved or rejected.
• Data Classifications Rejected – Data classes which were rejected for the object.

All of the following data profiling result properties may be used as filters and as columns in a worksheet:

• “Data Profiling”.”Distinct”
• “Data Profiling”.Duplicate
• “Data Profiling”.Valid
• “Data Profiling”.Empty
• “Data Profiling”.Invalid
• “Data Profiling”.Min
• “Data Profiling”.Max
• “Data Profiling”.Mean
• “Data Profiling”.Variance
• “Data Profiling”.Median
• “Data Profiling”.”Lower Quantile”
• “Data Profiling”.”Upper Quantile”
• “Data Profiling”.”Avg Length”
• “Data Profiling”.”Min Length”
• “Data Profiling”.”Max Length”
• “Data Profiling”.”Inferred Data Types”

You may filter by which collections each object in the result is a member of.

You may specify multiple collections that the results may be a member of.

You may specify:

• equals: If any of the collections contain an object it is included in the results (same as equals any)

• not equals: If any of the collections contain an object it is not included in the results
• equals any: If any of the collections contain an object it is included in the results (same as equals)
• not equals any: If any of the collections contain an object it is not included in the results (same as not equals)
• equals all: If any of the collections contain an object it is included in the results
• not equals all: If any of the collections contain an object it is included in the results
• exists: If any of the collections contain an object it is included in the results
• not exists: If none of the collections contain an object it is included in the results

You may filter by the existence or content of comments just as you may do so for curations such as endorsement and warning.

Filter by Object Image

The object image is a single image which may be added to the object page Overview tab optionally.

The filter option is object image.

Filter by Relationship

As we can see from the MANAGE > Metamodelpage, there is an association between the two different models for business policies and business rules.

We populated these in the Custom Models section, with worksheet examples there.

Filter by Data Documentation/Cataloging

As an effective tool to:

• Identify documented data elements
• Identify those data elements which still require documentation

You may define a worksheet which shows term defined, locally defined, mapped and inferred documentation (i.e., Name and Business Definition) for an imported object, e.g., the Fields of a data lake.

Please see examples in the Data Documentation section, such as an inferred documentation worksheet.

Filter by System Objects

As you may now include “system” objects, including:

• Worksheets
• Dashboards
• Collections
• Groups
• Users

in your worksheet searches, you may filter a worksheet by these objects. The objects may now be included in the Metadata Query Language (MQL) query.

Example

Open a new worksheet and under CATEGORY > System > Groups.

Metadata Query Language and Query Filters

MQL allows users to define powerful and complex metadata queries with a familiar SQL syntax.

The MQL is available through the REST API and constitutes the foundation of the search / worksheet page. For syntax details, refer to the Developer Guide.

Anything you do in the Filters is reflected in the MQL. You may also edit the Filters from the perspective of the MQL directly by using the Advanced model.

Steps

1. Open a worksheet by any of the create methods detailed, or go to
   • WORKSHEETS > Manage,
   • WORKSHEETS > RECENT >,
   • WORKSHEETS > FAVORITES >
2. In the Results panel, click Preferences > SHOW CURRENT QUERY in the worksheet Preferences.
3. Click +FILTER and +QUERY to add MQL directly as additional filters.

Example

Go to WORKSHEETS > Manage > Certified Terms.

Click SHOW CURRENT QUERY in the Worksheet results preferences.

Click +FILTER and +QUERY.

Click EDIT under the MQL FILTER, and build the query for Comment Count greater than or equal to 1.

Click OK.

Name the filter “Highly Commented”.