MDM query language and REST data access - 7.1

EnrichVersion
7.1
EnrichProdName
Talend Data Fabric
Talend MDM Platform
task
Data Governance > Consolidating data
Data Governance > Retrieving data
EnrichPlatform
Talend MDM Server
Talend MDM Web UI

MDM query language and REST data access

Talend MDM proposes a REST API that enables users to perform CRUD (Create/Request/Update/Delete) operations on MDM records in a more convenient manner.

This article focuses mainly on the read and write operations of the MDM REST API, and then gives examples of basic and more complex queries that you can include in REST API calls.

This article applies to all Talend Platform products with MDM 6.0 and later.

MDM REST API resources

Talend MDM REST API is a modern, developer-friendly API that simplifies the interfaces to MDM, allowing developers to interact with MDM data more easily and directly.

Talend MDM Web UI provides access to the MDM REST resources documentation, which allows you to refer to the query language syntax and related parameters and test the execution of operations.

You can also integrate data API and transactions API. For more information, see Transactions API integration.

You can enable the support of cross-origin resource sharing (CORS) in MDM when developing a web application consuming MDM REST resources. For more information, see Cross-origin resource sharing support in MDM.

Create a record

Creates a new record in the specified data container. The record will be provided in the request content as XML.
Request
POST /talendmdm/services/rest/data/{containerName}

The body is an XML representation of the record to be created. The parameter content type is application/xml or text/xml.

Query parameters
  • containerName: This is a String value which specifies the name of the data container in which you want to create a new data record.
  • container: This is a String value which represents the type of the data container. It is either MASTER (default) or STAGING.
  • updateReport: This is a Boolean value which controls whether to generate a journal entry (update report). By default, it is true.
  • beforeSaving: This is a Boolean value which controls whether to invoke a Before Saving process. By default, it is false. If the beforeSaving parameter is set to true, the updateReport parameter must also be set to true, since a journal entry (update report) needs to be generated to invoke the Before Saving process.
Headers
  • Content-Type: application/xml or text/xml
  • Authorization: Basic Authentication scheme
Response No content.
Status
  • 200 OK: The operation is executed successfully.
  • 400 BAD REQUEST: The request contains invalid parameter, for example, invalid storage name, invalid storage type, or wrong XML document.
  • 401 UNAUTHORIZED: Login fails, invalid username or password.
  • 403 FORBIDDEN: Missing required permission, for example, no required WRITE permission.
  • 404 NOT FOUND: The resource does not exist, for example, invalid service URL.
  • 500 INTERNAL SERVER ERROR: Other errors, for example, the Before Saving process returns error.
Limitation This REST API does not support creating a new record for the Role entity in the PROVISIONING system data container.

Sample request

<Product>
  <Picture>/imageserver/upload/TalendShop/tshirt.jpg?width=150&amp;amp;height=90&amp;amp;preserveAspectRatio=true</Picture>
  <Id>231035936</Id>
  <Name>Talend Fitted T-Shirt</Name>
  <Description>Fitted T. ultra-fine combed ring spun cotton</Description>
  <Features>
    <Sizes/>
    <Colors/>
  </Features>
  <Availability>false</Availability>
  <Price>15.99</Price>
  <Family>[1]</Family>
  <OnlineStore>Talend Shop@@http://www.cafepress.com/Talend.231035936</OnlineStore>
  <Stores>
    <Store></Store>
  </Stores>
</Product>

Update a record

Updates a data record completely in the specified data container. The record will be provided in the request content as XML.

If you want to preserve some existing values, use the partial update operation instead. For more information, see Update a record partially.

Request
PUT /talendmdm/services/rest/data/{containerName}

The body is an XML representation of the record to be updated. The element Id in the request body determines which record will be updated. The parameter content type is application/xml.

Query parameters
  • containerName: This is a String value which specifies the name of the data container in which you want to update a data record.
  • container: This is a String value which represents the type of the data container. It is either MASTER (default) or STAGING.
  • updateReport: This is a Boolean value which controls whether to generate a journal entry (update report). By default, it is true.
  • beforeSaving: This is a Boolean value which controls whether to invoke a Before Saving process. By default, it is false. If the beforeSaving parameter is set to true, the updateReport parameter must also be set to true, since a journal entry (update report) needs to be generated to invoke the Before Saving process.
Headers
  • Content-Type: application/xml
  • Authorization: Basic Authentication scheme
Response No content.
Status
  • 200 OK: The operation is executed successfully, and the whole data record is replaced by the provided input.
  • 400 BAD REQUEST: The request contains invalid parameter, for example, nonexistent storage name, wrong XML document, invalid storage type, or nonexistent record Id.
  • 401 UNAUTHORIZED: Login fails, invalid username or password.
  • 403 FORBIDDEN: Missing required permission, for example, no required WRITE permission.
  • 404 NOT FOUND: The resource does not exist, for example, invalid service URL.
  • 500 INTERNAL SERVER ERROR: Other errors, for example, the Before Saving process returns error.
Limitation This REST API does not support updating a record for the Role entity in the PROVISIONING system data container.

Sample request

<Product>
  <Picture>/imageserver/upload/TalendShop/tshirt.jpg?width=150&amp;amp;height=90&amp;amp;preserveAspectRatio=true</Picture>
  <Id>231035936</Id>
  <Name>Talend T-Shirt</Name>
  <Description>New Description</Description>
  <Features>
    <Sizes/>
    <Colors/>
  </Features>
  <Availability>false</Availability>
  <Price>15.99</Price>
  <Family>[1]</Family>
  <OnlineStore>Talend Shop@@http://www.cafepress.com/Talend.231035936</OnlineStore>
  <Stores>
    <Store></Store>
  </Stores>
</Product>

Delete a container

Deletes a container, either a Master one and a Staging one.
Request
DELETE /talendmdm/services/rest/data/{containerName}
Query parameters
  • containerName: This is a String value which specifies the name of the data container you want to delete.
  • drop: This is a Boolean value which determines if all data such as full text index will be deleted.
Headers
  • Authorization: Basic Authentication scheme
Response No content.
Status
  • 204 NO CONTENT: The delete operation is executed successfully.
  • 401 UNAUTHORIZED: Login fails, invalid username or password.
  • 404 NOT FOUND: The resource does not exist, for example, invalid service URL.
  • 500 INTERNAL SERVER ERROR: Other errors, for example, nonexistent storage name.

Update a record partially

Updates a data record partially in the specified data container. The record will be provided in the request content as XML.
Request
PATCH /talendmdm/services/rest/data/{containerName}

Although PATCH is a valid HTTP verb, not all REST tools support it. For more information, see http://stackoverflow.com/questions/4260084/testing-tools-for-rest-that-support-http-patch.

The body is an XML representation of part of the record to be updated. The element Id in the request body determines which record is to be updated. The parameter content type is application/xml.

Query parameters
  • containerName: This is a String value which specifies the name of the data container in which you want to update a data record partially.
  • container: This is a String value which represents the type of the data container. It is either MASTER (default) or STAGING.
  • updateReport: This is a Boolean value which controls whether to generate a journal entry (update report). By default, it is true.
Headers
  • Content-Type: application/xml
  • Authorization: Basic Authentication scheme
Response No content.
Status
  • 200 OK: The operation is executed successfully, and the data record identified by the element Id is updated partially according to the provided input.
  • 400 BAD REQUEST: The request contains invalid parameter, for example, nonexistent storage name, wrong XML document, invalid storage type, or nonexistent record Id.
  • 401 UNAUTHORIZED: Login fails, invalid username or password.
  • 403 FORBIDDEN: Missing required permission, for example, no required WRITE permission.
  • 404 NOT FOUND: The resource does not exist, for example, invalid service URL.
  • 500 INTERNAL SERVER ERROR: Other errors.
Limitation This REST API does not support updating a record partially for the Role entity in the PROVISIONING system data container.

Sample request

<Product>
  <Id>1</Id>
  <Name>New Name</Name>  
</Product>

Create or update records in batch mode

Creates or updates multiple data records pertaining to one entity in the specified data container in batch mode. The records will be provided in the request content as XML.

You can also perform partial update of many records that pertain to one entity in one single bulk operation through the REST API. For more information, see How to perform mass partial update on MDM data records through the REST API.

Request
POST /talendmdm/services/rest/data/{containerName}/batch

The body is an XML representation of one or more records to be created or updated, which are enclosed by a root tag with an arbitrary name, records for an example. Each record contains:

  • one or more primary key fields with values, which are mandatory because they will be used to specify the record you want to create or update. If the record does not exist in the data container, it will be created. If the record already exists in the data container, it will be updated accordingly.
  • all other fields defined for the specified entity, in which the mandatory fields must have values.

The parameter content type is application/xml.

Query parameters
  • containerName: This is a String value which specifies the name of the data container in which you want to create or update multiple data records.
  • container: This is a String value which represents the type of the data container. It is either MASTER (default) or STAGING.
  • updateReport: This is a Boolean value which controls whether to generate a journal entry (update report). By default, it is false.
  • beforeSaving: This is a Boolean value which controls whether to invoke a Before Saving process. By default, it is false. If the beforeSaving parameter is set to true, the updateReport parameter must also be set to true, since a journal entry (update report) needs to be generated to invoke the Before Saving process.
Headers
  • Content-Type: application/xml
  • Authorization: Basic Authentication scheme
Response No content.
Status
  • 200 OK: The operation is executed successfully.
  • 400 BAD REQUEST: The request contains invalid parameter, for example, invalid storage name, invalid storage type, or wrong XML document.
  • 401 UNAUTHORIZED: Login fails, invalid username or password.
  • 403 FORBIDDEN: Missing required permission, for example, no required WRITE permission.
  • 404 NOT FOUND: The resource does not exist, for example, invalid service URL.
  • 500 INTERNAL SERVER ERROR: Other errors, for example, the Before Saving process returns error.
Limitation This REST API does not support creating or updating multiple records in batch mode for the Role entity in the PROVISIONING system data container.

Sample request

<records>
  <Product>
    <Picture>/imageserver/upload/TalendShop/tshirt.jpg?width=150&amp;amp;height=90&amp;amp;preserveAspectRatio=true</Picture>
    <Id>231035935</Id>
    <Name>Talend Fitted T-Shirt</Name>
    <Description>Fitted T. ultra-fine combed ring spun cotton</Description>
    <Features>
      <Sizes/>
      <Colors/>
    </Features>
    <Availability>false</Availability>
    <Price>15.99</Price>
    <Family>[1]</Family>
    <OnlineStore>Talend Shop@@http://www.cafepress.com/Talend.231035936</OnlineStore>
    <Stores>
      <Store>[TalendFR]</Store>
    </Stores>
  </Product>
  <Product>
    <Picture>/imageserver/upload/TalendShop/tshirt.jpg?width=150&amp;amp;height=90&amp;amp;preserveAspectRatio=true</Picture>
    <Id>231035936</Id>
    <Name>Talend Fitted T-Shirt</Name>
    <Description>Fitted T. ultra-fine combed ring spun cotton</Description>
    <Features>
      <Sizes/>
      <Colors/>
    </Features>
    <Availability>false</Availability>
    <Price>15.99</Price>
    <Family>[1]</Family>
    <OnlineStore>Talend Shop@@http://www.cafepress.com/Talend.231035936</OnlineStore>
    <Stores>
      <Store>[TalendFR]</Store>
    </Stores>
  </Product>
  <Product>
    <Picture>/imageserver/upload/TalendShop/tshirt.jpg?width=150&amp;amp;height=90&amp;amp;preserveAspectRatio=true</Picture>
    <Id>231035937</Id>
    <Name>Talend Fitted T-Shirt</Name>
    <Description>Fitted T. ultra-fine combed ring spun cotton</Description>
    <Features>
      <Sizes/>
      <Colors/>
    </Features>
    <Availability>false</Availability>
    <Price>15.99</Price>
    <Family>[1]</Family>
    <OnlineStore>Talend Shop@@http://www.cafepress.com/Talend.231035936</OnlineStore>
    <Stores>
      <Store>[TalendFR]</Store>
    </Stores>
  </Product>
</records>

Delete records by query

Deletes one or more records that match the provided query. Query is provided in the request content as JSON.
Request
PUT /talendmdm/services/rest/data/{containerName}/delete

The body is a query language in JSON format. The parameter content type is application/json.

Query parameters
  • containerName: This is a String value which specifies the name of the data container from which the data record(s) will be deleted.
  • container: This is a String value which represents the type of the data container. It is either MASTER (default) or STAGING.
  • updateReport: This is a Boolean value which controls whether to generate a journal entry (update report). By default, it is false.
Headers
  • Content-Type: application/json
  • Authorization: Basic Authentication scheme
Response No content.
Status
  • 200 OK: The operation is executed successfully, and all records that match the provided query are deleted.
  • 400 BAD REQUEST: The request contains invalid parameter, for example, invalid storage name, invalid storage type, or wrong XML document.
  • 401 UNAUTHORIZED: Login fails, invalid username or password.
  • 403 FORBIDDEN: Missing required permission, for example, no required WRITE permission.
  • 404 NOT FOUND: The resource does not exist, for example, invalid service URL.
  • 500 INTERNAL SERVER ERROR: Other errors.
Limitation This REST API does not support deleting records by query for the Role entity in the PROVISIONING system data container.

Sample request

To delete the Product instances where Product/Price is greater than 18, use the following query:

{
   "select": {	
      "from": ["Product"],
      "where": {
         "gt":[
           {"field": "Product/Price"},
           {"value": "18"}
         ]
      }
   }
}

Get XML documents

Gets many XML documents from the database.
Request
POST /talendmdm/services/rest/data/{containerName}/documents

The body is a unique ids array. The parameter content type is application/json.

Query parameters
  • containerName: This is a String value which specifies the name of the data container from which you want to query the record.
  • encoding: It represents the char set of all request parameters and defaults to UTF-8.
Headers
  • Content-Type: application/json
  • Authorization: Basic Authentication scheme
  • Accept: application/json
Response A JSON array that contains the documents of the records.
Status
  • 200 OK: The validation operation is executed successfully, no matter what the validation result is.
  • 400 BAD REQUEST: The request contains invalid parameter, for example, invalid storage name.
  • 403 FORBIDDEN: Missing required permission, for example, no required READ permission.
  • 404 NOT FOUND: The resource does not exist, for example, invalid service URL.
  • 500 INTERNAL SERVER ERROR: Other errors.
Limitation This REST API does not support getting XML documents for the Role entity in the PROVISIONING system data container.

Sample request

[
  "Product.Product.231035933",
  "Product.Product.231035934",
  "Product.Product.231035935"
]

Sample response

[
  	"<ii>
	  <c>Product</c>
	  <dmn>Product</dmn>
	  <dmr/>
	  <sp/>
	  <t>1531809768682</t>
	  <taskId>null</taskId>
	  <i>231035933</i>
	  <p>
		<Product>
		  <Id>231035933</Id>
		  <Name>Talend Dog T-Shirt</Name>
		  <Description>Doggie t-shirt from American Apparel</Description>
		  <Features>
			<Sizes>
			  <Size>Small</Size>
			  <Size>Medium</Size>
			  <Size>Large</Size>
			  <Size>X-Large</Size>
			</Sizes>
			<Colors>
			  <Color>White</Color>
			</Colors>
		  </Features>
		  <Availability>true</Availability>
		  <Price>16.99</Price>
		  <Family>[14]</Family>
		  <OnlineStore>Talend Shop@@http://www.cafepress.com/Talend.231035933</OnlineStore>
		  <Stores/>
		</Product>
	  </p>
	</ii>",
	"<ii>
	  <c>Product</c>
	  <dmn>Product</dmn>
	  <dmr/>
	  <sp/>
	  <t>1531809768659</t>
	  <taskId>null</taskId>
	  <i>231035934</i>
	  <p>
		<Product>
		  <Id>231035934</Id>
		  <Name>Talend Jr. Spaghetti Tank</Name>
		  <Description>Spaghetti tank from American Apparel</Description>
		  <Features>
			<Sizes>
			  <Size>Small</Size>
			  <Size>Medium</Size>
			  <Size>Large</Size>
			  <Size>X-Large</Size>
			</Sizes>
			<Colors>
			  <Color>White</Color>
			  <Color>Light Blue</Color>
			  <Color>Light Pink</Color>
			  <Color>Lemon</Color>
			</Colors>
		  </Features>
		  <Availability>true</Availability>
		  <Price>16.99</Price>
		  <Family>[8]</Family>
		  <OnlineStore>Talend Shop@@http://www.cafepress.com/Talend.231035934</OnlineStore>
		  <Stores/>
		</Product>
	  </p>
	</ii>",
	"<ii>
	  <c>Product</c>
	  <dmn>Product</dmn>
	  <dmr/>
	  <sp/>
	  <t>1531809767902</t>
	  <taskId>null</taskId>
	  <i>231035935</i>
	  <p>
		<Product>
		  <Id>231035935</Id>
		  <Name>Talend Golf Shirt</Name>
		  <Description>Golf-style, collared t-shirt</Description>
		  <Features>
			<Sizes>
			  <Size>Medium</Size>
			</Sizes>
			<Colors>
			  <Color>White</Color>
			</Colors>
		  </Features>
		  <Availability>false</Availability>
		  <Price>16.99</Price>
		  <Family>[8]</Family>
		  <OnlineStore>Talend Shop@@http://www.cafepress.com/Talend.231035935</OnlineStore>
		  <Stores/>
		</Product>
	  </p>
	</ii>"
]

Get all documents unique ID

Gets the unique IDs of all documents as array.
Request
GET /talendmdm/services/data/{containerName}/documents/uniqueIds
Query parameters
  • containerName: This is a String value which specifies the name of the data container from which you want to get the unique ids of all documents.
  • type: This is a String value which specifies the name of the entity.
Headers
  • Authorization: Basic Authentication scheme
  • Accept: application/json
Response A JSON array that contains all primary keys of the entity in the specified container.
Status
  • 200 OK: The query operation is executed successfully, and the documents unique ids of the specified entity are returned in an array.
  • 400 BAD REQUEST: The request contains invalid parameter, for example, invalid storage name or invalid storage type.
  • 401 UNAUTHORIZED: Login fails, invalid username or password.
  • 403 FORBIDDEN: Missing required permission, for example, no required READ permission.
  • 404 NOT FOUND: The resource does not exist, for example, invalid service URL.
  • 500 INTERNAL SERVER ERROR: Other errors.
Limitation This REST API does not support getting the unique IDs of all documents for the Role entity in the PROVISIONING system data container.

Sample response

[
  "Product.Product.231035935",
  "Product.Product.231035941",
  "Product.Product.231035937",
  "Product.Product.231035938",
  "Product.Product.231035939",
  "Product.Product.231035940",
  "Product.Product.231035934",
  "Product.Product.231035936",
  "Product.Product.231035933"
]

Get an XML string of a document

Gets an XML string of a document.
Request
GET /talendmdm/services/data/{containerName}/documents/{uniqueId}
Query parameters
  • containerName: This is a String value which specifies the name of the data container from which you want to get an XML string of a document.
  • uniqueId: This is a String value which represents the unique id of the document.
  • encoding: It represents the char set of the result and defaults to UTF-8.
Headers
  • Authorization: Basic Authentication scheme
  • Accept: application/xml
Response The document of the record in the specified container.
Status
  • 200 OK: The query operation is executed successfully, and the record that matches the provided ID will be returned as a string.
  • 400 BAD REQUEST: The request contains invalid parameter, for example, invalid storage name or invalid storage type.
  • 401 UNAUTHORIZED: Login fails, invalid username or password.
  • 403 FORBIDDEN: Missing required permission, for example, no required READ permission.
  • 404 NOT FOUND: The resource does not exist, for example, invalid service URL.
  • 500 INTERNAL SERVER ERROR: Other errors.
Limitation This REST API does not support getting an XML string of a document for the Role entity in the PROVISIONING system data container.

Sample response

<ii>
    <c>Product</c>
    <dmn>Product</dmn>
    <dmr/>
    <sp/>
    <t>1531728848296</t>
    <taskId>null</taskId>
    <i>1</i>
    <p>
        <Product>
            <Id>1</Id>
            <Name>New Name</Name>
            <Description>2</Description>
            <Features>
                <Sizes>
                    <Size>Small</Size>
                    <Size>Medium</Size>
                    <Size>Large</Size>
                </Sizes>
                <Colors>
                    <Color>White</Color>
                    <Color>Light Blue</Color>
                    <Color>Light Pink</Color>
                </Colors>
            </Features>
            <Availability>true</Availability>
            <Price>3.00</Price>
            <Stores>
            </Stores>
        </Product>
    </p>
</ii>

Get records by query

Gets data records from the specified data container. Query is provided in the request content as JSON.

You can also retrieve a snapshot of a specific MDM record and navigate through the record history. For more information, see MDM Time Machine.

Request
PUT /talendmdm/services/data/{containerName}/query

The body is a query language in JSON format. The parameter content type is application/json.

Query parameters
  • containerName: This is a String value which specifies the name of the data container from which you want to read data records.
  • container: This is a String value which represents the type of the data container. It is either MASTER (default) or STAGING.
Headers
  • Content-Type: application/json
  • Authorization: Basic Authentication scheme
  • Accept: application/xml or application/json
Response The documents of the records in the specified container.
Status
  • 200 OK: The operation is executed successfully.
  • 400 BAD REQUEST: The request contains invalid parameter, for example, invalid storage name or invalid storage type.
  • 401 UNAUTHORIZED: Login fails, invalid username or password.
  • 404 NOT FOUND: The resource does not exist, for example, invalid service URL.
  • 500 INTERNAL SERVER ERROR: Other errors.
Limitation This REST API does not support getting records by query for the Role entity in the PROVISIONING system data container.

Sample request

{
    "select": { 
        "from": ["Product"],
        "where": {
            "gt":[
                {"field": "Product/Price"},
                {"value": "16"}
            ]   
        }
    }
}

Sample response in XML format

<results>
   <Product>
      <Picture>/imageserver/upload/TalendShop/dog.jpg</Picture>
      <Id>231035933</Id>
      <Name>partialNameUpdate1</Name>
      <Description>partialDescriptionUpdate1</Description>
      <Features>
         <Sizes>
            <Size>Small</Size>
            <Size>Medium</Size>
            <Size>Large</Size>
            <Size>X-Large</Size>
         </Sizes>
         <Colors>
            <Color>White</Color>
         </Colors>
      </Features>
      <Availability>true</Availability>
      <Price>111.00</Price>
      <Family>[14]</Family>
      <OnlineStore>Talend Shop@@http://www.cafepress.com/Talend.231035933</OnlineStore>
      <Stores/>
   </Product>
   <Product>
      <Picture>/imageserver/upload/TalendShop/tshirt.jpg?width=150&amp;amp;height=90&amp;amp;preserveAspectRatio=true</Picture>
      <Id>231035935</Id>
      <Name>partialNameUpdate3</Name>
      <Description>partialDescriptionUpdate3</Description>
      <Features>
         <Sizes>
            <Size>Medium</Size>
         </Sizes>
         <Colors>
            <Color>White</Color>
         </Colors>
      </Features>
      <Availability>false</Availability>
      <Price>333.00</Price>
      <Family>[1]</Family>
      <OnlineStore>Talend Shop@@http://www.cafepress.com/Talend.231035936</OnlineStore>
      <Stores/>
   </Product>
</results>

Sample response in JSON format

[
  {
    "product": {
      "picture": "/imageserver/upload/TalendShop/dog.jpg",
      "id": "1",
      "name": "New Name",
      "description": "Doggie t-shirt from American Apparel",
      "features": {
        "sizes": {
          "size": [
            "Small",
            "Medium",
            "Large",
            "X-Large"
          ]
        },
        "colors": {
          "color": [
            "White"
          ]
        }
      },
      "availability": "true",
      "price": "16.99",
      "family": "[14]",
      "onlinestore": "Talend Shop@@http://www.cafepress.com/Talend.231035933",
      "stores": {}
    }
  },
  {
    "product": {
      "picture": "/imageserver/upload/TalendShop/dog.jpg",
      "id": "231035933",
      "name": "Talend Dog T-Shirt",
      "description": "Doggie t-shirt from American Apparel",
      "features": {
        "sizes": {
          "size": [
            "Small",
            "Medium",
            "Large",
            "X-Large"
          ]
        },
        "colors": {
          "color": [
            "White"
          ]
        }
      },
      "availability": "true",
      "price": "16.99",
      "family": "[14]",
      "onlinestore": "Talend Shop@@http://www.cafepress.com/Talend.231035933",
      "stores": {}
    }
  },
  {
    "product": {
      "picture": "/imageserver/upload/TalendShop/spaghetti.jpg",
      "id": "231035934",
      "name": "Talend Jr. Spaghetti Tank",
      "description": "Spaghetti tank from American Apparel",
      "features": {
        "sizes": {
          "size": [
            "Small",
            "Medium",
            "Large",
            "X-Large"
          ]
        },
        "colors": {
          "color": [
            "White",
            "Light Blue",
            "Light Pink",
            "Lemon"
          ]
        }
      },
      "availability": "true",
      "price": "16.99",
      "family": "[8]",
      "onlinestore": "Talend Shop@@http://www.cafepress.com/Talend.231035934",
      "stores": {}
    }
  }
]

Validate records in a container

Validates records in the specified container. The records will be provided in the request content as XML.
Request
POST /talendmdm/services/rest/data/{containerName}/validate

The body is an XML representation of one or more records to be validated, which are enclosed by a root tag with an arbitrary name, records for an example.

Query parameters
  • containerName: This is a String value that represents the name of the container in which the records will be validated.

  • container: This is a String value which represents the type of the data container. It is either MASTER (default) or STAGING.
  • beforeSaving: This is a Boolean value which controls whether to invoke a Before Saving process associated to the record(s) during validation. By default, it is true. The validation operation should be an immutable action, thus it allows to disable the Before Saving process which in some cases can modify the data.

  • returnSource: This is a Boolean value which indicates whether to include the input records in the response. By default, it is false.

Headers
  • Content-Type: application/xml or text/xml
  • Authorization: Basic Authentication scheme
  • Accept: application/xml or application/json
Response The list of the validation result for each record in JSON format, including the validity status, a message from the validation rule or the Before Saving process or empty, and optionally the input record as an echo.
Status
  • 200 OK: The validation operation is executed successfully, no matter what the validation result is.
  • 400 BAD REQUEST: The request contains invalid parameter, for example, invalid storage name or wrong XML document.

  • 403 FORBIDDEN: Missing required permission, for example, no required WRITE permission.
  • 404 NOT FOUND: The resource does not exist, for example, invalid storage type.
  • 500 INTERNAL SERVER ERROR: Other errors.
Limitation This REST API does not support validating records for the following entity:
  • the User entity in the PROVISIONING system data container,
  • the Role entity in the PROVISIONING system data container,
  • the Conf entity in the CONF system data container,
  • the AutoIncrement entity in the CONF system data container,
  • the BrowseItem entity in the SearchTemplate system data container, and
  • the HierarchySearchItem entity in the SearchTemplate system data container.

Sample request

<records> 
    <ProductFamily> 
        <Name>Literature</Name> 
        <Status>Approved</Status> 
    </ProductFamily> 
    <ProductFamily> 
        <Name>Comics</Name> 
        <Status>Unknown</Status> 
    </ProductFamily> 
</records>

Sample response in XML format

<results>
   <result>
      <isValid>false</isValid>
      <message>&lt;msg>[EN:Validation faild]&lt;/msg></message>
      <sourceXml>
         <ProductFamily>
            <Name>Literature</Name>
            <Status>Approved</Status>
         </ProductFamily>
      </sourceXml>
   </result>
   <result>
      <isValid>false</isValid>
      <message></message>
      <sourceXml>
         <ProductFamily>
            <Name>Comics</Name>
            <Status>Unknown</Status>
         </ProductFamily>
      </sourceXml>
   </result>
</results>

Sample response in JSON format

[
  {
    "isValid": false,
    "message": "<msg>[EN:Validation faild]</msg>\n",
    "sourceXml": "<ProductFamily><Name>Literature</Name><Status>Approved</Status></ProductFamily>"
  },
  {
    "isValid": true,
    "message": "",
    "sourceXml": "<ProductFamily><Name>Comics</Name><Status>Unknown</Status></ProductFamily>"
  }
]

List primary keys for container and type

Lists all primary keys for container and type.
Request
GET /talendmdm/services/rest/data/{containerName}/{type}
Query parameters
  • container: This is a String value which represents the type of the data container. It is either MASTER (default) or STAGING.
  • containerName: This is a String value which represents the name of data container.
  • type: This is a String value which specifies the name of the entity.
Headers
  • Authorization: Basic Authentication scheme
  • Accept: application/json
Response Returns the json array that contains all primary keys of entity in container.
Status
  • 200 OK: The operation is executed successfully.
  • 400 BAD REQUEST: The request contains invalid parameter, for example, invalid storage name or invalid storage type.
  • 401 UNAUTHORIZED: Login fails, invalid username or password.
  • 403 FORBIDDEN: Missing required permission, for example, no required READ permission.
  • 404 NOT FOUND: The resource does not exist, for example, invalid service URL.
  • 500 INTERNAL SERVER ERROR: Other errors.
Limitation This REST API does not support listing primary keys for the Role entity in the PROVISIONING system data container.

Sample response

[
  {
    "$explicitprojection$": {
      "id": "231035933"
    }
  },
  {
    "$explicitprojection$": {
      "id": "231035934"
    }
  },
  {
    "$explicitprojection$": {
      "id": "231035935"
    }
  },
  {
    "$explicitprojection$": {
      "id": "231035936"
    }
  },
  {
    "$explicitprojection$": {
      "id": "231035937"
    }
  }
]

Update records partially in one bulk operation

Performs partial update of records in one bulk operation.
Request
PATCH /talendmdm/services/rest/data/{containerName}/{type}/bulk

The body is an XML representation of records to be partial updated. The parameter content type is application/xml.

Query parameters
  • containerName: This is a String value which specifies the name of the data container in which you want to update records partially in one bulk operation.
  • type: This is a String value which specifies the name of the entity.
  • container: This is a String value which represents the type of the data container. It is either MASTER (default) or STAGING.
  • updateReport: This is a Boolean value which controls whether to generate a journal entry (update report). By default, it is true.
Headers
  • Content-Type: application/xml
  • Authorization: Basic Authentication scheme
Response No content.
Status
  • 200 OK: The operation is executed successfully.
  • 400 BAD REQUEST: The request contains invalid parameter, for example, invalid storage name, invalid storage type, or wrong XML document.
  • 401 UNAUTHORIZED: Login fails, invalid username or password.
  • 403 FORBIDDEN: Missing required permission, for example, no required WRITE permission.
  • 404 NOT FOUND: The resource does not exist, for example, invalid service URL.
  • 500 INTERNAL SERVER ERROR: Other errors.
Limitation This REST API does not support updating records partially in one bulk operation for the Role entity in the PROVISIONING system data container.

Sample request

<records>
	<Product>
		<Id>231035933</Id>
		<Name>partialNameUpdate1</Name>
		<Description>partialDescriptionUpdate1</Description>
		<Price>111</Price>
	</Product>
	<Product>
		<Id>231035934</Id>
		<Name>partialNameUpdate2</Name>
		<Description>partialDescriptionUpdate2</Description>
		<Price>222</Price>
	</Product>
	<Product>
		<Id>231035935</Id>
		<Name>partialNameUpdate3</Name>
		<Description>partialDescriptionUpdate3</Description>
		<Price>333</Price>
	</Product>
</records>

Get a record by ID

Gets a data record by ID.
Request
GET /talendmdm/services/rest/data/{containerName}/{type}/{id}
Query parameters
  • containerName: This is a String value which represents the name of the container where you want to query the record.
  • container: This is a String value which represents the type of the data container. It is either MASTER (default) or STAGING.
  • type: This is a String value which specifies the name of the entity.
  • id: This is a String value which specifies the ID of the record.
  • datetime: This is a String value that specifies the date at which the record you want to retrieve was updated. The date can be provided as number of milliseconds since EPOCH or an XML formatted date.
  • swing: It controls how datetime is interpreted, CLOSEST, AFTER, or BEFORE.
Headers
  • Authorization: Basic Authentication scheme
  • Accept: application/json
Response The content of the record in JSON array format.
Status
  • 200 OK: The operation is executed successfully.
  • 400 BAD REQUEST: The request contains invalid parameter, for example, invalid storage name.
  • 403 FORBIDDEN: Missing required permission, for example, no required READ permission.
  • 404 NOT FOUND: The resource does not exist, for example, invalid service URL.
  • 500 INTERNAL SERVER ERROR: Other errors.
Limitation This REST API does not support getting a record by ID for the Role entity in the PROVISIONING system data container.

Sample response

[
  {
    "product": {
      "picture": "/imageserver/upload/TalendShop/dog.jpg",
      "id": "231035933",
      "name": "Talend Dog T-Shirt",
      "description": "Doggie t-shirt from American Apparel",
      "features": {
        "sizes": {
          "size": [
            "Small",
            "Medium",
            "Large",
            "X-Large"
          ]
        },
        "colors": {
          "color": [
            "White"
          ]
        }
      },
      "availability": "false",
      "price": "16.99",
      "family": "[10]",
      "onlinestore": "Talend Shop@@http://www.cafepress.com/Talend.231035933",
      "stores": {}
    }
  }
]

Delete a record by ID

Deletes a data record by the specified ID.
Request
DELETE /talendmdm/services/rest/data/{containerName}/{type}/{id}
Query parameters
  • containerName: This is a String value which specifies the name of the data container from which you want to delete a data record.
  • container: This is a String value which represents the type of the data container. It is either MASTER (default) or STAGING.
  • type: This is a String value which specifies the name of the entity from which you want to delete a data record.
  • id: The ID of the data record to be deleted. For composite IDs, you can use a dot-separated value. For example, to delete an instance Address with the ID 1 in the data container Customer, use http://localhost:8180/talendmdm/services/rest/data/Customer/Address/1, and to delete an instance Address with the ID 1 and 2, use http://localhost:8180/talendmdm/services/rest/data/Customer/Address/1.2.
  • updateReport: This is a Boolean value which controls whether to generate a journal entry (update report). By default, it is true.
Headers
  • Content-Type: application/json
  • Authorization: Basic Authentication scheme
Response No content.
Status
  • 200 OK: The operation is executed successfully, and the data record specified by the element Id will be deleted.
  • 400 BAD REQUEST: The request contains invalid parameter, for example, invalid storage name or invalid entity name.
  • 403 FORBIDDEN: Missing required permission.
  • 404 NOT FOUND: The resource does not exist, for example, invalid service URL.
  • 500 INTERNAL SERVER ERROR: Other errors.
Limitation This REST API does not support deleting a record by ID for the Role entity in the PROVISIONING system data container.

Query language

In the REST API calls, you can use the query language to narrow down the data records of interest, order the query results or implement other functions according to your needs.

SELECT clauses

You can use basic SELECT clauses in REST API calls.

Select all instances of an entity

You can query all instances of an entity. For example, to return all instances of "Type1" records, use the query:

{
  "select": {
    "from": ["Type1"]
  }
}

Note that the "from" element is an array type, allowing you to select multiple types. This does not mean the query can return all Type1 and Type2 instances. The array is used for joins.

Paging: start and limit

Paging may be needed in some use cases such as building a table. This can be done with "start" and "limit" elements.

For example, to return results which start with the first instance of "Type1" (index is 0 based), together with the next 10 instances, use the query:

{
  "select": {
    "from": ["Type1"],
    "start": 0,
    "limit": 10
  }
}
Pay attention to the following:
  • Standard execution of such queries implies a creation of a list of size limit on the server side. If you specify a large value, it may lead to performance issues or an OutOfMemoryError. Therefore, make sure to use paging with relatively small values of limit.

  • A limit of Integer.MAX_VALUE (2^31 -1) is considered by MDM code as "no limit". Make sure to use a streaming strategy instead of a list of results. However, this means a limit of (2^31 - 2) will try to create a list of that size (usually causing OutOfMemoryError).

Select field values

Instead of querying the whole record, you can query only one field in the record.

For example, to return values of "id" field of "Type1" type, use the query:

{
  "select": {
    "from": ["Type1"],
    "fields": [
      {"field": "Type1/id"}
    ]
  }
}

To select more than one field, repeat the "field" element. For example, to return values of "id", "value1" and "value2" for each record of "Type1", use this query:

{
  "select": {
    "from": ["Type1"],
    "fields": [
      {"field": "Type1/id"},
      {"field": "Type1/value1"},
      {"field": "Type1/value2"}
    ]
  }
}

Aliasing

For naming or client reasons, you may want to alias the returned element. Aliasing can help differentiate values in case of name collision. This can be achieved with the "alias" element.

For example, to return all "id" values in all Type1 instances, but instead of wrapping values inside an element "id", the wrapping element will be named "a", use the query:

{
  "select": {
    "from": ["Type1"],
    "fields": [
      {"field": "Type1/id"},
      {
        "alias": [
          {"name": "a"},
          {"field": "Type1/id"}
        ]
      }
    ]
  }
}

For example, use the query to return results with "Type1/containedField1/value1" wrapped in "v1" and "Type1/containedField2/value1" in "v2", thus avoiding the possible naming collision that both "Type1/containedField1/value1" and "Type1/containedField2/value1" would be returned wrapped in a "value1" element:

{
  "select": {
    "from": ["Type1"],
    "fields": [
      {
        "alias": [
          {"name": "v1"},
          {"field": "Type1/containedField1/value1"}
        ]
      },
      {
        "alias": [
          {"name": "v2"},
          {"field": "Type1/containedField2/value1"}
        ]
      }
    ]
  }
}

Distinct field values

You can use the "distinct" keyword to get all distinct values for an entity field.

For example:

{
  "select": {
    "from": ["Type1"],
    "fields": [
      {"distinct": {"field": "Type1/value1"}}
    ]
  }
}

This query will return all distinct values for "value1" in type "Type1".

You can also alias a result using the "alias" element. For example:

{
  "select": {
    "from": ["Type1"],
    "fields": [
      {
        "alias": [
          {"name": "a0"},
          {"distinct": {"field": "Type1/value1"}}
        ]
      }
    ]
  }
}

This returns all distinct values for "value1" but inside an element named "a0".

Count

You can count how many results are returned by a query.

For example, to count how many Type1 instances are available, use the query:

{
  "select": {
    "from": ["Type1"],
    "fields": [
      {"count": {}}
    ]
  }
}

You can also combine the count operation with conditions to count how many instances match the specified criteria.

Note: The count function cannot be used together with the contains condition operand or the full text search.

For example, to return how many instances of Type1 have a value1 greater than 1, use the query:

{
  "select": {
    "from": ["Type1"],
    "fields": [
      {"count": {}}
    ],
    "where": {
      "gt": [
        {"field": "Type1/value1"},
        {"value": "1"}
      ]
    }
  }
}

Count may also be used in order_by clauses.

Maximum and minimum

You can select the maximum and/or minimum value for a field in an entity.

For example, to retrieve the smallest value of "id" in the entity Type1, use the query:

{
  "select": {
    "from": ["Type1"],
    "fields": [
      {"min": {"field": "Type1/id"}}
    ]
  }
}

For example, to retrieve the largest value of "id" in Type1, use the query:

{
  "select": {
    "from": ["Type1"],
    "fields": [
      {"max": {"field": "Type1/id"}}
    ]
  }
}

For example, to retrieve both the largest and the smallest values of "id" in Type1, use the query:

{
  "select": {
    "from": ["Type1"],
    "fields": [
      {"max": {"field": "Type1/id"}},
      {"min": {"field": "Type1/id"}}
    ]
  }
}

Order by field's value

You can use as many "order by" clauses as needed in a query either in descending or ascending order to set the order in the returned result.

For example, to select all instances of Type1 and order results by id (from the largest to the smallest) and value1 (from the smallest to the largest), use the query:

{
  "select": {
    "from": ["Type1"],
    "order_bys": [
      {
        "order_by": [
          {"field": "Type1/id"},
          {"direction": "DESC"}
        ]
      },
      {
        "order_by": [
          {"field": "Type1/value1"},
          {"direction": "ASC"}
        ]
      }
    ]
  }
}

Order by field value occurrence

You can also order by field value occurrence so that the most common value is returned first down to the least common value (in case of DESC direction).

Here is one query example to return the most common value for "value1" down to the least common:

{
  "select": {
    "from": ["Type1"],
    "fields": [
      {"field": "Type1/value1"},
    ],
    "order_bys": [
      {
        "order_by": [
          {"count": {"field": "Type1/value1"}},
          {"direction": "DESC"}
        ]
      }
    ]
  }
}

You can also include "limit" in the query to obtain the top N common values.

Here is one query example to obtain the top 10 values:

{
  "select": {
    "from": ["Type1"],
    "fields": [
      {"field": "Type1/value1"},
    ],
    "order_bys": [
      {
        "order_by": [
          {"count": {"field": "Type1/value1"}},
          {"direction": "DESC"}
        ]
      }
    ],
    "limit": 10
  }
}
Note: This kind of query does not work if you are using the DB2 or MS SQL Server database, because they don't support the order by clause with the count function.

Conditions

Value comparisons (=, <, <=, >, >=)

For example, use the query below to return all instances of Type1 where "id" equals 1:

{
  "select": {
    "from": ["Type1"],
    "where": {
      "eq": [
        {"field": "Type1/id"},
        {"value": "1"}
      ]
    }
  }
}

Bear in mind the following:

  • If the field is of numeric type (int, short, integer, long, float, double, decimal) and you provide a numeric value for it, the value can be interpreted with or without being enclosed in double quotes. Otherwise, the value must be enclosed in double quotes. Make sure that the value matches the element type. For example, "value": "abc" is incorrect if the field "Type1/id" is of numeric type.
  • The structure is always "operand": { field, value }.

The following operands are available for use:

Query language operand

Meaning

eq

equals

gt / gte

greater than/greater than or equals

lt / lte

less than / less than or equals

startsWith

starts with

contains

contains

Index based condition

You may specify an index for your condition in case the field is a repeatable element. The index is 0-based.

For example, to return all instances of Type1 where the second value of "list" equals "1", use the query:

{
  "select": {
    "from": ["Type1"],
    "where": {
      "eq": [
        {"index": [
          {"field": "Type1/list"},
          1
        ]},
        {"value": "1"}
      ]
    }
  }
}

Conditions between fields

You may use conditions between different fields. Only "eq" (EQUALS) is currently supported for such comparisons.

For example, to return the instances of Type1 where "id" equals "value1", use the query:

{
  "select": {
    "from": ["Type1"],
    "where": {
      "eq": [
        {"field": "Type1/id"},
        {"value": {"field": "Type1/value1"}}
      ]
    }
  }
}

in operator

You can use the operator in within a query.

For example, to return all instances of Type1 where "id" equals 1 or 2 or 5, use the query:

{
  "select": {
    "from": ["Type1"],
    "where": {
      "in": [
        {"field": "Type1/id"},
        {"value": ["1","2","5"]}
      ]
    }
  }
}

Bear in mind the following:

  • If the field is of numeric type (int, short, integer, long, float, double, decimal) and you provide a numeric value for it, the value can be interpreted with or without being enclosed in double quotes. Otherwise, the value must be enclosed in double quotes. Make sure that the value matches the element type. For example, "value": ["a", "b", "c"] is incorrect if the field "Type1/id" is of numeric type.
  • The "value" is formatted as a JSON array, and the values listed must be static. For example, ["value1", "value2", "value3"...].
  • Only simple type fields at the root level are supported. However, the mandatory multi-occurrence foreign key fields are not supported.
  • The operator in cannot be used together with the full text search.
  • The operator in can work with the operator not. For more information, see the section not operator.

Logic operators

To allow more complex queries, boolean operators (and/or/not) may be included. The structure is similar to what is previously presented "operator": { left, right } where left and/or right can be logic operators too.

For example, to return all Type1 instances where "id = 1 OR value1 = 0", use the query:

{
  "select": {
    "from": ["Type1"],
    "where": {
      "or": [
        {
          "eq": [
            {"field": "Type1/id"},
            {"value": "1"}
          ]
        },
        {
          "eq": [
            {"field": "Type1/value1"},
            {"value": "0"}
          ]
        }
      ]
    }
  }
}

A binary logic operator (and/or) must have 2 and only 2 children. For example, you can write a condition with "a OR b OR c":

{
  "select": {
    "from": ["Type1"],
    "where": {
      "or": [
        {
          "eq": [
            {"field": "Type1/id"},
            {"value": "1"}
          ]
        },
        {
          "or": [
            {
              "eq": [
                {"field": "Type1/id"},
                {"value": "2"}
              ]
            },
            {
              "eq": [
                {"field": "Type1/value1"},
                {"value": "4"}
              ]
            }
          ]
        }
      ]
    }
  }
}

You can also use mixed logic operators. There is no ambiguity in the evaluation order of conditions.

For example, to return all Type1 instances where "(id = 1 OR (id = 2 and value1 = 4))", use the query which mix both and and or:

{
  "select": {
    "from": ["Type1"],
    "where": {
      "or": [
        {
          "eq": [
            {"field": "Type1/id"},
            {"value": "1"}
          ]
        },
        {
          "and": [
            {
              "eq": [
                {"field": "Type1/id"},
                {"value": "2"}
              ]
            },
            {
              "eq": [
                {"field": "Type1/value1"},
                {"value": "4"}
              ]
            }
          ]
        }
      ]
    }
  }
}  

not operator

The operator not is supported and follows the structure "not": { condition }.

For example, to return all Type1 instances where "id" is not equal to 1, use the query:

{
  "select": {
    "from": ["Type1"],
    "where": {
      "not": {
        "eq": [
          {"field": "Type1/id"},
          {"value": "1"}
        ]
      }
    }
  }
}

You can also use the operator not in a more complex condition. For example:

{
  "select": {
    "from": ["Type1"],
    "where": {
      "not": {
        "or": [
          {
            "eq": [
              {"field": "Type1/id"},
              {"value": "2"}
            ]
          },
          {
            "eq": [
              {"field": "Type1/id"},
              {"value": "4"}
            ]
          }
        ]
      }
    }
  }
}

You can use the operator not together with the operator in. The queried result does not include instances where the specified field has a null value.

For example:

{
  "select": {
    "from": ["Type1"],
    "where": {
      "not": {
        "in": [
            {"field": "Type1/id"},
            {"value": ["1","3","4","5"]}        
            ]
      }
    }
  }
}

This will return all instances of Type1 where "id" does not equal any of the values listed, that is, 1, 3, 4 and 5.

Full text (field scope)

The query language includes support for full text search.

For example, to perform a full text search on the field "Type1/id" with the value "1", use the query:

{
  "select": {
    "from": ["Type1"],
    "where": {
      "full_text": [
        {"field": "Type1/id"},
        {"value": "1"}
      ]
    }
  }
}

The full text query also works on subelements. For example, if you have the following type structure:

Type1
  * id
  * element
    * value1
    * value2

Then the query:

{
  "select": {
    "from": ["Type1"],
    "where": {
      "full_text": [
        {"field": "Type1/element"},
        {"value": "1"}
      ]
    }
  }
}

Is equivalent to:

{
  "select": {
    "from": ["Type1"],
    "where": {
      "or": [
        {
          "full_text": [
            {"field": "Type1/element/value1"},
            {"value": "1"}
          ]
        },
        {
          "full_text": [
            {"field": "Type1/element/value2"},
            {"value": "1"}
          ]
        }
      ]
    }
  }
}

Full text (entity scope)

If you do not want to specify any field but want to look for a value in the entity, the following query gives an example of how to achieve this:

{
  "select": {
    "from": ["Type1"],
    "where": {
      "full_text": [
        {"value": "1"}
      ]
    }
  }
}

If you omit the field, MDM performs full text search on all Type1 fields.

Metadata fields

MDM adds "metadata" fields to the records it manages. These fields are not present in the user data model but can be used in the query.

Supported metadata fields are:

  • taskId (also known as groupId)
  • timestamp (last modification time)
  • groupSize

For the staging storage, there are additional metadata fields:

  • Staging error
  • Staging source
  • Staging status
  • Staging block key
  • Staging has task

    Note that the field "Staging has task" indicates whether a Talend Data Stewardship task is linked with the record.

If the query contains staging-only fields, you may expect an error that indicates the incompatible expressions. The incompatible expressions are checked in selected fields, conditions, order by and join expressions.

Get metadata field values

To get a metadata field, use "metadata" instead of "field" in the selected fields as shown in the following example:

{
  "select": {
    "from": ["Type1"],
    "fields": [
      {"metadata": "timestamp"},
      {"metadata": "task_id"},
      {"metadata": "group_size"},
      {"metadata": "staging_error"},
      {"metadata": "staging_source"},
      {"metadata": "staging_status"},
      {"metadata": "staging_blockkey"},
      {"metadata": "staging_hastask"}
    ]
  }
}

You can also mix fields from the entity with fields from the record:

{
  "select": {
    "from": ["Type1"],
    "fields": [
      {"metadata": "timestamp"},
      {"metadata": "task_id"},
      {"field": "Type1/id"}
    ]
  }
}

You can also use the "distinct" keyword for metadata fields. For example, to return all distinct values of "taskId" wrapped in an element named "distinctTaskId", use the query:

{
  "select": {
    "from": ["Type1"],
    "fields": [
      {
        "alias": [
          {"name": "distinctTaskId"},
          {"distinct": {"metadata": "task_id"}}
        ]
      }
    ]
  }
}

Conditions involving metadata fields

You can involve metadata fields in conditions. As for the selected fields, just use "metadata" instead of "field". For example:

{
  "select": {
    "from": ["Type1"],
    "where": {
      "eq": [
        {"metadata": "task_id"},
        {"value": "1"}
      ]
    }
  }
}

Cache

MDM can cache query results. In this case, it keeps the query in cache and serves the cached result every time this query is executed.

For example, to cache a query result, use the query:

{
  "select": {
    "from": ["Type1"],
    "where": {
      "eq": [
        {"field": "Type1/id"},
        {"value": "1"}
      ]
    },
    "cache": true
  }
}

By default, "cache" is false, but this query is also valid:

{
  "select": {
    "from": ["Type1"],
    "where": {
      "eq": [
        {"field": "Type1/id"},
        {"value": "1"}
      ]
    },
    "cache": false
  }
}

A cached result can have at most 50 records. When the query yields too many results, the following log will appear in the MDM log at DEBUG level: Query is yielding more results than cache is allowed to keep, bypassing cache for query.

Join to other types

Joins are a way to retrieve data from another type following the foreign keys.

For example, you can retrieve the Address's street element by following the FK between Person and Address types.

A simple example of join would be:

{
  "select": {
    "from": [
      "Type1",
      "Type2",
    ],
    "fields": [
      {"field": "Type1/id"},
      {"field": "Type2/value1"}
    ],
    "joins": [
      {
        "from": "Type1/fk2",
        "on": "Type2/id"
      }
    ]
  }
}

This query returns all Type1's "id" element as well as the "value1" element located in Type2. The join is performed using the "fk2" in Type1.

You may chain joins in the query too. For example:

{
  "select": {
    "from": [
      "Type1",
      "Type2",
      "Type3"
    ],
    "fields": [
      {"field": "Type1/id"},
      {"field": "Type2/value1"},
      {"field": "Type3/value1"}
    ],
    "joins": [
      {
        "from": "Type1/fk2",
        "on": "Type2/id"
      },
      {
        "from": "Type2/fk3",
        "on": "Type3/id"
      }
    ]
  }
}

This query displays "id" from Type1, "value1" from Type2 and "value1" from Type3. The value for Type3's "value1" is obtained via a join between Type1, Type2 and Type3.

History related operators - “as of" operator

The "as_of" operator allows you to retrieve a record as it was at any date.

For example:

{
  "select": {
    "from": ["Type1"],
    "as_of": {
      "date": "1000"
    }
  }
}

The "as of" operator will not fail in case no history is available. Rather, it will return the record at its current state.

History browsing relies on the journal. If you inserted data straight into the SQL database or disabled event creation for tMDMOutput/tMDMDelete components, you should expect only partial history.

MDM can only build a history of the records based on information it stored about updates: if you disabled journal for some updates, it cannot be a complete and accurate history.

For example, to return all Type1 instances with the values they had one second after January 1st 1970 (1000 ms after EPOCH), use the query:

{
  "select": {
    "from": ["Type1"],
    "as_of": {
      "date": "1970-01-01T00:00:01"
    }
  }
}

History navigation shortcuts

The element "date" for the "as_of" element supports some handy shortcuts:

  • yesterday: date 24 hours ago.
  • creation: date of record creation. When you use "creation", you are not allowed to use the swing parameter. Otherwise, an expected error will be thrown.
  • now: current date.

For example, to return all instances of Type1 as they were yesterday (24 hours ago), use the query:

{
  "select": {
    "from": ["Type1"],
    "as_of": {
      "date": "yesterday"
    }
  }
}

Filter history (get by id)

You may want to filter the history records instead of getting all instances.

For example, to return the instance of Type1 with "id = 1" as it was on January 1st 1970, use the query:

{
  "select": {
    "from": ["Type1"],
    "where": {
      "eq": [
        {"field": "Type1/id"},
        {"value": "1"}
      ]
    },
    "as_of": {
      "date": "1000"
    }
  }
}

An improvement on this query would be to use all conditions with "as_of", which allows you to query on previous values. Note that conditions do not need to be only on id elements.

For example, to return the Type1 instance that contained "text" on January 1st, 1970, use the query:

{
  "select": {
    "from": ["Type1"],
    "where": {
      "contains": [
        {"field": "Type1/value"},
        {"value": "text"}
      ]
    },
    "as_of": {
      "date": "1000"
    }
  }
}

However, this has scalability issues since it might force MDM to build a complete set of Type1 instances, thus leading to memory issues since states are computed in memory.

Using Joins together with as_of

You can use "joins" in a query together with the "as_of" operator.

For example, using the query below returns three fields ("id", "value1", "value2") coming from two types ("Type1", "Type2"):

{
  "select": {
    "from": [
      "Type1",
      "Type2"
    ],
    "fields": [
      {"field": "Type1/id"},
      {"field": "Type1/value1"},
      {"field": "Type2/value2"}
    ],
    "joins": [
      {
        "from": "Type1/fk2",
        "on": "Type2/id"
      }
    ],
    "as_of": {
      "date": "1000"
    }
  }
}

The returned values have the features below:

  • The "as_of" returns the values of "Type1" as they were on 01/01/1970.
  • The FK value used for Type2 is the FK value as of 01/01/1970.
  • The values for Type2 are those on 01/01/1970.

Security and user roles

Bear in mind that the query passed to the service is not always the query MDM will execute. Before executing the query, MDM will prune all elements the logged user is not allowed to access. This includes selected fields, conditions, and order by. All elements to hide from users are removed from the query without reporting any errors.

For example,

{
  "select": {
    "from": ["Type1"],
    "fields": [
      {"field": "Type1/id"},
	  {"field": "Type1/secret_field"}
    ]
  }
}

In this example, MDM will automatically remove "secret_field" from the query if the current user (that is, the user used to log on the REST API) is not allowed to access "secret_field". Therefore, the actual query executed will be:

{
  "select": {
    "from": ["Type1"],
    "fields": [
      {"field": "Type1/id"}
    ]
  }
}

Another example of such queries is:

{
  "select": {
    "from": ["Type1"],
    "where": {
      "gt": [
        {"field": "Type1/secret_field"},
        {"value": "1"}
      ]
    }
  }
}

This query will be read as "return all instances of Type1 where Type1's secret_field is greater than 1", and the returned result is expected to be Type1 instances where secret_field > 1.

However, if the current user's roles do not have access right to "secret_field", the actual query becomes:

{
  "select": {
    "from": ["Type1"]
  }
}

This query will be read as "return all instances of Type1", and the returned result now is expected to be all Type1 instances since the current user is not allowed to access "secret_field".

This prevents unprivileged users to indirectly guess what are the values in "secret_field" using value conditions on this field.

An example of how to execute an HTTP GET operation with MDM REST API from Talend MDM Web UI

From Talend MDM Web UI, authorized users can use the Tools menu to access the MDM REST API documentation page based on Swagger, and test REST API operations on this page if needed.

The following introduces an example of executing an HTTP GET operation with MDM REST API to list all primary keys of the master data records in a container on the MDM server.

Before you begin

Make sure that you have already imported the MDM Demo project and created several master data records for the Product entity of the Product container.

Procedure

  1. Open the REST API documentation page from Talend MDM Web UI.
  2. From the list of operations under the User data management category, click the GET link which is followed by /data/{containerName}/{type} to displays its details. You can also click Lists all primary keys for container and type to display the operation details.
  3. Select Master from the Container list.
  4. Enter Product in the containerName field.
  5. Enter Product in the type field.
  6. Click Execute to execute the GET operation.

    The HTTP request URL, response body, response code and response headers are displayed accordingly. In this example, all primary keys of the Product container are listed.

  7. To verify the execution result of this GET operation, go to the Master Data Browser page, then search the master data records of the Product container, and validate that the execution result matches the search result.

Transactions API integration

Transactions API can be integrated with data API.

For more information about the MDM transactions, see MDM Transactions.

Attaching a REST request to an existing transaction

Modifications made through the REST data API can be attached to a transaction created through the transactions API.

To attach a REST request to an existing transaction, you just need to set the custom HTTP header "transaction-id" to the id of the transaction. If the provided transaction id does not exist, it will be created.

For example:
PUT /talendmdm/services/rest/transactions, creates a new transaction and returns its id
POST /talendmdm/services/rest/data/Product (with transaction-id set to the id returned at step1), creates a new product but does not commit changes to the database
POST /talendmdm/services/rest/transactions/<id> commits the transactions and thus changes made at step 2

This feature also works with batch insert operation (/talendmdm/services/rest/data/Product/batch).