Setting up JSON metadata for an output file

Talend ESB Studio User Guide

EnrichVersion
6.5
EnrichProdName
Talend ESB
task
Design and Development
EnrichPlatform
Talend Studio

This section describes how to define JSON metadata for an output file. To define JSON metadata for an input file, see Setting up JSON metadata for an input file.

Defining general properties

  1. In the wizard, fill in the general information in the relevant fields to identify the JSON file metadata, including Name, Purpose and Description.

    The Name field is required, and the information you provide in the Description field will appear as a tooltip when you move your mouse pointer over the file connection.

    Note

    In this step, it is advisable to enter information that will help you distinguish between your input and output connections, which will be defined in the next step.

  2. If needed, set the version and status in the Version and Status fields respectively.

    You can also manage the version and status of a repository item in the [Project Settings] dialog box. For more information, see Version management and Status management respectively.

  3. If needed, click the Select button next to the Path field to select a folder under the File Json node to hold your newly created file connection.

  4. Click Next to set the type of metadata.

Setting the type of metadata and loading the template JSON file

In this step, the type of schema is set as either input or output. For this procedure, the schema of interest is output.

  1. From the dialog box, select Output JSON click Next to proceed to the next step of the wizard.

  2. Choose whether to create the output metadata manually or from an existing JSON file as a template.

    If you choose the Create manually option you will have to configure the schema and link the source and target columns yourself. The output JSON file/field is created via a Job using a JSON output component such as tWriteJSONField.

    In this example, we will create the output metadata by loading an existing JSON file. Therefore, select the Create from a file option.

  3. Click the Browse... button next to the JSON File field, browse to the access path to the JSON file the structure of which is to be applied to the output JSON file/field, and double-click the file. Alternatively, enter the full path to the file or the URL which links to the template JSON file.

    The File Viewer area displays a preview of the JSON structure, and the File Content area displays a maximum of the first 50 rows of the file.

  4. Enter the Encoding type in the corresponding field if the system does not detect it automatically.

  5. In the Limit field, enter the number of levels in the JSON hierarchical depth to which you want to limit the JsonPath or XPath query, 0 for no limits.

    Setting this parameter to a value less than 5 can help prevent the wizard from hanging in case of a large JSON file.

  6. Optionally, specify an output file path.

  7. Click Next to define the schema.

Defining the schema

Upon completion of the previous operations, the columns in the Linker Source area are automatically mapped to the corresponding ones in the Linker Target area, as indicated by blue arrow links..

In this step, you need to define the output schema. The following table describes how:

To...Perform the following...
Define a loop element

In the Linker Target area, right-click the element of interest and select Set As Loop Element from the contextual menu.

Note

It is a mandatory operation to define an element to run a loop on.

Define a group element

In the Linker Target area, right-click the element of interest and select Set As Group Element from the contextual menu.

Note

You can set a parent element of the loop element as a group element on the condition that the parent element is not the root of the JSON tree.

Create a child element for an element

In the Linker Target area,

  • Right-click the element of interest and select Add Sub-element from the contextual menu, enter a name for the sub-element in the dialog box that appears, and click OK.

  • Select the element of interest, click the [+] button at the bottom, select Create as sub-element in the dialog box that appears, and click OK. Then, enter a name for the sub-element in the next dialog box and click OK.

Create an attribute for an element

In the Linker Target area,

  • Right-click the element of interest and select Add Attribute from the contextual menu, enter a name for the attribute in the dialog box that appears, and click OK.

  • Select the element of interest, click the [+] button at the bottom, select Create as attribute in the dialog box that appears, and click OK. Then, enter a name for the attribute in the next dialog box and click OK.

Create a name space for an element

In the Linker Target area,

  • Right-click the element of interest and select Add Name Space from the contextual menu, enter a name for the name space in the dialog box that appears, and click OK.

  • Select the element of interest, click the [+] button at the bottom, select Create as name space in the dialog box that appears, and click OK. Then, enter a name for the name space in the next dialog box and click OK.

Delete one or more elements/attributes/name spaces

In the Linker Target area,

  • Right-click the element(s)/attribute(s)/name space(s) of interest and select Delete from the contextual menu.

  • Select the element(s)/attribute(s)/name space(s) of interest and click the [x] button at the bottom.

  • Select the element(s)/attribute(s)/name space(s) of interest and press the Delete key.

    Note

    Deleting an element will also delete its children, if any.

Adjust the order of one or more elements

In the Linker Target area, select the element(s) of interest and click the and buttons.

Set a static value for an element/attribute/name space

In the Linker Target area, right-click the element/attribute/name space of interest and select Set A Fix Value from the contextual menu.

Note

  • The value you set will replace any value retrieved for the corresponding column from the incoming data flow in your Job.

  • You can set a static value for a child element of the loop element only, on the condition that the element does not have its own children and does not have a source-target mapping on it.

Create a source-target mapping

Select the column of interest in the Linker Source area, drop it onto the node of interest in the Linker Target area, and select Create as sub-element of target node, Create as attribute of target node, or Add linker to target node according to your need in the dialog box that appears, and click OK.

If you choose an option that is not permitted for the target node, you will see a warning message and your operation will fail.

Remove a source-target mappingIn the Linker Target area, right-click the node of interest and select Disconnect Linker from the contextual menu.
Create a JSON tree from another JSON fileRight-click any schema item in the Linker Target area and select Import JSON Tree from the contextual menu to load another JSON file. Then, you need to create source-target mappings manually and define the output schema all again.

Note

You can select and drop several fields at a time, using the Ctrl + Shift technique to make multiple selections, therefore making mapping faster. You can also make multiple selections for right-click operations.

  1. In the Linker Target area, right-click the element you want to set as the loop element and select Set As Loop Element from the contextual menu.

    In this example, define a loop to run on the details element.

  2. Customize the mappings if needed.

  3. Click Next to finalize the schema.

Finalizing the end schema

The last step of the wizard shows the end schema generated and allows you to customize the schema according to your needs.

  1. If needed, rename the schema (by default, metadata) and leave a comment.

    Customize the schema if needed: add, remove or move schema columns, export the schema to an XML file, or replace the schema by importing an schema definition XML file using the tool bar.

    Make sure the data type in the Type column is correctly defined.

    For more information regarding Java data types, including date pattern, see Java API Specification.

    Below are the commonly used Talend data types:

    • Object: a generic Talend data type that allows processing data without regard to its content, for example, a data file not otherwise supported can be processed with a tFileInputRaw component by specifying that it has a data type of Object.

    • List: a space-separated list of primitive type elements in an XML Schema definition, defined using the xsd:list element.

    • Dynamic: a data type that can be set for a single column at the end of a schema to allow processing fields as VARCHAR(100) columns named either as 'Column<X>' or, if the input includes a header, from the column names appearing in the header. For more information, see Dynamic schema.

    • Document: a data type that allows processing an entire XML document without regarding to its content.

  2. If the JSON file which the schema is based on has been changed, click the Guess button to generate the schema again. Note that if you have customized the schema, the Guess feature does not retain these changes.

  3. Click Finish. The new file connection, along with its schema, is displayed under the relevant File Json metadata node in the Repository tree view.

Now you can drag and drop the file connection or the schema of it from the Repository tree view onto the design workspace as a new tWriteJSONField component or onto an existing component to reuse the metadata. For further information about how to use the centralized metadata in a Job, see How to use centralized metadata in a Joband How to set a repository schema in a Job.

To modify an existing file connection, right-click it from the Repository tree view, and select Edit JSON to open the file metadata setup wizard.

To add a new schema to an existing file connection, right-click the connection from the Repository tree view and select Retrieve Schema from the contextual menu.

Warning

If you are working on an SVN or Git managed project while the Manual lock option is selected in Talend Administration Center, be sure to lock manually your connection in the Repository tree view before retrieving or updating table schemas for it. Otherwise the connection is read-only and the Finish button of the wizard is not operable. For more information about locking and unlocking a project item and on different lock types, see Working collaboratively on project items.

To edit an existing file schema, right-click the schema from the Repository tree view and select Edit Schema from the contextual menu.