Managing files using MongoDB GridFS - 6.3

Talend Components Reference Guide

EnrichVersion
6.3
EnrichProdName
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for Big Data
Talend Open Studio for Data Integration
Talend Open Studio for Data Quality
Talend Open Studio for ESB
Talend Open Studio for MDM
Talend Real-Time Big Data Platform
task
Data Governance
Data Quality and Preparation
Design and Development
EnrichPlatform
Talend Studio

In this scenario, the MongoDB GridFS components are used to create a Job to manage video files in MongoDB GridFS.

For further information about the GridFS system of MongoDB, see When to use GridFS.

For demonstration purposes, only one video file, called custom_hadoop.mp4, is used; you can use any of your own video files to replicate this scenario.

Linking the components

  1. In the Integration perspective of the Studio, create an empty Job, named FS_video for example, from the Job Designs node in the Repository tree view.

    For further information about how to create a Job, see Talend Studio User Guide.

  2. In the workspace, enter the name of the component to be used and select this component from the list that appears. In this scenario, the components are tMongoDBConnection, tMongoDBGridFSPut, tMongoDBGridFSList, tMongoDBGridFSProperties, tFilterColumns, tLogRow, tMongoDBGridFSGet and tMongoDBGridFSDelete.

  3. Connect tMongoDBConnection to tMongoDBGridFSPut using the Trigger > On Subjob Ok link.

  4. Repeat this operation to connect tMongoDBGridFSPut to tMongoDBGridFSList, tMongoDBGridFSList to tMongoDBGridFSGet, and then tMongoDBGridFSGet to tMongoDBGridFSDelete.

  5. Connect tMongoDBGridFSList to tMongoDBGridFSProperties using the Row > Iterate link. This link allows tMongoDBGridFSList to send data to tMongoDBGridFSProperties iteratively.

  6. Connect tMongoDBGridFSProperties to tFilterColumns using the Row > Main link.

  7. Do the same to connect tFilterColumns to tLogRow.

Connecting to MongoDB

  1. Double-click tMongoDBConnection to open its Component view.

  2. From the DB version list, select the MongoDB version you are using.

  3. In the Server and Port fields, enter the authentication information required for the connection to MongoDB.

    If you use the host name of the MongoDB server, ensure that you have added the mapping between this host name and its IP address in the hosts file of the operating system in which the current Job is executed.

  4. In the Database field, enter the name of the database hosting GridFS. This database is created on the fly if it dose not exist.

Copying data to MongoDB GridFS

  1. Double-click tMongoDBGridFSPut to open its Component view.

  2. Select the Use existing connection check box and from the Connection list, select the component in which the MongoDB connection to be used is defined.

  3. In the Bucket field, enter the bucket to be used to store files in GridFS. In this example, it is talend_channel/61.

  4. In the Local folder field, enter the path, or browse to the folder where the files to be uploaded to GridFS are stored. As explained previously, it is a video file called custom_hadoop.mp4.

  5. In the Files table, add one row by clicking the [+] button and in the Filemask column, enter *.mp4 within the double quotation marks. This allows tMongoDBGridFSPut to copy all the files with the .mp4 extension from the local folder you have specified to the bucket to be used in GridFS.

  6. Leave the New name column empty, that is to say, leave the double quotation marks in this column as is, so as to keep the name of this video unchanged after being copied to GridFS.

Listing files stored in MongoDB GridFS

Iterating on the files

  1. Double-click tMongoDBGridFSList to open its Component view.

  2. Select the Use existing connection check box and from the Connection list, select the component in which the MongoDB connection to be used is defined.

  3. In the Bucket field, enter the bucket in which the files to be listed are stored. In this example, it is talend_channel/61.

  4. In the Query field, enter the query to select the files you want tMongoDBGridFSList to iterate on to generate different file lists. In this example, leave the default one in order to iterate on all of the files stored in this talend_channel/61 bucket.

    As explained previously, only one file, custom_hadoop.mp4, is expected.

Extracting file metadata

  1. Double-click tMongoDBGridFSProperties to open its Component view.

  2. Select the Use existing connection check box and from the Connection list, select the component in which the MongoDB connection to be used is defined.

  3. In the Bucket field, enter the bucket in which the files to be used are stored. In this example, it is talend_channel/61.

  4. From the Query type list, select the approach you want to use to select the files about which you need to extract the metadata. In this example, select Filename to use the filename attribute of each GridFS file for query.

  5. In the Filename field, press Ctrl + Space to display the variable list and choose the variable to be used. In this example, select tMongoDBGridFSList.CURRENT_FILENAME from the list. Then the expression to use the CURRENT_FILENAME variable is automatically added.

    This allows tMongoDBGridFSProperties to read each file name returned by tMongoDBGridFSList.

Filtering attributes

  1. Double-click tFilterColumns to open its Component view.

  2. Click the [...] button next to Edit schema to open the schema editor.

  3. On the left side (the input side), select the column to be used and click the button to move this column to the right side (the output side). In this example, move every column to the right side except the contentType column.

    Each column represents a file attribute and the pre-defined schema of tMongoDBGridFSProperties automatically contains these columns.

  4. Click OK to validate these changes and accept the propagation prompted by the pop-up dialog box.

Downloading files from MongoDB GridFS

  1. Double-click tMongoDBGridFSGet to open its Component view.

  2. Select the Use existing connection check box and from the Connection list, select the component in which the MongoDB connection to be used is defined.

  3. In the Bucket field, enter the bucket in which the files to be retrieved are stored. In this example, it is talend_channel/61.

  4. In the Local folder field, enter the path to the local folder in which you want to store the downloaded files. In this scenario, it is C:/tmp/output.

  5. Select the Use Document ID as output filename check box to rename each downloaded file using the value of its ObjectID attribute.

    Since a file in GridFS is distinct by ID rather than by name, it is possible that several files are using the same file name. For this reason, when downloading this kind of files into the same directory without renaming them differently, an exception is returned to alert that the file being downloaded already exists. In order to avoid this error, you can either select the Overwrite local files check box to replace the existing one with the latest downloaded file or rename these files on the fly using their IDs. In this example, the strategy of renaming these files is adopted.

Remove files from MongoDB GridFS

  1. Double-click tMongoDBGridFSDelete to open its Component view.

  2. Select the Use existing connection check box and from the Connection list, select the component in which the MongoDB connection to be used is defined.

  3. In the Bucket field, enter the bucket in which the files to be deleted are stored. In this example, it is talend_channel/61.

  4. From the Query type list, select the approach you want to use to select the files to be deleted. In this example, select Filename to use the filename attribute of each GridFS file for query.

  5. In the Filename field, enter the name of the file to be deleted.

Executing the Job

Then you can run this Job.

The tLogRow component is used to present the execution result of the Job.

  1. If you want to configure the presentation mode on its Component view, double-click the tLogRow component to open the Component view and in the Mode area, then, select the Table (print values in cells of a table) radio box.

  2. Press F6 to run this Job.

Once done, the Run view is opened automatically, where the metadata of the video custom_hadoop.mp4 in GridFS is displayed.

The downloaded file can be found in the directory C:/tmp/output, using its ID as its file name.