Design the data flow of the Job working with S3 and Databricks on AWS - 7.1

Amazon S3

author
Talend Documentation Team
EnrichVersion
7.1
EnrichProdName
Talend Big Data
Talend Big Data Platform
Talend Data Fabric
Talend Data Integration
Talend Data Management Platform
Talend Data Services Platform
Talend ESB
Talend MDM Platform
Talend Open Studio for Big Data
Talend Open Studio for Data Integration
Talend Open Studio for ESB
Talend Open Studio for MDM
Talend Real-Time Big Data Platform
task
Data Governance > Third-party systems > Amazon services (Integration) > Amazon S3 components
Data Quality and Preparation > Third-party systems > Amazon services (Integration) > Amazon S3 components
Design and Development > Third-party systems > Amazon services (Integration) > Amazon S3 components
EnrichPlatform
Talend Studio

Procedure

  1. In the Integration perspective of the Studio, create an empty Spark Batch Job from the Job Designs node in the Repository tree view.
  2. In the workspace, enter the name of the component to be used and select this component from the list that appears. In this scenario, the components are tS3Configuration, tFixedFlowInput, tFileOutputParquet, tFileInputParquet and tLogRow.
    The tFixedFlowInput component is used to load the sample data into the data flow. In the real-world practice, you could use the File input components, as well as the processing components, to design a sophisticated process to prepare your data to be processed.
  3. Connect tFixedFlowInput to tFileOutputParquet using the Row > Main link.
  4. Connect tFileInputParquet to tLogRow using the Row > Main link.
  5. Connect tFixedFlowInput to tFileInputParquet using the Trigger > OnSubjobOk link.
  6. Leave tS3Configuration alone without any connection.