Executing Big Data Streaming Jobs from Big Data Streaming Conductor

Talend Administration Center User Guide

EnrichVersion
6.4
EnrichProdName
Talend Data Fabric
Talend Big Data Platform
Talend Real-Time Big Data Platform
Talend Big Data
Talend Data Integration
Talend Data Services Platform
Talend Data Management Platform
Talend ESB
task
Administration and Monitoring
Deployment
EnrichPlatform
Talend Administration Center

This module is used almost the same way as the Job Conductor module. Therefore, you are recommended to refer to Executing data integration Jobs from Job Conductor to discover how to use this Big Data Streaming Conductor module while the current section outlines only the distinct capability differences of Big Data Streaming Conductor from the capabilities of Job Conductor:

  1. This Conductor is used to manage the Spark Streaming Jobs and the Storm Jobs only.

  2. The Triggers feature is not available to Big Data Streaming Conductor, because once a streaming Job has launched, it continuously runs and processes each incoming message as long as it is published.

  3. When this Conductor is used to manage a Storm Job, the Monitor button in the Big Data Streaming page directs you to the Storm UI web console provided by the Storm system itself.

  4. Since a Storm topology cannot be paused or resumed, you cannot pause or resume a Storm task from the Big Data Streaming page, either.

  5. When executing a Storm task from this Big Data Streaming Conductor, the Conductor kills and removes any topology that has the same name as this topology you want to execute in the Storm cluster, and then submits this new topology to the cluster. This feature overrides the way you have defined to run this topology in the Storm configuration tab of the corresponding Storm Job in the Studio.

  6. Killing a Storm task from the Big Data Streaming page using the Kill button kills and removes the Storm topology running in the Storm cluster. This feature overrides the way you have defined to kill this topology in the Storm configuration tab of the corresponding Storm Job in the Studio.

    For further information about how the Storm configuration tab is used, see the scenario described in the tKafkaInput section in Talend Components Reference Guide.

  7. The JVM parameters feature is not available to the Big Data Streaming Conductor module.