Executing Big Data Streaming Jobs from Big Data Streaming Conductor

In the Big Data Streaming page of Talend Administration Center, an execution task gathers the script generation, deployment and execution phases of Big Data Streaming Jobs, that is to say, the Spark Streaming Jobs and the Storm Jobs. You can launch this task, from this single web-based application.

This module is used almost the same way as the Job Conductor module. Therefore, you are recommended to refer to Executing data integration Jobs from Job Conductor to discover how to use this Big Data Streaming Conductor module while the current section outlines only the distinct capability differences of Big Data Streaming Conductor from the capabilities of Job Conductor:

• This Conductor is used to manage the Spark Streaming Jobs and the Storm Jobs only.
• The Triggers feature is not available to Big Data Streaming Conductor, because once a streaming Job has launched, it continuously runs and processes each incoming message as long as it is published.
• When this Conductor is used to manage a Storm Job, the Monitor button in the Big Data Streaming page directs you to the Storm UI web console provided by the Storm system itself.
• Since a Storm topology cannot be paused or resumed, you cannot pause or resume a Storm task from the Big Data Streaming page, either.
• When executing a Storm task from this Big Data Streaming Conductor, the Conductor kills and removes any topology that has the same name as this topology you want to execute in the Storm cluster, and then submits this new topology to the cluster. This feature overrides the way you have defined to run this topology in the Storm configuration tab of the corresponding Storm Job in the Studio.
• Killing a Storm task from the Big Data Streaming page using the Kill button kills and removes the Storm topology running in the Storm cluster. This feature overrides the way you have defined to kill this topology in the Storm configuration tab of the corresponding Storm Job in the Studio.
• The JVM parameters feature is not available to the Big Data Streaming Conductor module.