Reading messages from a given Kafka topic

Procedure

Double-click tKafkaInput to open its Component view.
In the Broker list field, enter the locations of the brokers of the Kafka cluster to be used, separating these locations using comma (,). In this example, only one broker exists and its location is localhost:9092.
From the Starting offset drop-down list, select the starting point from which the messages of a topic are consumed. In this scenario, select From latest, meaning to start from the latest message that has been consumed by the same consumer group and of which the offset has been committed.
In the Topic name field, enter the name of the topic from which this Job consumes Twitter streams. In this scenario, the topic is twitter_live.
This topic must exist in your Kafka system. For further information about how to create a Kafka topic, see the documentation from Apache Kafka or use the tKafkaCreateTopic component. However, note that tKafkaCreateTopic is not available to the Spark Jobs.
Select the Set number of records per second to read from each Kafka partition check box. This limits the size of each micro batch to be sent for processing.

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!