This scenario aims at helping you set up and use connectors in a pipeline. You are
advised to adapt it to your environment and use case.
Procedure
-
Click .
-
Browse to the local_file-to-azure_event_hubs.csv file and select it.
-
Rename it, for example Baltimore restaurants.
-
Click .
-
In the panel that opens, select the type of connection you
want to create.
Example
Azure
EventHubs
-
Select your engine
in the Engine list.
Note:
- It is recommended to use the Remote Engine Gen2 rather than
the Cloud Engine for Design for advanced
processing of data.
- If no Remote Engine Gen2 has been created from Talend Management Console or if it exists but appears as unavailable
which means it is not up and running, you will not be able to select
a Connection type in the list nor to
save the new connection.
- The list of available connection types depends on the engine you
have selected.
-
Select the type of connection you want to create.
Here, select Azure Event Hubs.
-
Fill in the connection properties to safely access your Azure service resources
(endpoint, shared access signature keys) as described in Azure Event Hubs properties, check the connection and click
Add dataset.
-
In the Add a new dataset
panel, name your dataset. In this example, the
baltimore-restaurants
Event Hub that is currently empty will be used to publish
the data about Baltimore restaurants.
Example
-
Name your dataset, Restaurant Event Hub for
example.
-
Click Validate to save your dataset.
-
Click Add
pipeline on the Pipelines page. Your new pipeline opens.
-
Give the pipeline a meaningful name.
Example
From local file to Azure Event
Hubs - publish an event
-
Click ADD SOURCE and select your source dataset,
Baltimore restaurants in the panel that opens.
-
Click
and
add a Strings processor to the pipeline
in order to change the case of some records. The configuration panel
opens.
-
Give a meaningful name to the processor.
Example
change case of police district records
-
Configure the processor:
-
Select Change to title
case in the Function
name list as you want to change the case of the records
from upper case to title case.
-
Select .police_disctrict in the Fields to process list as you want to apply this change
to the values of these specific records.
-
Click Save to
save your configuration.
-
(Optional) Look at the preview of the processor to see the
data after the case change.
-
Click the ADD
DESTINATION item on the pipeline to open the panel allowing to
select the Azure Event Hub in which your output data will be loaded,
Restaurant Event Hub.
-
In the Configuration
tab of the destination, the Round-Robin model is the
default Partition Type used when publishing an event but
feel free to specify a partition key or a partition ID according to your
case.
-
On the top toolbar of Talend Cloud Pipeline Designer,
click the Run button to open the panel allowing you to select
your run profile.
-
Select your run profile in the list (for more information, see Run profiles), then click Run to
run your pipeline.
Results
Your pipeline is being executed, the Baltimore restaurant data from your
local file has been processed and the output flow is sent to the
baltimore-restaurants Azure Event Hub you have
defined:
What to do next
Once the event is published, you can consume the messages in another pipeline and use
them as an Azure Event Hub source:
