This scenario aims at helping you set up and use connectors in a pipeline. You are
advised to adapt it to your environment and use case.
Procedure
-
Click .
-
In the panel that opens, select the type of connection you
want to create.
Example
REST HTTP BIN
-
Select your engine
in the Engine list.
Note:
- It is recommended to use the Remote Engine Gen2 rather than
the Cloud Engine for Design for advanced
processing of data.
- If no Remote Engine Gen2 has been created from Talend Management Console or if it exists but appears as unavailable
which means it is not up and running, you will not be able to select
a Connection type in the list nor to
save the new connection.
- The list of available connection types depends on the engine you
have selected.
-
Select the type of connection you want to create.
Here, select REST.
-
Fill in the connection properties and URL address of the REST
server to be invoked as described in REST properties, check the connection and click ADD DATASET. In this example, the https://httpbin.org API service returning
random universally unique identifiers (UUIDs) is invoked.
-
In the Add a new dataset
panel, name your dataset. The REST connector returns a JSON with a parsed body
entry.
Example
UUID
-
Fill in the required properties to invoke the service and click View
sample to see a preview of your dataset sample:
-
In the Type field, select
Streaming.
-
In the Resource field, type in
uuid to retrieve the uuid records from the https://httpbin.org API
service.
-
In the HTTP method field, select
GET.
-
In the Answer body format field, select
JSON.
-
Click Validate to save your dataset.
-
Do the same to add the Test dataset that will be used as a
destination in your pipeline to receive logs. Fill in the connection properties
as described in Test connection properties.
-
Click Add
pipeline on the Pipelines page. Your new pipeline opens.
-
Give the pipeline a meaningful name.
Example
Invoke REST API to get streaming
UUIDs
-
Click ADD SOURCE and
select your source dataset, UUID in the
panel that opens. The pipeline turns into a streaming pipeline as you have
selected the Streaming dataset
type.
-
In the Configuration
tab of this source dataset, set the polling interval to stream the API service
in the Delay field. By default, it is
5000ms, that is to say data will be retrieved every 5 seconds.
-
Click the ADD
DESTINATION item on the pipeline to open the panel allowing to
select the output dataset in which your output logs will be stored.
-
In the Configuration
tab of the destination, enable the Log records to
STDOUT option to capture the logs into this output
destination.
-
Click Save to
save your configuration.
-
On the top toolbar of Talend Cloud Pipeline Designer,
click the Run button to open the panel allowing you to select
your run profile.
-
Select your run profile in the list (for more information, see Run profiles), then click Run to
run your pipeline.
Results
Your streaming pipeline is being executed, the data that was invoked from the REST API
service is retrieved and the output logs keep receiving more UUIDs as you refresh them.
You can see that UUIDs are retrieved every 5 seconds as previously defined.
