These properties are used to configure tFilterRow running in the Spark Streaming Job framework.
The Spark Streaming tFilterRow component belongs to the Processing family.
This component is available in Talend Real-Time Big Data Platform and Talend Data Fabric.
Schema and Edit Schema
A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. When you create a Spark Job, avoid the reserved word line when naming the fields.
Logical operator used to combine conditions
Select a logical operator to combine simple conditions and to combine the filter results of both modes if any advanced conditions are defined.
And: returns the boolean value of true if all conditions are true; otherwise false. For each two conditions combined using a logical AND, the second condition is evaluated only if the first condition is evaluated to be true.
Or: returns the boolean value of true if any condition is true; otherwise false. For each two conditions combined using a logical OR, the second condition is evaluated only if the first condition is evaluated to be false.
Click the plus button to add as many simple conditions as needed. Based on the logical operator selected, the conditions are evaluated one after the other in sequential order for each row. When evaluated, each condition returns the boolean value of true or false.
Input column: Select the column of the schema the function is to be operated on
Function: Select the function on the list
Operator: Select the operator to bind the input column with the value
Value: Type in the filtered value, between quotes if needed.
Use advanced mode
Select this check box when the operations you want to perform cannot be carried out through the standard functions offered, for example, different logical operations in the same component. In the text field, type in the regular expression as required.
If multiple advanced conditions are defined, use a logical operator between two conditions:
This component is used as an intermediate step.
This component, along with the Spark Streaming component Palette it belongs to, appears only when you are creating a Spark Streaming Job.
Note that in this documentation, unless otherwise explicitly stated, a scenario presents only Standard Jobs, that is to say traditional Talend data integration Jobs.
In the Spark Configuration tab in the Run view, define the connection to a given Spark cluster for the whole Job. In addition, since the Job expects its dependent jar files for execution, you must specify the directory in the file system to which these jar files are transferred so that Spark can access these files:
This connection is effective on a per-Job basis.