TPS-4149 (cumulative patch) - 7.3

author
Talend Documentation Team
EnrichVersion
7.3
EnrichProdName
Talend Data Fabric
EnrichPlatform
Talend Studio

TPS-4149 (cumulative patch)

Info Value
Patch name Patch_20200307_TPS-4149_v1-7.3.1
Release date 2020-07-03
Target version 20200219_1130-7.3.1
Product affected Talend Studio

Introduction

This monthly release includes all previous generally available patches for Talend Studio 7.3.1. For more information about the new features and bug fixes included, see Talend Data Fabric Release Notes.

NOTE: For information on how to obtain this patch, reach out to your Support contact at Talend.

New features

Fixed issues

This patch contains the following fixes:

  • TBD-10758 Compilation error on spark 2.1 for class "org.apache.spark.ui.jobs.JobProgressListener"
  • TBD-10863 HDP 2.x cannot be found in dynamic distributions list

Prerequisites

Consider the following requirements for your system:

  • Talend Studio 7.3.1 must be installed.

Installation

Installing the patch using Software update

  1. Log in to Talend Administration Center and go to Settings->Configuration->Software Update. Enter the correct values and save them referring to the documentation: Configuring the Software Update repository in Talend Administration Center.
  2. Download the new patch from the Settings->Software Update page into the nexus repository.
  3. Log in to Talend Studio with remote mode.
  4. Click the Update button displayed on the login window to install the patch.

Installing the patch manually using Talend Studio (for Cloud users)

  1. Create a folder named "patches" in your Studio installer directory and copy the patch .zip file to this folder.
  2. Restart your Studio.
  3. Click OK when prompted to install the patch, or restart the commandline to install the patch automatically.

Installing the patch using CommandLine

Execute the following commands to install the patch:

  1. Talend-Studio-win-x86_64.exe -nosplash -application org.talend.commandline.CommandLine -consoleLog -data commandline-workspace startServer -p 8002 --talendDebug
  2. initRemote {tac_url} -ul {TAC login username} -up {TAC login password}
  3. checkAndUpdate -tu {TAC login username} -tup {TAC login password}

After installing the patch, you need to stop CommandLine and clean the org.eclipse.osgi folder under the {Talend-Studio}/configuration directory, where {Talend-Studio} is the installation directory of your Talend Studio.

Installing the patch using Continuous Integration

To install the patch using the CI builder, use the -Dpatch.path option at build time. See Building and Deploying for details.

Note: It is strongly recommended to resynchronize poms after installing the patch by clicking the Force full re-synchronize poms button in the Build -> Maven view in the Project Settings dialog box in Talend Studio.

Changes from previous monthly releases

New features

This patch contains the following features:

  • TBD-9999 Support CDP Milestone I: Dynamic Distro for CDP Data Center 7.x
  • TBD-9737 Support HDInsight 4.0 Static Distro - DI / Spark / Spark streaming
  • TBD-10362 Support Databricks instance pools in Spark jobs
  • TUP-26669 Adapt HDInsight wizard for HDInsight 4.0
  • TUP-26699 Enhance context propagation over reference project - Resolve link conflict with Git
  • TUP-27259 Change default driver to Microsoft official for MSSql in Metadata
  • TUP-27157 Searchable Stitch connectors
  • TDI-43962 Talend Data Preparation Support for Dynamic Schema
  • TDI-44214 Change default driver to Microsoft official for MSSQL connector
  • TDI-43717 Tck component CosmosDBSQLAPI for Studio
  • TDI-44216 AWS Redshift Partner Tracking
  • TDI-44171 Observability: Add "Component" label to component-related events
  • TDI-44172 Observability: Create new "component execution" event and emit it frequently
  • TDI-44061 Snowflake - External Oauth2 support
  • TDI-44181 Support new authentication for ADLSgen2
  • TDQ-18462 MSSQL Database: Microsoft is now the default driver
  • TBD-10198 Add database/table parameters on DeltaLake components
  • TDM-7679 Flattener: generate the flattening map (via New Map wizard)
  • TDM-7095 Get the index of the parent Loop
  • TDI-43218 Azure storage components provide connectivity through SAS Token or Account key. This feature request is to provide connectivity though AD authentication.
  • TBD-9991 Support for EMR 5.29 Static Distro - DI / Spark batch / Spark streaming
  • TBD-10328 Merge operation for Delta Lake in tDeltaLakeOutput
  • TBD-10169 Upgrade Delta Lake connectivity
  • TBD-10265 Upgrade Snowflake connectivity
  • TBD-9643 Support Oracle 19c
  • TBD-10206 Upgrade httpclient to align with other components
  • TUP-16546 Prompt users for new password in Studio Connection when AD/LDAP credentials change in TAC
  • TUP-25708 Avoid GIT Branch conflicts due to Timestamp and ID
  • TUP-26186 Use the operators =, <=, <, >=, >, <> instead of EQ, LE, LT, GE, GT, NE in tELTTeradataMap
  • TUP-26284 Upgrade AWS SDK for driver in metadata
  • TUP-26956 Update Studio top bar to reflect monthly deliveries for better understanding
  • TUP-26288 Enhance context propagation over reference project
  • TDI-43878 Accessing DynamoDB GSI
  • TDI-43724 Extend AWS STS assume role field for Studio
  • TDI-43719 Rename tSQLDWHXXX component to tAzureSynapseXXX and support adls gen2
  • TDI-44179 Support new load method with COPY statement in tSynapseBulkExec
  • TESB-27806 Expose Talend Microservices Metrics in a monitoring system like Prometheus
  • TESB-24998 Interpret context variables used in provider and consumer endpoints in Data Services and Routes
  • TMC-19740 Delete "update corresponding task" check box from Studio publish to Cloud dialog
  • TDI-39575 Support AD authentication for Azure SQL Data Warehouse with tMSSQL connectors
  • TDQ-17784 DQ components (Spark Batch) support Databricks on Azure & AWS
  • TDQ-18049 tMatchModel: Feature importance report can be saved on Databricks (Azure/AWS) and HDInsight (Azure)

Fixed issues

This patch contains the following fixes:

  • TBD-10838 "URLClassLoader cannot be resolved to a type" in tHDFSConnection after Studio patch.
  • TBD-10799 HDI4 DI tHiveConnection issue
  • TBD-10798 Wrong repository url / password with proxy on dynamic distribution
  • TBD-10787 org.apache.spark.SparkException: Task failed while writing rows
  • TBD-10541 Upgrade jackson dependencies to 2.9.10.4 for MapR 6.0.1
  • TBD-10730 Revert TBD-10506 from maintenance/7.3 and master
  • TBD-10636 Need Test TUP-26669 Adapt HDInsight wizard for HDInsight 4.0
  • TBD-10695 Deprecate Altus
  • TBD-10711 Change default value for Hive Catalog
  • TBD-10048 When executing a job with tHive components, warning: set mapred.job.name=xxx... Encountered: . Expected: = CAUSED BY: Exception: Syntax error
  • TBD-10285 Sqoop job failing with exception Unknown dataset URI pattern
  • TBD-10654 [BUG] tFileInputDelimited error
  • TBD-10676 Compile error on simple Spark Streaming job
  • TBD-10680 [CDH6.1] java.lang.NoSuchMethodError on simple Spark Batch job
  • TBD-10213 [CDH61] java.lang.NoClassDefFoundError on job with OracleInput component
  • TBD-10429 additional double quotation is added to output file of tFileOutputDelimited
  • TBD-10436 Hive component fails with java.lang.NoClassDefFoundError: org/apache/log4j/Logger
  • TBD-10454 [7.3.1] DI job using tHDFSExist works from Studio to RE but not from TMC to RE
  • TBD-10581 Context for Spark Streaming Job isn't updating properly
  • TBD-10620 apache.spark.sql.catalyst.analysis.NoSuchDatabaseException for non-default database in HiveInput
  • TBD-10621 ClassNotFound/NoSuchMethod on Hive components on HDP 3.1
  • TBD-10657 Change in behavior of iterating fields using hive components
  • TBD-10700 "hive-exec-shade-2.1.1-cdh6.1.1-talend.jar" not found in talend-update site for dynamic distribution.
  • TBD-10712 [CDH 6.1.1]" java.lang.NoSuchFieldError: resources" error on both Spark Batch and Streaming jobs
  • TUP-27642 Issue when login project setup with Implicit Context Load + JDBC component
  • TUP-27074 Kafka Job builds in Studio, but fails in CI Builder
  • TUP-26970 [7.3.1] custom component published to nexus as .zip
  • TUP-26683 "Generate Doc as HTML" Inputs label in documentation not the same as FlowDiagram
  • TUP-26896 Error create a Sybase connection in metadata and cannot retrieve tables
  • TUP-26812 Old plugin folders deletion failed after installed new updatesite patch
  • TUP-27530 [7.3.1] After install Patch of TUP-27315, studio can't build job when first time login on
  • TUP-27315 Patch installation issues for zero install CI
  • TUP-27533 [7.3.1] Update ci-builder version to 7.3.2 in org.talend.designer.maven.tis
  • TUP-27078 Studio upload the libraries (jar files) into the Artifactory Server at each start
  • TUP-27053 Error: Could not find or load main class due to no JVM parameters configured in Studio
  • TUP-27072 Unexpected behaviour occurred when context name is in Japanese
  • TUP-25889 Display issue for the line between Linker source and target on Mac and Ubuntu
  • TUP-26476 tELTMap Group By in Where Clause
  • TUP-27343 Version is missing in Studio top bar after installing monthly patch by Nexus Update
  • TUP-27210 Version isn't correct in Help->"About Talend Studio" after installing monthly patch
  • TUP-27403 Wrong default values set when retrieve schema on Sybase
  • TUP-27394 Missing DB Type for sybase
  • TDI-44038 tSalesforceOutput file upload: byte[] or base64 encoded strings
  • TDI-44273 tSalesforceOutputBulkExec reject link is providing nulls for Id
  • TDI-44261 tSnowflakeOutput/tSnowflakeOutputBulkExec/tSnowflakeBulkExec Upsert invalid identifier
  • TDI-44247 "Illegal operation on INFORMATION_SCHEMA" error on tBigQueryInput
  • TDI-30505 Don't log 'Has inserted/update/delete x recorde(s)' when DBoutput checked 'Use an existing connection' and DBconnection doesn't check 'Auto commit'
  • TDI-44221 MongoDBBulkLoad--Exception in thread "Thread-2" java.lang.StringIndexOutOfBoundsException
  • TDI-44269 Outline ERROR_MESSAGE is null if "die on error" is not checked
  • TDI-44185 Authentication Fields missing in tMongoDBBulkLoad
  • TDI-44196 JDBC/MSSQL(Input) -> SqlServer(Output) Dynamic schema ERR:Colname [$] is converted to [_]
  • TDI-44253 tMongoDBOutput java.util.Date cannot be cast to java.lang.String
  • TDI-44128 Update axis2 dependency (CVE: axis2-kernel:1.5)
  • TDI-44281 tSetProxy in "local proxy" mode failing with NoClassDefFound
  • TDI-44292 tJDBCInput not converting date in expected format
  • TDI-43833 tDBBulkExec bug when using PostGress driver
  • TDI-44166 AzureADLSGen2Output: Blob Path starting with slash leads to unclear error
  • TDI-44104 CVE: xmlrpc-common:3.1.2
  • TDI-44105 CVE: xmlrpc-common:3.0
  • TDI-44115 CVE: xmlrpc-client:3.1.2
  • TDI-44169 CVE: xmlrpc-client:3.0
  • TESB-29280 tESBConsumer component fails with NAMESPACE_ERR: An attempt is made to create or change an object in a way which is incorrect with regard to namespaces
  • TESB-28929 cSOAP - Advanced Settings - Log Messages
  • TESB-29315 P2 depoy failed because of missing osgihelper-maven-plugin
  • TESB-29150 Context password for Microservice could not be used correctly when using Http Basic
  • TESB-29506 Failed to request REST Job service + SL as microservice
  • TDM-8089 Problem with camel headers when cJMS and cMap are used
  • TDM-8125 DatabaseLookup creating new DataSources for each message on the ESB
  • TDM-8084 Using thmap is getting an error when using a map with X12_5050_HIPPA structure
  • TDM-8085 Cannot rename a json structure during the creation process.
  • TDM-8088 tHMap : read/write to COBOL representation - "Decimal" with decimal places has wrong result
  • TDM-8094 Databaselookup fails on new runtime unless it is a top-level expression
  • TDM-8126 Infinite loop when generating route or job if map name contains DatabaseLookup
  • TDM-7985 Weird result when change CSV "skip header reading" option for special CSV
  • TDM-7986 UI issue on selection changed event in Map editor
  • TDM-8029 Need to regenerate projectProps when GIT pull
  • TDM-8076 Importing the map form 6.2 to 7.3 is not working
  • TDM-8092 XML Reader should honor encoding set in the XML Representation
  • TDM-8144 TDM tuj (command line in Linux) will return error on 731 June monthly patch
  • TDQ-18470 Data Quality components: updated all DQ library jars to the correct version
  • TDQ-18347 tMatchGroup: fixed the difference between the Chart results and the Job results
  • TDQ-18491 Importing DQ rules: fixed the item conflict
  • TDQ-18513 tMSSql(In)ValidRows: fixed the version of talend_DB_mssqlUtil jar
  • TBD-10616 Compiler error when spark streaming job use tMap that linked with tFixFlowInput,tHiveInput
  • TBD-10615 Compiler error when spark streaming job use tFileInputDelimited
  • TBD-10613 [BUG] tDeltaLakeOutput - SQL Merge option and time-travel corrections
  • TBD-10606 [CDH 6.1 Spark Batch] Hive job fails with "Unable to instantiate SparkSession with Hive support because Hive classes are not found" error
  • TBD-9956 Sqoop issue with parquet/avro format with HDP 2.6
  • TBD-10095 Sqoop issue with parquet/avro format with EMR 5.15(2.8.3)
  • TBD-10113 [BUG] Streaming, no output/NPE on tFileStreamInputJSON
  • TBD-10363 lost data after read by tFileInputDelimited component
  • TBD-10374 Cannot connect to hive against CDH 5.16.2
  • TBD-10401 [7.3.1]Getting [ERROR]: org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator in a few job post migration to Talend 7.3.1
  • TBD-10442 tFileOuputDelimited - wrong columns order
  • TBD-10443 CDH - Dynamic distribution - CDH 6.X not available
  • TBD-10449 NullPointerException when column is empty
  • TBD-10479 [BUG] Compile error with tHiveOutput
  • TBD-10572 Streaming - tExtractDelimitedFields - Compile error
  • TBD-9802 Compile error in spark job with HBase on HDP 3.1
  • TBD-10067 Incorrect Encoding in FileInputJSON
  • TBD-10153 Compiler error for sqoop components in DI job when using java 11
  • TBD-10159 (Spark Local 2.4) java.lang.NoSuchMethodError on Oracle components
  • TBD-10337 [Spark Streaming, CDH 6.1] Compile errors on tHiveOutput component using CDH 6.1 when all data types are used in schema
  • TBD-10340 [Spark Streaming, CDH 6.1] NoClassDefFoundError on tHiveOutput component
  • TBD-10370 "Hadoop configuration jar not found" when run spark job with jobserver on Windows
  • TBD-10436 Hive component fails with java.lang.NoClassDefFoundError: org/apache/log4j/Logger
  • TBD-10454 [7.3.1] DI job using tHDFSExist works from Studio to RE but not from TMC to RE
  • TBD-10476 [CDH6.1] java.lang.NoSuchMethodError on job with Hive components
  • TBD-10505 Databricks on AWS, cannot output to DBFS when S3 configured
  • TBD-10089 Standardize encoding UI for I/O BD components
  • TBD-10470 Wrong logo for tKinesisInput component
  • TUP-27262 tAzureAdlsGen2Output component fails with class not found error
  • TUP-25961 org.talend.commons.exception.CommonExceptionHandler - java.util.ConcurrentModificationException
  • TUP-26639 Talend Cloud studio Issue -Changes to Context Parameter Names is not reflected on the Connection for Snowflake
  • TUP-26870 Incorrect sort when listing version of Jobs
  • TUP-26958 Test case: possible to remove input / output node
  • TUP-26961 When saving/deleting a test case, it might break the poms / CI
  • TUP-26990 Possible conflict in talend project
  • TUP-26994 Usage data collector: change the way Studio Unique Id is calculated
  • TUP-27000 Talend Salesforce Einstein connector Repository connection issue in Talend 7.3
  • TUP-27003 Should disable the Commit button of Uncommitted files found dialog when project is in MERGING state
  • TUP-27199 DB version of sybase isn't hide for other database on tCreateTable
  • TDI-44066 Illegal argument exception in tSAPBapi name field
  • TDI-44089 Program Z_TLD_BI_INFOPROV not found when moving from 7.1.1 to 7.2.1
  • TDI-44159 Data viewer on tSybaseInput get error in context model with sybase 16 anywhere database
  • TDI-43619 Null Value Treated as in Subjob
  • TDI-44051 tJDBCInput Dynamic Schema does not preserve special characters
  • TDI-44130 UPDATE_OR_INSERT Mode in tDBOutput for MSSQL does not display SQL
  • TDI-43935 NString type for prepared statement in tMSSQLRow component
  • TDI-44185 Authentication Fields missing in tMongoDBBulkLoad
  • TDI-43822 tDataPrepRun Token expiration must be taken into account
  • TDI-44191 Job Deployed to MDM server and using tFileCopy fails with java.lang.ClassNotFoundException: org.talend.FileCopy
  • TDI-44122 Proxy settings not being picked up in tSalesforceBulkExec API v2
  • TDI-43605 tDBConnection to Snowflake exception "Not connected" when API call is made
  • TDI-43612 Problem with retrieving Snowflake tables from Studio
  • TDI-43629 Dynamic schema issue in tSnowflakeOutput Component
  • TDI-43682 Get error when retrieving record types in tNetsuiteInput/tNetsuiteOutput
  • TDI-43752 Snowflake component throw warnings
  • TDI-44114 Snowflake multi statement execution error
  • TDI-43903 7.3 Enhancements to tSnowflakeBulkexec broke existing functionality in the Copy (manual) command
  • TBD-10507 7.3.1 Simple Databricks Delta Lake Job is trying to do a broadcast hash join
  • TBD-10520 Files with commas get truncated by tFileInputDelimited & tFileInputFullRow Databricks / Spark 2.4
  • TDM-8036 The tHMap component giving error in 7.3.1 Studio
  • TDM-8001 Streaming with XML output creates extra tags
  • TDM-8013 Job working in 7.2.1 fails in 7.3.1 with class org.apache.commons.compress.archivers.zip.ZipFile$1 not implementing InputStreamStatistics
  • TDM-8028 Remove dependency on avro-mapred-1.7.7
  • TDM-8031 Studio keeps crashing after opening the tHMap component and running the Job
  • TDM-7667 Flattener: flattening structure that contains element with same label does not generate correct name
  • TDM-7871 Attributes are lost when importing avro schema
  • TDM-7925 Data type is changed to STRING for element with data type NONE after export to avro schema
  • TDM-7964 Problem to parse JSON array having heteregeneous elements
  • TDM-7998 Regression: test run will return error when using &/$ in variable name of get/set variable function
  • TESB-28369 CI Publish To Cloud Error when publishing demorest job to Cloud using CI on Studio
  • TESB-28130 Duplicate dependencies in POM.xml for routes lead to compile issues
  • TESB-28815 tRestClient when called in a DS Job using tRunJob fails to load in the runtime
  • TMC-20647 Publishing large Job to the Cloud always ends up with SocketTimeoutException even if it has been uploaded to Cloud
  • TDQ-18174 Remote DQ project: Fixed Git conflict with some system and technical files
  • TDQ-18322 Support the retrievial of the schema on Sybase SQL anywhere
  • TDQ-18091 tSynonymOutput: Improved the error message on incorrect path parameter
  • TDQ-18383 Fixed the high vulnerability on Log4j dependency
  • TDQ-18444 Fixed CVE issues in Nimbus
  • TUP-26809 Fix: Build job could contain jars from test cases as well. (while it should only contain jars from job)
  • TUP-27077 NoClassDefFoundException when using "independent process to run subjob" and tAzureAdlsGen2Input
  • TUP-26596 Proxy of libraries not working when studio have no internet access
  • TUP-26189 metadata connection with proxy may not select proxy properly
  • TUP-26264 tELTMap generated query has extra symbols generated when more input mapping in component
  • TUP-26156 tCreateTable: change "DBType" and "Property Type" not work
  • TUP-26482 Studio is very slow to build the job(the performance issue)
  • TUP-26212 Share more than one car component to nexus3.19.1 will fail to generate the right index file
  • TUP-26213 Compilation issue after migration to v7.2
  • TUP-26539 High Memory Consumption by Studio with GIT
  • TUP-26793 JDBC Redshift in context mode still asked for jars which is not required
  • TUP-26876 NoClassDefFoundError when I run spark job with JobServer
  • TDI-43810 fix MongoDB issue with option "Create empty element if needed"
  • TDM-7969 TDM adds unencrypted passwords to error message
  • TDM-7957 Studio commandLine error on Headless Linux following Git pull
  • TDM-7952 Nested distinct element causes performance issues
  • TDM-7932 Flatten: some value isn't mapped after flatten map
  • TDM-7931 Flatten: will return error when create flatten map from JSON structure
  • TDM-7929 Importer CSV need clean not supported header in Xquery,rriahi
  • TDM-7928 Move "Flattening Map" option at 2nd position in New Map Wizard
  • TDM-7924 merged file path is wrong after thmapfile job when "merge file path" is the same as "output folder"
  • TDM-7894 CSV show sample on element only display one record
  • TDM-7870 importing avro schema is failed for openAPI json
  • TDM-7867 Exclude Commons Collections 3.2.1
  • TDM-7865 Bad initial values and exception on the LoopIndex function dialog
  • TDM-7858 IsPresent updates are not persisted
  • TDM-7855 Exception thrown on drag/drop of looping element where dialog was expected
  • TDM-7840 Variable names in GetVariable/SetVariable produce unfriendly errors
  • TDM-7839 LoopIndex in a distinct NestedContext Simple Loop can not detect the index_of element
  • TDM-7836 Distinct funtion generate a not valid xquery in the context of AgConcat
  • TDM-7835 Distinct function generate NullPointer Exception in the context of root elements
  • TDM-7834 Distinct function generate NullPointer Exception in case of SimpleLoop Having NestedContext
  • TDM-7829 tHMapFile merge functions doesnt not check correctly the folder name
  • TDM-7826 thmapFile merge option doesnt work with Hadoop 3.1
  • TDM-7789 CSV reader should use the optimization done for the CSV writer
  • TDM-7772 Don't include a _osdtTerminator column when importing a CSV
  • TDQ-18135 tDataEncrypt cannot generate a crypto file with context variable
  • TDQ-17954 tMatchIndex, tMatchIndexPredict: parameter "Nodes" and "Index" can not use context mode value
  • TDQ-18220 After migrating from 6.3 to 7.2 the tMatchGroup job has compiler errors
  • TBD-10355 Missing Dataset to RDD method call in a migrated spark job
  • TBD-10005 HDP3.1 Compile error tHiveOutput with partitioning: hiveContext_tHiveOutput_1 can not be resolved
  • TBD-10101 tFileOutputDelimited can not keep the format of date column
  • TBD-10167 Compiler error when run spark job on Databricks with tS3configuration that use context value
  • TBD-10212 [BUG] Avro dependencies cause error in databricks
  • TBD-9674 tHiveOutput append action in spark big data batch job under HDP3.1
  • TBD-9872 byte[] type is written to file incorrectly
  • TBD-9986 user token is printed as plain text in joblog when databrick debug level log is on
  • TBD-10081 Fix wrong release maven url with -SNAPSHOT in artifact id
  • TBD-10108 Date conversion in tExtractJSONFields is not correct
  • TBD-10157 Advanced option is not considered when using tHDFSPut in Standard jobs
  • TBD-10158 Migration task can alter the pattern selected by user
  • TBD-10210 Cannot load component "tFileOutputJSON"
  • TBD-10217 org.apache.parquet cannot be resolved to a type
  • TBD-10297 slf4j logger does not contains error() method
  • TBD-10008 Wrong timestamp in tFileOutputDelimited component
  • TBD-10063 Errors on spark with Yarn Cluster with custom hadoop path on Remote Engines
  • TBD-10251 Fix issues with some characters for hadoop configuration jar
  • TUP-26576 Remove the warning about repository setup even if artifact repository is disabled in TMC
  • TUP-26751 Fix CI issues, due to invalid test case
  • TUP-26165 Fix possible missing jars when build job
  • TUP-26728 Missing spark dependencies when using test cases and fix spark compilation issues
  • TBD-10324 Fix compilation issue with RDD with tMap
  • TBD-10284 Fix issues with custom hadoop configuration with DI jobs and Spark jobs with Yarn-Cluster setup