Cannot submit pipeline <PIPELINE_NAME>, too many Livy sessions are used.
where <PIPELINE_NAME> is the name you gave to your pipeline.
For safety reasons the number of concurrent pipeline executions is limited, but this limit is configurable for the Remote Engine Gen2.
Configure the number of allowed concurrent executions
- <engine_directory>/default/.env if you are using the engine in the AWS USA, AWS Europe, AWS Asia-Pacific or Azure regions.
- <engine_directory>/eap/.env if you are using the engine as part of the Early Adopter Program.
LIVY_SERVER_SESSION_MAX_CREATION=(memory - 4)/spark.driver.memory
where memory corresponds to the memory allocated to the engine, 4 corresponds to the 4GB of memory necessary for other services of the engine, and spark.driver.memory corresponds to the memory allocated to each pipeline execution (1GB by default).
The spark.driver.memory default value can be changed by adding the parameter and value in the Advanced configuration section of the Add run profile form in Talend Cloud Management Console.