EMR YARN Dynamic Executor Allocation
In a YARN-managed Spark deployment on Amazon EMR, dynamic allocation adjusts the number of executors based on the current workload:
- Scale up: when pending tasks exceed available executor slots, the ApplicationMaster requests new containers from the ResourceManager
- Scale down: when executors sit idle beyond a timeout threshold, they are released back to the cluster
- Graceful decommission: before releasing an executor, Spark migrates cached data and waits for running tasks to complete
This mechanism prevents over-provisioning while ensuring sufficient resources during computation spikes.
A YARN ResourceManager and three EMR NodeManagers whose Spark executors grow from two to six under a backlog of tasks and shrink back to one as they sit idle, with the executor count plotted over time against the workload.