EMR YARN Dynamic Executor Allocation

In a YARN-managed Spark deployment on Amazon EMR, dynamic allocation adjusts the number of executors based on the current workload:

• Scale up: when pending tasks exceed available executor slots, the ApplicationMaster requests new containers from the ResourceManager
• Scale down: when executors sit idle beyond a timeout threshold, they are released back to the cluster
• Graceful decommission: before releasing an executor, Spark migrates cached data and waits for running tasks to complete

This mechanism prevents over-provisioning while ensuring sufficient resources during computation spikes.