In my previous NAP post, I wrote about an Azure Policy that blocked node provisioning when Node Auto Provisioning (NAP) fell back to managed OS disks. After fixing that, we continued testing NAP as a replacement for the cluster autoscaler (CAS) on AKS.
This time, we looked at consolidation: how NAP removes or replaces nodes to make better use of capacity.
We decided not to migrate for now. For our applications, avoiding interruptions to daily calculations mattered more than the extra savings we expected from consolidation.
The Daily Calculations Need to Finish
Our financial applications run as Kubernetes Deployments. The application inside each pod performs complex calculations each day. These are not Kubernetes Jobs or CronJobs; the daily processing happens inside the running application.
Once a calculation starts, we want it to finish without an avoidable interruption. With the application’s current recovery behavior, interrupted processing needs to be manually retriggered. A replacement pod restores the application, but it does not resume the calculation from where it stopped.
That was the main concern. Kubernetes could bring the workload back up, but someone would still need to deal with the interrupted processing.
What We Saw During Consolidation
With WhenEmptyOrUnderutilized, we observed repeated pod rescheduling as NAP consolidated nodes.
Under this policy, NAP can consider removing empty nodes, moving workloads onto existing capacity, or replacing nodes with a cheaper option. The AKS disruption documentation describes this behavior.
That makes sense for workloads that can handle eviction and restart. For ours, it meant a node could be selected for consolidation while the application was still working through its daily calculations.
When I say pods were “moving,” I mean the existing pods were terminated and the Deployment’s ReplicaSet created replacements that were then scheduled. A running pod is not transferred to another node with its process state intact.
This was what we saw in our test environment, not something every NAP deployment will experience. Workload patterns, resource requests, and disruption settings affect the result.
Could We Control the Disruption?
Yes. NAP has consolidation settings, disruption budgets, and pod-level controls. The issue was not that we had no control; it was that the configuration we tested did not suit these daily calculations.
NodePool disruption budgets can limit voluntary disruption, including blocking it during scheduled windows. But allowing only one node to be disrupted still leaves a problem if a pod on that node is in the middle of a calculation. Scheduled windows also need to allow for processing that runs longer than expected.
PodDisruptionBudgets can block voluntary evictions too. They are useful, but they do not track whether the calculations inside a pod have completed.
Using WhenEmpty
One alternative we considered was to consolidate only empty nodes. This is a fragment of a NodePool, not a complete manifest; the five-minute delay is just an example:
spec:
disruption:
consolidationPolicy: WhenEmpty
consolidateAfter: 5mIn AKS NAP, WhenEmpty limits consolidation to nodes without workload pods. It avoids consolidating an occupied node simply because its workloads could run elsewhere more cheaply. It does not disable other replacement paths, such as drift or expiration.
Using do-not-disrupt
We also considered protecting sensitive pods with this annotation in the Deployment’s pod template (this is a fragment, not a complete Deployment):
spec:
template:
metadata:
annotations:
karpenter.sh/do-not-disrupt: "true"The annotation belongs on the pod template, not on the Deployment’s own metadata. A static annotation protects the pod for its lifetime; it does not automatically switch off when the day’s calculations finish.
The annotation excludes nodes hosting those pods from consolidation. It also blocks drift-based disruption unless a node terminationGracePeriod is configured, in which case drift can still proceed and the deadline can force pod termination. See Karpenter’s pod-level controls alongside the AKS documentation, since upstream features can differ from the managed NAP version.
It does not prevent expiration from initiating disruption, manual deletion, or infrastructure failures. It can also leave a node waiting to drain while protected pods remain. It is a useful control, not a guarantee of an uninterrupted run.
What That Meant for the Cost Savings
Using WhenEmpty and workload protection looked closer to what we needed. But it also reduced the reason we were considering the move.
If occupied nodes cannot be consolidated, partially used capacity may need to stay. Finishing the day’s calculations does not make the node empty: the Deployment pods can remain running between calculation runs. With a static do-not-disrupt annotation, their protection also stays in place. That leaves fewer opportunities to save money through consolidation.
NAP would still offer dynamic VM sizing for pending pods and empty-node removal. It would not become the same as CAS. But for our workload, the expected extra savings from that more conservative setup were not enough to make a migration worthwhile.
That is our assessment, not a measured cost comparison. We are not claiming that NAP and CAS would produce the same bill.
Staying with CAS for Now
CAS can also drain occupied nodes during scale-down. Staying with it does not make our applications immune to eviction, AKS maintenance, or infrastructure failures.
For us, keeping CAS means retaining the setup we already operate rather than changing autoscalers for a benefit that became less convincing once we accounted for protecting the calculations. We still need to manage scale-down and maintenance carefully.
For now, we are comfortable accepting some unused capacity to avoid adding more manual recovery of interrupted calculations to day-to-day operations.
We may revisit NAP for workloads that tolerate eviction, or after improving calculation recovery with checkpointing and safe retries. For these applications as they work today, we decided to keep CAS.
