Running CloudNativePG on Spot Nodes

I (currently) run fifteen small Postgres clusters with CloudNativePG (CNPG) on an EKS cluster whose application nodes are spot instances managed by Karpenter. Each cluster has two instances, a primary and a streaming replica, spread across two availability zones. That shape is meant to survive losing a node. These are all instances for small projects (with very few users), and the applications themselves were resilient to short periods of database unavailability. One afternoon recently, a spot interruption took out the node that happened to hold every database primary, and every one of those databases did an unplanned failover. The applications could handle this, but what I did not expect (and apparently I could not handle) was getting 45+ notifications from my monitoring in the span of 3 minutes. I needed to finally dig in and figure out what was going on.

Karpenter got the interruption notice, and CNPG already knows how to move a primary off a node that’s being drained. The defaults just don’t fit inside a two-minute warning. A handful of settings changed that, and about forty minutes later an interruption hit the node that by then held every primary again. That time all fifteen clusters did a switchover, and none of them failed over. But for twelve of the fifteen, the new primary only took over after the two minutes were up, and the reason for that wasn’t in the CNPG settings at all.

What Happened#

AWS issues a spot interruption notice two minutes before it stops or terminates the instance. Karpenter reads that notice from an SQS queue and starts its termination flow. For the 00:01 interruption, the timeline (UTC) looked like this:

• 00:01:35: Karpenter logged initiating delete from interruption message with messageKind: spot_interrupted, added the karpenter.sh/disrupted:NoSchedule taint to the node, and started evicting pods. Every eviction event for the node has that same timestamp, including the evictions of all fifteen Postgres primaries.
• 00:01:35: Failed to drain node, 58 pods are waiting to be evicted.
• 00:03:19 to 00:03:25: CNPG logged Current primary isn't healthy, initiating a failover for the first four clusters.
• About two minutes after the taint: Failed to drain node, 12 pods are waiting to be evicted.
• 00:03:37: The instance shut down.
• 00:04:18 to 00:05:12: CNPG started failovers for the other eleven clusters.

The evictions went through right away, and no PodDisruptionBudget held anything back.

Why the Primaries Probably Didn’t Stop in Time#

When CNPG’s instance manager gets the termination signal, it shuts Postgres down in two steps. From the CNPG instance manager docs:

1. It issues a CHECKPOINT, then starts a smart shutdown, which lasts for up to smartShutdownTimeout seconds.
2. If Postgres is still up, it requests a fast shutdown, and waits for up to the rest of stopDelay.

The defaults are 180 seconds for smartShutdownTimeout and 1800 seconds for stopDelay. Both of those are longer than the 120 seconds a spot instance gets.

The smart shutdown is the likely problem, because of what it waits for. The Postgres documentation describes it as letting “existing sessions end their work normally”, and it “shuts down only after all of the sessions terminate”. That’s sessions, not transactions. My databases see very little traffic, and no transaction in them runs for anything close to a minute, but every application keeps a pool of idle connections open, and a few of the clusters sit behind PgBouncer. Those connections don’t go away on their own, so a smart shutdown can use its entire timeout waiting for them.

A fast shutdown is the one that actually ends things. It disconnects the sessions and aborts any open transactions, so for a lightly loaded database it finishes quickly.

I can’t show this directly. The instance logs from that node didn’t survive (this is a separate problem that’s outside the scope of this post), so I don’t have the shutdown sequence of any of those primaries, and four of the failovers started before the instance went away. What I do know is that the primaries started a shutdown that was allowed to take three minutes, on a machine with two minutes left, and that CNPG never got to do a switchover.

CNPG Already Handles Drains#

The part I hadn’t appreciated is that while CNPG already has the behavior I wanted, the default configuration was very much not applicable for my use case. Its documentation on Kubernetes upgrades and maintenance says:

If a node is to be drained and contains a cluster’s primary instance, a switchover happens ahead of the drain. Once the instance in the node is downgraded to replica, the draining can resume.

In the operator source (I read v1.29.0, the version I run), the cluster controller watches Node objects, and on each reconcile it checks whether the node under the current primary has one of a configurable list of taints (internal/controller/replicas.go). If it does, and there’s a ready replica on another node with an active streaming connection, CNPG sets that replica as the target primary and does a switchover.

The list of taints is the operator setting DRAIN_TAINTS, and its default (internal/configuration/configuration.go) already includes Karpenter’s:

var DefaultDrainTaints = []string{
	"node.kubernetes.io/unschedulable",
	"ToBeDeletedByClusterAutoscaler",
	"karpenter.sh/disrupted",
	"karpenter.sh/disruption",
}

That matters for Karpenter in particular, because Karpenter doesn’t cordon the node in the kubectl cordon sense. The node I lost was never cordoned: it still showed Unschedulable: false at the end. The drain signal was the taint.

So CNPG had the signal, but I had not given it the time for a proper, and safe, switchover. Karpenter added the taint and evicted the primary in the same second, and the mechanism that’s supposed to hold the eviction until the switchover is done was turned off. I didn’t find any log line showing CNPG noticing the drain between the taint and the failovers.

The Settings#

These are the settings I ended up with on every cluster:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: app-db
spec:
  instances: 2
  enablePDB: true
  stopDelay: 60
  smartShutdownTimeout: 10
  switchoverDelay: 30
  primaryUpdateMethod: switchover

enablePDB: true. This is the default, and mine had been turned off. The clusters were single-instance when the setting was added, and a PodDisruptionBudget on a single-instance database blocks every node drain. With two instances, it does the job it’s meant to. CNPG creates a <cluster>-primary PDB with minAvailable: 1 that selects only the pod with the primary role. Karpenter’s eviction of the primary is refused, while the rest of the drain carries on. Meanwhile CNPG sees the taint and does the switchover. Once that’s done, the old primary has the replica role, the PDB no longer selects it, and a later eviction can go through.

stopDelay: 60. This is the total time an instance gets to shut down, and CNPG uses it as the pod’s terminationGracePeriodSeconds (pkg/specs/pods.go). Before the change my Postgres pods had a grace period of 1800 seconds, and after it they had 60. Sixty seconds is enough for a fast shutdown of a small, quiet database.

smartShutdownTimeout: 10. This is the part of stopDelay that goes to the smart shutdown. Given that pooled connections hold a smart shutdown open, I want it short. Ten seconds gives a well-behaved client a chance to disconnect, and leaves fifty seconds for the fast shutdown. CNPG’s docs say the fast shutdown window needs to be at least 15 seconds, and the field description asks you to leave room for it (stopDelay minus smartShutdownTimeout).

switchoverDelay: 30. This one only applies to the old primary during a switchover. CNPG does a fast shutdown of the old primary before it promotes the replica, so that all the data is available on the new primary. switchoverDelay limits how long that shutdown can take, and the default is 3600 seconds. It does not limit the switchover as a whole: the promotion of the new primary comes after it, and has no limit of its own here. The CNPG docs carry a warning about this setting: a low value favors recovery time over recovery point, and can lead to data loss at the cluster or backup level if the old primary hasn’t finished archiving its WAL. For my databases and their write volume, I’m taking that trade.

primaryUpdateMethod: switchover. This one isn’t about spot at all, but I found it while rolling out the others. The default is restart. When I changed the shutdown settings (which changes the pod spec), CNPG updated each replica, then deleted and recreated each primary pod, logging Primary instance is being restarted without a switchover. Each database was without a primary until the new pod came up. With switchover, CNPG promotes the replica it has already updated, and only then restarts the old primary as a replica. Changing this field on its own didn’t restart anything: all thirty instance pods kept their UIDs.

The rollout of the other four settings did restart every instance, replica first and then primary. Plan for that.

The 00:41 Interruption#

About forty minutes later, Karpenter received another spot_interrupted message, this time for the node that by then held all fifteen primaries again.¹ The settings above had finished rolling out a few minutes earlier.

• 00:41:12: Karpenter tainted the node and started the drain.
• 00:41:15 to 00:41:29: Each of the fifteen clusters logged SwitchingOver, with the message Current primary is running on unschedulable node ..., switching over from ... to ....

There were no FailingOver events, and afterwards all thirty instances were ready. That looks like a clean result, and I was pretty happy to call this done at that point. Then I compared the time of the taint with each cluster’s status.currentPrimaryTimestamp, the time its new primary took over:

• Three clusters: 75.6, 90.9 and 91.8 seconds.
• The other twelve: 133.6 to 147.4 seconds.

Twelve of the fifteen handovers finished after the two-minute notice, and after the old node had gone away. The old primaries had stopped within seconds of the switchover starting, so those twelve clusters had no writable primary for most of that time. Nothing failed over, and no data was lost to a failover, but it was a long write outage.

Why the Promotions Were Slow#

The old primaries weren’t the slow part. Each one did its checkpoint, a fast shutdown, and an archive of its last WAL file in 6 to 13 seconds. The time went to the new primaries.

When Postgres promotes a standby, it calls restore_command several times in a row. It looks for the next timeline history file, the next WAL segment, and the current history file, and most of those calls are expected to come back with “does not exist”. On these clusters, there were about five calls per promotion. I archive WAL with the barman-cloud plugin, so each call runs barman-cloud-wal-restore in the plugin’s sidecar container, which reads from S3.

During the interruption, I looked at two of the clusters in detail, and each call took about 12 to 36 seconds: 12.2 to 15.8 seconds on one, and 13.1 to 35.6 seconds on the other. The five calls added up to 69.5 of the 80.7 seconds from target to promotion on the first, and 109.6 of 130.7 seconds on the second. On both, the first two calls started before Postgres logged “received promote request”, with the startup process blocked inside restore_command, so the slow calls delayed the promotion before it began as well as during it. Later, I ran a controlled switchover on one cluster, on a node with 4 vCPUs and no added load. The same five calls took 0.37 to 0.44 seconds each, and the promotion finished 3.9 seconds after CNPG picked the target.

The difference was the node. All fifteen new primaries were on one node with two vCPUs, and its CPU requests rose from about 79% to about 97% of its allocatable CPU during the promotions. During the promotions it had a one-minute load average of 46 to 52, 93 runnable processes, no idle CPU, no iowait, a CPU pressure (PSI) value of 0.99, and node-exporter scrapes that failed. The CNPG operator logged Error loading plugins ... context deadline exceeded during the promotions, although the same error also appears a few minutes before the interruption.

The barman-cloud sidecar has no CPU request (resources: {}), so when the CPU is contended, it gets the smallest share. That made it the obvious suspect, so I tested it.

Testing the Sidecar#

I used one two-instance cluster with low traffic, and did each switchover by setting the cluster status, which is the same change kubectl cnpg promote makes. For the baseline run and all four loaded runs, the new primary was the same instance, on the same node with 4 vCPUs. For the loaded runs, a pod on that node ran eight busy loops (while :; do :; done) with a CPU request of 500m, and each run started 20 seconds after that pod was ready. The only thing I changed between runs was the CPU request of the plugin-barman-cloud sidecar on the target pod. I changed it with an in-place pod resize (kubectl patch pod --subresource=resize), so the container didn’t restart.

The duration is from the cluster’s targetPrimaryTimestamp to its currentPrimaryTimestamp. Every promotion made five restore_command calls, one after the other.

The 1m run stands in for a second run with no request. Kubernetes won’t let a resize remove a request (“resource requests cannot be removed”), and 1m gives about the same CPU weight as no request at all, because the kernel’s minimum is two shares.

I also ran four more switchovers with no load, but those promoted the other instance, on a 2-vCPU node that held other clusters’ primaries and had some load of its own. They took 5.3 to 6.8 seconds, with 0.52 to 1.27 seconds per call, so they aren’t part of the comparison.

Under the same load, a 50m request on the sidecar cut each restore call by about half, and the promotion went from 12 to 15 seconds down to about 8.

There are limits to what this shows. It’s two runs per condition, which is a small sample. The load was much lighter than the interruption: eight busy loops on four vCPUs, against a two-vCPU node with a load average of 46 to 52 and 93 runnable processes. So the test confirms the mechanism and its direction, but it doesn’t reproduce the 71.4 to 137.8 seconds from target to promotion in the interruption. And 50m is just the value I tested. I haven’t tested other values, so don’t read it as a recommended size.

The plugin lets you set resources for the sidecar through spec.instanceSidecarConfiguration.resources on its ObjectStore. I haven’t set it on my clusters. A CPU request on thirty sidecars takes node capacity, and I’m keeping this cluster small. The trade-off is between the size of the cluster and how long a switchover takes when the node is busy.

The Time Budget#

Here’s the sequence on an interruption now, what bounds each step, and what I measured on 00:41:

The step that switchoverDelay limits took 6 to 13 seconds. The promotion has no limit, and its time depends on whatever the restore command depends on. If you run larger or busier databases, or a different archive setup, measure your own. kubectl cnpg promote on a test cluster gives you the switchover time with no other load, which is the best case, not the one you get during an interruption.

Things to Watch Out For#

The restore path is part of your failover time. Anything that slows restore_command slows every promotion, and the promotion makes several calls in sequence. On a contended node, a sidecar with no CPU request is an easy place for that time to go.

Every primary can end up on one node. Both times a node with primaries was interrupted, every primary in the cluster was on that node, and after the 00:41 switchovers every new primary was on a single node too. The zone anti-affinity spreads the two instances of a cluster, but nothing spreads primaries across clusters. With the fix in place, losing that node means fifteen switchovers at once rather than fifteen failovers, and fifteen promotions at once on whichever node holds the replicas.

A primary PDB blocks consolidation of its node. Karpenter won’t voluntarily disrupt a node when a PDB would block one of its evictions. It reports this as a DisruptionBlocked event with Pdb prevents pod evictions. A node that holds a primary can’t be consolidated until the primary moves. With two-instance clusters, CNPG doesn’t create a PDB for replicas at all, since it only creates the replica PDB at three or more instances (pkg/specs/poddisruptionbudget.go).

The notice isn’t guaranteed. AWS says interruption notices are “emitted on a best effort basis”. Everything above depends on getting the two minutes. Without them, the PDB and the drain taints never come into play, and the result is a failover. I would always recommend determining how much time your particular switching should need, and configuring CNPG appropriately. The faster you can keep the switchover, the more headroom you have if AWS doesn’t deliver the expected notice. If your systems cannot switchover within the 2 minute notice period, a single-leader database on spot instances might just not be a good fit for your workload.

A failover can lose data. The CNPG docs say directly that to avoid data loss, you shouldn’t delete the pod where the primary is running, and should perform a switchover first. On the 00:01 interruption, every database did exactly what that warns against. If you can’t tolerate losing the last few transactions, this matters more than the downtime does.

primaryUpdateMethod: switchover has a restriction. CNPG rejects a change that updates the image and the PostgreSQL parameters at the same time. You have to do one, wait for the rollout, and then do the other.

Summary#

If you’re running CNPG on spot nodes with Karpenter, check these on every Cluster:

• enablePDB is true, and the cluster has at least two instances.
• stopDelay is well under 120 seconds. It becomes the pod’s termination grace period.
• smartShutdownTimeout is short, and leaves at least 15 seconds of stopDelay for the fast shutdown.
• switchoverDelay fits in the window, and you’ve accepted what a low value means for your recovery point. It only limits the old primary’s shutdown.
• primaryUpdateMethod is switchover, so rolling updates don’t take the primary down either.

Then check what a promotion costs on your setup. If restore_command goes to object storage through a sidecar, give that sidecar enough CPU to make its calls quickly on a busy node, or keep the new primaries from all landing on one node. And check that the operator’s DRAIN_TAINTS still includes karpenter.sh/disrupted if you’ve overridden it.