A Peak-Traffic RDS Failover Outage, and Why Disaster Recovery Isn’t a Configuration

At Home Depot, a Prometheus alert fired on elevated write-error rates across the checkout service during a peak shopping window — write operations spiked from a 0.1% baseline to over 18% in four minutes, with live revenue impact.

What happened

We rolled back the most recent deployment as a precaution. The error rate didn’t move — the deployment wasn’t the cause. The actual root cause was two compounding problems: the application was hardcoded to the primary RDS instance endpoint instead of the cluster endpoint, and when Multi-AZ failover promoted the standby, the cluster endpoint updated automatically but the application kept routing writes to the old primary, now a read-only standby. The connection pool held those stale connections for five minutes before timing out. The failover itself worked exactly as designed — the application just wasn’t written to survive one.

The fix

We updated the application config to use the RDS cluster endpoint and performed a rolling restart. Write errors dropped to zero within three minutes of the restart completing. Total incident duration: 47 minutes. Afterward, I built an automated CI/CD check that fails the build if it detects a hardcoded RDS instance endpoint in an application config, added monthly failover testing in staging via AWS Fault Injection Simulator, and cut the connection pool timeout from five minutes to thirty seconds.

The lesson

Multi-AZ failover protects the database. It does not protect you from applications that were never written to handle one gracefully. Infrastructure resilience and application resilience are two separate engineering problems, and testing one doesn’t validate the other — this incident is part of why I rebuilt our monitoring around SLIs and SLOs afterward, replacing confidence based on configuration with confidence based on evidence.