Why Your Readiness Probe Is Lying to You
At Luxottica, a new version of the Java Ad Service deployed to production EKS. The pipeline reported success, all six pods showed Ready 1/1 — and within eight minutes, Prometheus was alerting on an elevated error rate from a service that, by every Kubernetes signal, was healthy.
What happened
Application code regression was ruled out quickly — every error was a connection or initialization error, not a logic error. The actual problem was the readiness probe itself: it checked whether the HTTP server had started, not whether the application was actually ready to serve traffic. The /health endpoint returned 200 within five seconds of pod start, but dependency initialization — database connection pool, cache client, downstream API clients — took eighteen to twenty-two seconds. Every pod had a guaranteed ten-second window where Kubernetes was actively routing production traffic to a pod that wasn’t ready to handle it.
The fix
The immediate fix needed no code change: I increased initialDelaySeconds to 30 on the readiness probe and triggered a rolling restart. Error rate dropped to zero within four minutes. The real fix came from the application team, who implemented a /health/ready endpoint that checks actual dependency readiness instead of just process existence — that became the standard across every service. I published a Kubernetes probe standards document per runtime (Java, Golang, Python), embedded correct probe defaults directly into the Helm service template so engineers have to explicitly justify any override, and added a staging smoke test that sends traffic during the first 30 seconds after a new version comes up and fails the pipeline if the error rate exceeds 0.1%.
The lesson
A readiness probe that passes the moment the HTTP server starts isn’t a readiness probe — it’s a process existence check. Real readiness means the application can actually serve the traffic that’s about to be routed to it the instant the probe goes green, and that’s a meaningfully different thing to verify than “is a server listening on the port.”