The Terraform State Corruption Incident That Made DynamoDB Locking Non-Negotiable

At Luxottica, on Project VisionLens, our Terraform remote state lived in S3 without a locking mechanism in front of it. It worked fine — until two engineers applied changes to the same staging environment within 90 seconds of each other: one adding a new EKS node group, the other updating an RDS parameter group.

What happened

Both applies reported success. Twenty minutes later, a third engineer ran terraform plan and saw a wall of unexpected resource changes. With no DynamoDB lock on the S3 backend, both applies had read the same initial state, computed their diffs independently, and written their results back — last write won, partially overwriting the other’s changes. The AWS resources themselves were correct: the node group existed, the parameter group had been updated. It was Terraform’s own state file that no longer matched reality — dangerous, because the next apply would have tried to “correct” real infrastructure back to stale values.

The fix

I put a manual lock in DynamoDB to stop any further applies, then used terraform state show to find exactly which resources were out of sync, and terraform state rm plus terraform import to rebuild the state representation for those specific resources — surgical correction rather than a full restore, since the real AWS resources were already correct. I implemented DynamoDB state locking immediately and made it non-negotiable for any team touching shared infrastructure going forward, and restructured state into separate files per environment and per component, so a bad apply against one component can’t touch another’s even if two people are working concurrently. Recovery time: two and a half hours.

The lesson

Remote state without locking isn’t a best-practice gap you get around to eventually. It’s a production incident waiting for the right — wrong — moment, usually two people working at once under deadline pressure, which is exactly when it’s most likely to happen. S3 versioning is what made recovery possible at all; without it, this would have been significantly worse. The incident in staging made a stronger case for locking than any architecture document could have. After that, it stopped being a recommendation.