A corrupted or accidentally-overwritten Terraform state file threatens to make you lose track of an entire environment's real infrastructure — how would you design for recoverability?

Use a remote backend with built-in versioning (S3 with versioning enabled is the standard example) so every historical version of the state file is automatically retained and recoverable, combined with state locking (preventing concurrent corruption from simultaneous writes) — this gives you both prevention (locking stops the most common corruption cause) and recovery (versioning lets you roll back to a known-good version if corruption or an accidental bad write happens anyway).

State file loss doesn't mean infrastructure loss — the real resources keep running — but it does mean losing Terraform's own record of what it's managing and how, which without a backup essentially means starting over with a large import effort; designing for recoverability is specifically about avoiding that painful reconstruction.

• The state file's history should be recoverable, not just its current version — a bad write shouldn't be permanently unrecoverable.
• Concurrent writes from multiple simultaneous Terraform operations should be prevented, since this is a primary corruption cause.
• Recovery from a bad state version should be possible without needing to fully reconstruct the environment from scratch via import.

Remote backend versioning is the primary recovery mechanism: S3 (or the equivalent versioned storage for whatever backend is in use) retaining every historical version of the state object means an accidental bad write, corruption, or unwanted overwrite doesn't destroy history — the previous, known-good version remains retrievable, and restoring it (copying that specific version back to be the current one) recovers the state to exactly where it was before the incident.

State locking prevents the most common cause of actual corruption in the first place: two concurrent Terraform operations writing to the same state simultaneously (without locking preventing this) is a classic corruption scenario — proper locking (via the backend's locking mechanism, like DynamoDB for an S3 backend) is prevention, which is more valuable than recovery, since it stops the most common incident from happening at all rather than just making it recoverable after the fact.

Separate, deliberate backups beyond the backend's own versioning add defense in depth: while S3 versioning is generally sufficient on its own, some organizations also maintain periodic explicit exports/backups of critical state files to a genuinely separate location (a different bucket, different account) as an additional layer, protecting against the unlikely scenario where the primary backend itself has a broader issue (accidental bucket deletion, a broader account-level incident) that versioning alone within that same bucket wouldn't protect against.

Recovery procedure should be tested, not just theoretically available: knowing that S3 versioning exists isn't the same as having actually practiced restoring a previous state version and confirming a subsequent terraform plan shows no unexpected changes — testing this procedure (in a non-production environment, or as a deliberate game-day exercise) is what actually validates the recovery capability works as expected, rather than discovering a gap in it during a real incident.

Access controls on who can delete/overwrite the state (beyond just recovering from an accident) matter too: restricting who has permission to delete state versions entirely (not just write new ones) protects against a scenario where recovery itself becomes impossible because the historical versions were also deliberately or accidentally removed — this is a distinct concern from normal write-locking, addressing deletion specifically.

Enabling versioning and maintaining it (potentially with lifecycle rules to eventually age out very old versions, balancing storage cost against how far back recovery might realistically be needed) adds a small amount of ongoing storage cost and configuration compared to unversioned storage — this cost is negligible relative to the value of avoiding a full environment reconstruction via import if state is ever lost or corrupted, making it a clearly worthwhile default for any state managing real production infrastructure.

• Remote backend versioning (S3 versioning, for the common case) is the primary recovery mechanism, retaining every historical state version for rollback if needed.
• State locking is prevention, addressing the most common actual corruption cause (concurrent simultaneous writes) — more valuable than recovery alone, since it stops the incident from happening.
• Separate, periodic explicit backups beyond the backend's own versioning add defense in depth against a broader incident affecting the primary backend itself.
• Test the actual recovery procedure (restoring a previous version, confirming a clean subsequent plan) rather than assuming it works based on versioning being theoretically enabled.

• Terraform: Backend Configuration — S3
• AWS: Using versioning in S3 buckets