>DevOps Interview KB

A container was recreated to deploy a new image version, and all its data disappeared — what went wrong with the volume configuration?

IntermediateTroubleshootingDocker6 min read

Short Answer

The container's data was almost certainly living only in the container's own writable layer, not in a volume at all — docker rm destroys that writable layer along with everything in it, and no volume declaration means there's nothing separate from the container's lifecycle to survive the removal. The fix is mounting a named volume (or bind mount) at the exact path the database writes its data to, so that data lives in Docker-managed (or host) storage independent of the container's own lifecycle, surviving docker rm and being reattached when the replacement container starts.

Detailed Explanation

A container's own writable layer is fundamentally ephemeral, tied to that specific container's lifecycle — this is a deliberate design property (containers are meant to be disposable and replaceable), and it's exactly why anything meant to persist beyond a single container's lifetime needs to explicitly live in a volume, not the container's own filesystem.

Symptoms

  • After docker rm (removing the old container) and docker run (starting a replacement), all previously-existing data is gone.
  • The replacement container behaves as if it's a completely fresh instance, with no memory of prior state.
  • No error occurred during either the removal or the new container's startup — the loss is silent.

Possible Causes

  • No -v/--mount flag was specified when the original container was run, meaning the database wrote its data files directly into the container's own writable layer, which is destroyed along with the container on docker rm.
  • A volume was mounted, but at the wrong path — not matching the actual path the database process writes its data files to, meaning the volume was present but the data was never actually being written into it.
  • The volume was correctly configured, but the replacement container's docker run command didn't mount the same volume (a different volume name, or no volume flag at all on the new container), meaning even though the old data survived in the original volume, the new container simply isn't attached to it.

Investigation Steps

Check whether the original container had any volume mount at all: docker inspect <old-container-id> (if the container object still exists, even removed containers can sometimes still be inspected briefly, or check deployment scripts/history) shows the Mounts section — confirming whether any volume was ever actually configured is the first, most direct check.

Check the database's actual data directory path against whatever was mounted: comparing the database software's documented data directory (where it actually writes its files) against the specific path that was mounted as a volume confirms whether a volume existed but was mounted at the wrong location, missing the actual data.

Check whether the underlying volume still exists, even if the new container isn't using it: docker volume ls shows all volumes Docker currently knows about — if the original volume still exists (Docker volumes survive docker rm of a container that used them, unless explicitly removed too), the data may not actually be lost, just not attached to the new container, which is a very different (and recoverable) situation than genuine permanent data loss.

Resolution

If the original volume still exists and simply wasn't reattached, stop the new container, remove it, and re-run it with the correct -v <original-volume-name>:<data-path> flag matching both the existing volume and the correct data path — this recovers the "lost" data immediately, since it was never actually destroyed. If no volume ever existed and the data was genuinely only in the removed container's writable layer, the data is permanently unrecoverable, and the resolution is purely forward-looking: establish a deployment process that always explicitly mounts a named volume at the correct data path, and treat this as a trigger to also set up actual backups (a volume surviving container recreation is not the same as being backed up against volume-level loss, host failure, or accidental docker volume rm).

Interview Follow-Up Questions

  • How would you design a deployment script/process to make it structurally difficult to accidentally recreate a stateful container without its volume correctly attached?
  • How would you back up data in a named volume on a regular, automated basis?
  • How does this same class of mistake manifest differently in Kubernetes, given Pods and PersistentVolumeClaims have a different lifecycle relationship than Docker containers and volumes?

Key Takeaways

  • A container's own writable layer is ephemeral and tied to that specific container — anything not explicitly placed in a volume is destroyed on docker rm.
  • Check docker volume ls before assuming data is permanently lost — if a volume was used but just not reattached to the replacement container, the data may still exist and be recoverable.
  • A volume mounted at the wrong path (not matching where the application actually writes data) is functionally the same as having no volume at all for that data.
  • Volume persistence across container recreation is not the same as a backup — a volume itself can still be lost (accidental removal, host failure), so genuine backups are a separate, necessary concern.

References

Last updated August 22, 2026 · Last reviewed August 22, 2026