How would you find the same df/du phantom-disk-usage issue on a containerized workload, where the culprit process might be in a different mount namespace?

lsof +L1 run on the host still finds deleted-but-open files across all processes visible to it, since the host can see into every container's process from the host's own PID namespace — the complication isn't that the file becomes invisible, it's correctly identifying which container a given open file descriptor belongs to, since the process holding it appears in the host's process list but its filesystem context is a different container's overlay/mount namespace, requiring correlating the PID back to a specific container.

Containers share the host kernel, so the host's own process and file-descriptor visibility already spans every container running on it — the missing piece is purely the mapping from a raw host-visible PID to the specific container that PID belongs to.

• Host-level disk usage (df) shows a filesystem at or near full, but du totals across the visible filesystem don't account for the gap.
• The suspected cause is a container's process holding a deleted file open, but which container isn't immediately obvious.

• A containerized process has a deleted-but-open file, most commonly a log file rotated incorrectly inside the container.
• The container's writable layer or a bind-mounted host directory is where the phantom disk usage is actually accumulating.

1. Run lsof +L1 on the host (not inside any individual container) — since containers are just processes from the host's perspective (sharing the host kernel), the host's lsof can see every process's open file descriptors, including deleted-but-open ones, across every container.
2. For any deleted-but-open file found, identify the owning process's PID from lsof's output.
3. Correlate that PID back to a specific container: docker inspect --format '{{.State.Pid}}' <container> for each running container, matching against the PID found, or more directly, use docker top <container> for each container to see its process list, or a single pass via for c in $(docker ps -q); do echo "$c: $(docker top $c)"; done to build the mapping quickly.
4. Once the owning container is identified, confirm whether the growing file lives in that container's writable layer (typically under /var/lib/docker/overlay2/<container-id>/... on the host) or on a bind-mounted host directory — this determines whether restarting the container alone frees the space, or whether the bind-mounted host path itself needs attention.

sudo lsof +L1

for pid in <found-pids>; do
  for c in $(docker ps -q); do
    cpid=$(docker inspect --format '{{.State.Pid}}' "$c")
    [ "$cpid" = "$pid" ] && echo "PID $pid belongs to container $c"
  done
done

docker top <container>
docker inspect --format '{{.GraphDriver.Data.MergedDir}}' <container>

Once the specific container is identified, the fix mirrors the non-containerized case: restart the container (or specifically the process inside it, if the container supports that) to release the deleted file's held-open descriptor and free the space — for a containerized workload, restarting the container is usually the simplest reliable way to guarantee the process re-opens a fresh handle, since directly signaling a process inside another mount/PID namespace to reopen a log file can be more awkward to target correctly from the host.

• Configure log rotation inside the container image/application the same way as the non-containerized case (copytruncate or signal-based reopening) — container log rotation footguns are the same underlying issue, just easier to lose track of across many containers.
• Monitor each container's actual disk usage (via docker stats, or cAdvisor/Prometheus metrics if using Kubernetes) rather than only host-level aggregate disk usage, so a specific container's growing phantom usage is visible before the host-level filesystem fills up entirely.
• Prefer sending container logs to stdout/stderr and letting the container runtime's own logging driver handle rotation externally, rather than each application managing its own log files inside the container — sidesteps this entire class of issue for logs specifically.

• lsof +L1 run on the host still finds deleted-but-open files across all containerized processes, since containers share the host kernel and are visible to the host's process/file-descriptor view.
• The complication is correlating a found PID back to a specific container, not the file itself becoming invisible.
• docker inspect/docker top provide the PID-to-container mapping needed to identify the culprit.
• Sending logs to stdout/stderr and letting the container runtime handle rotation externally avoids this entire class of issue for logging specifically.

• Linux man-pages: lsof(8)
• Docker Docs: View logs for a container or service