>DevOps Interview KB

What's etcd's actual role in a cluster, and what happens if it becomes unavailable?

BeginnerConceptualKubernetesEtcd5 min read

Short Answer

etcd is the single source of truth for all cluster state — every object (Pods, Deployments, Secrets, everything) that exists in the cluster is stored there, and the API server is the only component that talks to it directly. If etcd becomes unavailable, the API server can no longer read or write cluster state, so anything requiring a new API call (deployments, scaling, kubectl commands, the scheduler placing new pods) stops working — but already-running pods keep running, since the kubelet on each node manages actual container lifecycle locally and doesn't need to reach etcd to keep existing workloads alive.

Detailed Explanation

etcd is a distributed, consistent key-value store, and Kubernetes' only persistent state store: every Kubernetes object's full desired and current state is serialized and stored in etcd — there's no other database or persistent store involved in a standard Kubernetes control plane; etcd's consistency guarantees (via the Raft consensus protocol) are what let multiple API server replicas safely read and write cluster state concurrently.

Only the API server talks to etcd directly — this is a deliberate architectural boundary: no other control-plane component (scheduler, controller-manager) or client accesses etcd directly; everything goes through the API server, which enforces authentication, authorization, and validation before any read or write reaches etcd. This means etcd's own security and access boundary is effectively defined by whoever can reach the API server with sufficient permissions, not by etcd's own separate access control.

When etcd is unavailable, the API server can't serve most requests: kubectl get pods, creating new Deployments, scaling, and any other operation that needs to read or write cluster state through the API server will fail or hang — the API server itself may still be running and accepting connections, but any request touching etcd-backed state fails.

Already-running pods keep running, because the kubelet doesn't depend on etcd for that: each node's kubelet manages the containers it's already been told to run, directly through the container runtime, independent of ongoing API server/etcd availability — this is a deliberate resilience property: an etcd outage doesn't cause a mass, immediate termination of running workloads, even though the cluster becomes unable to make new scheduling or state-change decisions during the outage.

What breaks during etcd unavailability, concretely: new pod scheduling, Deployment rollouts, scaling operations (including HPA, since it also needs to read/write via the API server), Service/Endpoints updates reflecting pod changes, and any kubectl-driven operation. What keeps working: already-running pods continue serving traffic (as long as their existing Service/Endpoints configuration doesn't need to change), and kube-proxy's already-programmed iptables/IPVS rules on each node keep routing existing traffic, since those rules were already set before the outage and don't require live etcd access to keep functioning.

Recovery depends on etcd actually coming back with its data intact: this is exactly why etcd backup/snapshot strategy matters so much — a cluster's entire state is only as durable as etcd's own data durability, and losing etcd's data without a usable backup means losing the record of what the cluster is supposed to look like, even though the actual running containers might still technically be executing until they're eventually restarted or fail on their own.

Interview Follow-Up Questions

Key Takeaways

  • etcd is Kubernetes' single source of truth for all cluster state, accessed exclusively through the API server.
  • When etcd is unavailable, any operation requiring a read/write through the API server fails — new deployments, scaling, most kubectl commands.
  • Already-running pods keep running during an etcd outage, since the kubelet manages existing container lifecycle locally without needing etcd access.
  • Recovery depends entirely on etcd's own data durability — this is why etcd backup strategy is foundational to overall cluster disaster recovery.

References

Last updated August 22, 2026 · Last reviewed August 22, 2026