Why might a pod with a very specific nodeSelector never get scheduled, even though matching nodes exist with capacity?

Beyond a genuine label mismatch (a typo in the key or value, which is the first thing to rule out), the other common causes are that the matching nodes are cordoned or tainted (a nodeSelector match doesn't override a taint the pod doesn't tolerate), the label was removed or changed after someone last checked, or the "capacity" that looks free doesn't actually account for reserved/allocated resources correctly.

nodeSelector matching is necessary but not sufficient for scheduling — a node satisfying the label requirement can still be excluded by an entirely separate mechanism (taints, cordoning, resource accounting), and confirming the label matches is only the first of several independent checks needed.

• The pod remains Pending.
• kubectl get nodes --show-labels appears to show one or more nodes with the exact label the nodeSelector requires.
• No obvious resource shortage is apparent from a casual look at cluster capacity.

• A subtle mismatch between the nodeSelector's key/value and the node's actual label (case sensitivity, extra whitespace, a similar-but-different key name) that isn't obvious from a visual scan.
• The matching node(s) are cordoned (SchedulingDisabled), which prevents new pod scheduling regardless of label match.
• The matching node(s) have a taint the pod doesn't tolerate, which is an entirely independent constraint from nodeSelector matching.
• The apparent "free capacity" doesn't account for resource requests already reserved by other pods, DaemonSet overhead, or system-reserved resources the kubelet holds back.

Confirm the exact label match with a precise query, not a visual scan: kubectl get nodes -l <exact-key>=<exact-value> — if this returns no nodes despite kubectl get nodes --show-labels appearing to show a match, there's a subtle discrepancy (case, whitespace, a similar key) that a visual scan missed; this query is the authoritative check.

Check whether the matching nodes are cordoned: kubectl get nodes — the STATUS column showing Ready,SchedulingDisabled on the matching nodes explains the symptom entirely, independent of the label matching correctly.

Check the matching nodes' taints: kubectl describe node <node> for the Taints field — a nodeSelector match says nothing about taint tolerance, which is a completely separate gate; the pod needs a matching toleration in addition to satisfying the nodeSelector.

Check actual allocatable versus allocated resources on the matching nodes: kubectl describe node <node> shows Allocatable and the sum of already-requested resources under Non-terminated Pods — a node that looks like it has free capacity from kubectl top node's actual-usage view can still be fully committed from a requests perspective (which is what the scheduler actually uses for its decision), since usage and requests are different numbers.

Fix depends on which specific gate is actually blocking scheduling: correct the label if there's a genuine mismatch, uncordon the node if that's blocking it, add a matching toleration if a taint is the cause, or free up requested-resource headroom (or choose different/additional nodes) if the node is request-committed even though usage looks low. Confirm resolution by watching the pod transition out of Pending, and re-running the same kubectl get nodes -l query used in the investigation to verify the intended node is genuinely eligible.

• nodeSelector matching is necessary but not sufficient — cordoning and taints are separate, independent gates that a label match alone doesn't bypass.
• Use kubectl get nodes -l <key>=<value> as the authoritative match check, since a visual scan of --show-labels output can miss subtle discrepancies.
• "Free capacity" from actual usage (kubectl top node) and "free capacity" from requested-resource accounting (what the scheduler actually uses) are different numbers — a node can be request-committed while usage looks low.
• Investigate each gate (label, cordon, taint, resource accounting) independently rather than assuming the first plausible-looking cause is the actual one.

• Kubernetes: Assign Pods to Nodes
• Kubernetes: Node-pressure Eviction