>DevOps Interview KB

How would you dedicate a set of nodes exclusively to one team's workloads, while still letting that team's pods run elsewhere too?

AdvancedArchitectureScenarioKubernetes7 min read

Short Answer

Taint the dedicated nodes (preventing everyone except pods with a matching toleration from being scheduled there) and give the requesting team's pods that toleration — this alone permits their pods to land there, but doesn't guarantee it, so pair it with node affinity on those same pods to actually attract them toward the dedicated nodes when appropriate, while leaving affinity soft (or entirely absent when they want the shared pool) so those same pods can still run on general-purpose nodes when that's the better choice.

Detailed Explanation

No single scheduling primitive achieves both "exclusive to this team" and "still flexible for this team" at once — exclusion and attraction are handled by different mechanisms in Kubernetes, and the design has to deliberately combine them rather than reach for whichever one seems most directly related to "dedicated nodes."

Requirements

  • Only the requesting team's pods (with explicit opt-in) should ever be schedulable on the dedicated node pool.
  • The requesting team's pods should still be able to run on the shared, general-purpose node pool when appropriate.
  • Other teams' pods must never be schedulable on the dedicated pool, even accidentally.

Architecture

A taint on the dedicated nodes is what actually excludes everyone else — this is the core exclusion mechanism: tainting the dedicated node pool (kubectl taint nodes <node> team=payments:NoSchedule, or configuring this at the node pool level in cloud infrastructure) means no pod schedules there unless it has a matching toleration — this single mechanism is what guarantees other teams' pods (which have no reason to carry this specific toleration) can never accidentally land there.

A toleration alone permits, but doesn't attract — affinity is what actually gets the team's pods there: giving the requesting team's pods a toleration for the team=payments taint makes the dedicated nodes eligible for their pods, but the scheduler has no particular reason to prefer those nodes over any other eligible one — without an accompanying affinity rule, the team's pods might still land on the shared pool most of the time purely by scheduling happenstance, undermining the point of having dedicated nodes at all.

Node affinity targeting a label on the dedicated nodes completes the pairing: labeling the dedicated nodes (team: payments) and adding a corresponding nodeAffinity rule to the team's pods (matching that label) means the scheduler now actively prefers or requires those nodes for this team's workloads — taint+toleration handles exclusion of everyone else, while label+affinity handles active placement of the intended team.

Use preferredDuringSchedulingIgnoredDuringExecution (soft) affinity, not required, to preserve the "can still run elsewhere" requirement: a required affinity rule would force every one of the team's pods onto the dedicated pool exclusively, which contradicts the stated requirement that they should still be able to run on the shared pool when appropriate — soft affinity expresses "prefer the dedicated pool" while still allowing the shared pool as a valid fallback, giving the team genuine flexibility rather than a rigid, all-or-nothing placement.

This taint+toleration+affinity combination is the standard, correct pattern precisely because using only one mechanism doesn't achieve both requirements simultaneously: taint+toleration alone (without affinity) permits but doesn't reliably attract; affinity alone (without a taint) attracts the intended team but doesn't exclude everyone else, since affinity has no bearing on what other pods without any affinity rule at all might still land there via normal scheduling. Only the combination achieves both "exclusive to this team" and "not mandatory for this team" simultaneously.

Trade-offs

This design requires coordinated maintenance of both the taint/toleration pair and the label/affinity pair — a node pool's label or taint changing without the corresponding update on the consuming team's pod specs (or vice versa) can silently break either the exclusion guarantee or the attraction behavior. This is a reasonable, standard operational cost for genuine workload isolation, but worth documenting clearly (which taint key/value and label a given dedicated pool uses) so it doesn't become tribal knowledge that's easy to get wrong when onboarding a new team or node pool.

Interview Follow-Up Questions

  • How would you audit a cluster to confirm no unintended pods have somehow acquired the toleration needed to land on a dedicated node pool?
  • How would you extend this design for multiple teams each needing their own dedicated node pool, without the taint/label scheme becoming unwieldy?
  • What would you do if the requesting team's workload occasionally needs a hard guarantee (required affinity) for specific critical pods, while most of their workload stays flexible?

Key Takeaways

  • Tainting the dedicated nodes is the actual exclusion mechanism — it's what guarantees other teams' pods can never accidentally land there.
  • A toleration alone only permits scheduling on the dedicated nodes; it doesn't attract the intended team's pods there without an accompanying affinity rule.
  • Soft (preferred) node affinity, not hard (required), preserves the requirement that the team's pods can still run on the shared pool when appropriate.
  • The full taint+toleration+affinity combination is necessary because any single piece alone only achieves one of the two requirements (exclusion or attraction), not both.

References

Last updated August 22, 2026 · Last reviewed August 22, 2026