How would you distinguish a genuine memory leak from a legitimately growing in-memory cache, using only Kubernetes-level metrics?

Look for a plateau: a legitimate cache growing toward some bounded size (traffic volume, a configured max-cache-size, working-set size) will level off and then stay roughly flat, cycling within a stable range; a genuine leak keeps climbing indefinitely with no plateau, eventually hitting the container's memory limit and getting OOMKilled, then repeating the same unbounded climb after each restart. Correlating the growth against restarts, traffic volume, and time is the main Kubernetes-level signal available without direct application profiling.

Without application-level profiling, the available signal is entirely behavioral — how the memory trend shapes over time and how it relates to restarts and traffic — rather than which specific code path is responsible.

• Container memory usage (container_memory_working_set_bytes in Prometheus/cAdvisor terms) trends upward over time.
• Uncertainty about whether the trend will plateau naturally or continue until an OOMKill occurs.

• A genuine application-level memory leak — objects retained longer than intended, an unbounded collection that's never cleared.
• A legitimately growing, eventually-bounded in-memory cache approaching its working-set size or a configured maximum.
• Traffic growth genuinely increasing the application's real memory footprint (more concurrent connections, more cached data proportional to more distinct users/keys) without either being a "leak" in the buggy sense.

1. Look at the memory graph's shape over a longer time window than the immediate concern — does growth continue linearly/indefinitely, or does it visibly level off into a plateau after some period?
2. Check whether memory usage correlates with a bounded input — if a cache is configured with a max size or TTL-based eviction, check whether the memory trend's plateau roughly matches when that bound would be reached (e.g. cache reaches max entries, then memory stabilizes).
3. Correlate the growth against container restarts: kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].restartCount}' combined with the memory graph — a leak shows the same climbing pattern resetting and repeating after each OOMKill-triggered restart; a legitimate cache reaching a stable plateau wouldn't show this sawtooth-then-OOM pattern repeating.
4. Correlate growth against traffic volume metrics (request rate, active connections) over the same window — memory growth that tracks traffic growth and stabilizes when traffic stabilizes points toward legitimate, traffic-proportional usage (cache or otherwise) rather than an unconditional leak.
5. If available, compare memory usage against the same period in a previous, similar traffic cycle (day-over-day or week-over-week) — a leak that resets on restart but reaccumulates at a consistent rate independent of traffic pattern is a stronger leak signal than growth that varies with traffic.

kubectl top pod <pod> --containers

kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].restartCount}'

If the pattern indicates a genuine leak (unbounded growth, repeating sawtooth-then-OOM cycles uncorrelated with traffic), the fix requires application-level investigation (heap profiling, object retention analysis) beyond what Kubernetes-level metrics alone can diagnose — Kubernetes metrics can strongly suggest a leak exists and roughly how fast it accumulates, but confirming the actual leaking code path requires application-level tooling. If the pattern indicates a legitimate, bounded cache, the appropriate response is usually just right-sizing the container's memory limits/requests to comfortably accommodate the cache's actual plateau size, with some margin, rather than treating the growth itself as a problem to fix.

• Set memory limits with enough margin above the expected legitimate working set (including cache) that normal, bounded growth doesn't trigger unnecessary OOMKills.
• Add application-level memory metrics (heap size, cache entry count) alongside container-level metrics where possible, giving a more direct signal than inferring from container memory alone.
• Alert specifically on repeating OOMKill-restart cycles (a much stronger leak signal than a single upward trend) rather than just on high memory usage in isolation.

• A legitimate cache's memory growth plateaus and stabilizes; a genuine leak keeps climbing indefinitely with no plateau.
• Correlating growth against restart history is a strong signal — a repeating sawtooth-then-OOM pattern uncorrelated with traffic points toward a leak.
• Correlating growth against traffic volume distinguishes legitimate traffic-proportional usage from an unconditional leak.
• Kubernetes-level metrics can strongly suggest a leak exists but confirming the specific leaking code requires application-level profiling beyond what container metrics alone provide.

• Kubernetes Docs: Resource Management for Pods and Containers
• Google SRE Workbook: Monitoring Distributed Systems