A Cloud Function has been silently failing for days with no alert firing — what's typically missing in the observability setup, and how do you fix it?

Almost always, this means no alerting policy was ever configured on the function's error rate/execution count metrics in Cloud Monitoring — the error was faithfully logged the whole time, but logging alone doesn't notify anyone; it just makes the information available if someone happens to look. The fix is setting up an actual alerting policy on Cloud Functions' built-in execution-count-by-status metrics, so an elevated error rate proactively pages someone instead of waiting for a routine review to stumble onto it.

Cloud Logging capturing an error and someone being notified of that error are two entirely separate things — the function did exactly what it was supposed to do (log the failure), and the gap is specifically in the missing alerting layer on top of that logging.

• A function has been failing on every invocation for an extended period, discovered only via manual review rather than an alert.
• The error itself is clearly visible in Cloud Logging once specifically searched for.
• No PagerDuty/notification/alert fired at any point during the failure window.

• No Cloud Monitoring alerting policy exists at all for this function's error rate or execution status.
• An alerting policy exists but is scoped to a different function, or uses a threshold/condition that doesn't actually match this failure pattern (e.g., only alerting on total execution count dropping to zero, which wouldn't catch "every execution runs but fails").
• The function's error handling swallows the exception in a way that still logs it but doesn't propagate it as a genuine execution failure that Cloud Functions' own status metrics would count as an error.

Check whether any alerting policy exists for this function at all: reviewing Cloud Monitoring's alerting policies for anything referencing this function's name or its metrics directly confirms whether the gap is a complete absence of alerting, versus a misconfigured one that exists but doesn't actually fire for this failure pattern.

Confirm the function's executions are actually being recorded as errors in Cloud Functions' own metrics: cloudfunctions.googleapis.com/function/execution_count broken down by status in Cloud Monitoring shows whether these failing executions are correctly counted as errors — if the function's own error handling is catching and logging the exception without re-throwing or otherwise signaling failure, Cloud Functions itself might record these as successful executions despite the logged error, which would explain why even a correctly-configured error-rate alert wouldn't have caught it.

Check log-based alerting as an alternative or complementary mechanism: if the failure pattern is better identified from a specific log message/pattern than from the execution-status metric (for instance, a specific error type logged within an otherwise "successful" execution), a log-based alerting policy (matching that specific log pattern) is the more direct mechanism, separate from execution-count-based alerting.

Set up a Cloud Monitoring alerting policy on the function's execution count broken down by status, alerting when the error-status count (or error rate) exceeds a reasonable threshold over a reasonable window — this is the direct, standard fix for "failures happen but nobody's notified." If the function's error handling was swallowing exceptions without correctly signaling failure to Cloud Functions' own status tracking, fix that first (re-throw, or otherwise ensure a genuine failure is recorded as such), since an alert on execution status is only as good as the underlying status actually being accurate. Confirm the fix by deliberately triggering a test failure and verifying the alert actually fires.

• Logging an error and alerting on it are separate concerns — a function can faithfully log every failure while nobody is ever notified, if no alerting policy exists on top of that logging.
• Check Cloud Functions' own execution-count-by-status metric specifically, since error handling that swallows exceptions without properly signaling failure can cause genuinely-failing executions to be recorded as successful.
• Log-based alerting is a complementary mechanism for failure patterns better identified from specific log content than from the aggregate execution-status metric.
• Always test that a newly-configured alert actually fires, by deliberately triggering the failure condition, rather than assuming the configuration is correct.

• Google Cloud: Cloud Functions monitoring
• Google Cloud: Alerting overview