Safety alerting pipeline redesign

Context

A safety critical alerting path (events in, human notifications out) had grown by accretion over four years. Every incident review ended with another patch on the same queue.

Problem

Under load spikes, alert latency went from seconds to minutes, precisely when alerts mattered most. Duplicate suppression and priority ordering fought each other in one consumer, and no one could state the delivery guarantee.

The decision

Split the path into two lanes with different guarantees: a fast lane that delivers possibly duplicate critical alerts in bounded time, and a reconciliation lane that cleans up afterward. Duplicates became acceptable; lateness did not.

REJECTED: a full event sourcing rewrite. Correct on paper, but 6+ months of migration risk on a live safety system; the two lane split shipped in 3 weeks and was reversible at every step.

Execution

Ran the new lanes in shadow against production traffic for two weeks, diffing outputs nightly, then cut over one alert class at a time.

The same split governed how alerts were allowed to die. Routine lateness alerts could close themselves (a still valid check watched trip lifecycle events and retired them when the world moved on), but safety alerts never resolved on their own: each required a human close, with a reason and an audit of whether the alert fired accurately. A per trip hierarchy let high signal alerts suppress lower signal noise, and routing carried the trip’s context (accessibility trips, ride assistants, drivers still in their first fifty trips) into a weighted capacity assignment engine that rolled alerts over between specialist teams without ever looping.

Outcome

~90s

P99 alert latency

~90 min/day

Saved per employee

Zero

Migration downtime

Takeaway

Name the guarantee you're selling. Then design two systems: one that keeps it, one that cleans up after it.