Safety-alerting pipeline redesign

Context

A safety-critical alerting path — events in, human notifications out — had grown by accretion over four years. Every incident review ended with another patch on the same queue.

Problem

Under load spikes, alert latency went from seconds to minutes — precisely when alerts mattered most. Duplicate suppression and priority ordering fought each other in one consumer, and no one could state the delivery guarantee.

The decision

Split the path into two lanes with different guarantees: a fast lane that delivers possibly-duplicate critical alerts in bounded time, and a reconciliation lane that cleans up afterward. Duplicates became acceptable; lateness did not.

REJECTED — a full event-sourcing rewrite. Correct on paper, but 6+ months of migration risk on a live safety system; the two-lane split shipped in [duration — TBD] and was reversible at every step.

Execution

Shadow-ran the new lanes against production traffic for two weeks, diffing outputs nightly, then cut over one alert class at a time.

architecture diagram — [asset TBD]

Outcome

~90s

P99 alert latency

~90 min/day

Saved per employee

Zero

Migration downtime

Takeaway

Name the guarantee you're selling. Then design two systems: one that keeps it, one that cleans up after it.