Crash-free for the people who sign the renewals

Context

District administrators were the platform’s renewal decision makers, and nobody was measuring their experience specifically. Server dashboards were green while roughly one in five admins hit a crash, an error, or a page load of 2 seconds or more in any given week.

Problem

Error tracking treated all failures as equal, so engineering attention followed log volume instead of business impact. The admins whose experience drove renewals were a small share of traffic, which meant their pain was statistically invisible in aggregate metrics.

The decision

Define the metric before touching the code: an admin crash-free rate (the percentage of admin users who experienced no crash, error, or slow page in the trailing week) with a target of driving it from ~20% affected to ~5%. Rollbar categorized the errors; each one was then triaged by a simple question: does this need to be an error at all, or should it be a toast?

REJECTED: whack a mole on the loudest errors. Log frequency and admin pain turned out to be different lists, and only one of them renews contracts.

Execution

Low hanging fruit first: errors that shouldn’t have been errors became inline feedback. Then the latency tail, traced to the highest traffic admin endpoint, where class listings loaded everything eagerly. Strategic caching, corrected index ordering, pagination, and async loading below the fold cut the worst page loads, and cron jobs retired the old data bloating the hot tables. The metric reframed the conversation: performance work stopped being a backlog category and became a renewal lever tracked week over week, and the program landed at 3% of admins affected, beating its own ~5% target.

Outcome

~20% → 3%

Admins hitting a crash or slow page weekly

~5%

Original target, beaten

Per admin

Measured by person affected, not requests served

Takeaway

Measure the experience of the people who decide to renew, then fix errors in order of who they hurt, not how loudly they log.