The bug that survived 2.5 years of workarounds

Context

The data sync system pushed student information data from district systems into the product, via APIs and CSV uploads. It once had a dedicated team of five; by now it belonged to a team of three who also handled the escalation queue. For two and a half years the same families of sync bugs had been managed with scripts and known workarounds. And back to school, when thousands of districts sync fresh data, was months away.

Problem

When interdependent data failed partway through a sync (a user without a device, a membership without a valid class) it was saved as a “failed resource” to be corrected on the next incremental sync. Except corrections never landed. And the standard emergency fix, kicking off a full sync, quietly rewrote history: it skipped failed resources entirely, papering over each case while destroying the evidence. The code had no tests, so nobody had ever watched a failed resource live out its full lifecycle.

The decision

Stop treating the symptoms in parallel and force the root cause into view. I moved the full sync button behind my team’s ticket queue (accepting a short term load increase) so broken syncs stayed observable instead of being overwritten, then reproduced the failed resource lifecycle from scratch in staging with deliberately broken data.

REJECTED: another season of scripts. The workaround catalog was itself the reason the bug had survived 2.5 years: every full sync bought a week and burned the crime scene.

Execution

The staging reproduction found it: failed resources were saved with the school’s ID but queried on the next sync by the district’s ID, parent and child levels of the same hierarchy. They could never match; corrections were silently dropped. The fix was one query. The rollout was the hard part, because thousands of districts’ failed resources would suddenly start loading into live syncs. We phased it behind an environment variable, adding districts nightly for about a week while watching Datadog, built indexes concurrently off peak, purged stale failed resources (some five years old), and added a cron to expire them at six months so the table could never silently bloat again.

Outcome

78% → 98%

Rostered customer renewals

2.5 yrs

Lifetime of the bug, ended

~1 week

Phased rollout, no overload

Takeaway

Every workaround is evidence going stale. Take ownership of the tool that rewrites history, and the root cause surfaces.