Rebuilding image moderation at 600K images a day
Context
Every image shared on a K-12 communications platform (600K+ a day) passed through moderation before reaching classrooms. Machine flags landed in a human review queue staffed by a trust-and-safety team, inside a legacy admin tool that loaded every flagged image at once.
Problem
The vision vendor’s false positive rate kept humans reviewing images a machine should have cleared, and every unnecessary review was paid twice: once in hours, once in the emotional cost of exposure to potentially harmful content. The review tool itself timed out under load. The queue was growing faster than any hiring plan.
The decision
Attack both ends of the pipeline at once. Migrate classification from Google Vision to AWS Rekognition to cut the false positive rate at the source, and move human review out of the failing admin tool into a purpose built workflow: one image at a time, with more categories, flags, and platform control handed directly to the trust-and-safety team.
REJECTED: scaling the human side of the queue. More reviewers meant more cost and more people exposed to the worst content on the platform; the leverage was in sending humans fewer images, not hiring more humans.
Execution
I took the design from RFC through monitoring. The Rails monolith routes each image to a Rust service backed by an SQS worker, which calls Rekognition, compares labels against per category thresholds, and returns a verdict; flagged images have their links broken immediately while review is pending. Human review moved to a Retool interface that loads a single image with its flags, replacing a page that tried to render everything and routinely timed out. Review hours dropped by roughly half, and the trust-and-safety team gained direct control over categories and thresholds instead of filing engineering tickets.
Outcome
~48%
Fewer human review hours
600K+
Images moderated daily
1 at a time
Review UI: one image, not a wall of them
Takeaway
Machines should absorb the volume so humans can keep the judgment. Build the confidence threshold, not a bigger queue.