What Broke First at Scale: The Synchronous Bottleneck Behind WebIntell
Load-testing the WebIntell ingestion pipeline exposed a problem that hadn’t shown up anywhere in normal operation: under 50 requests per minute, everything looked fine. At roughly 200 requests per minute, per-job processing time went from 800 milliseconds to over 45 seconds. Past that, jobs started timing out entirely.
What happened
The root cause was synchronous HTTP calls sitting inside an async FastAPI application. The requests library is blocking — it doesn’t yield control back to the event loop while waiting on a response. Every item in the pipeline blocked the entire event loop during its enrichment call, which meant work that should have processed in parallel was actually serialized. At low volume this is invisible: the event loop recovers fast enough that nothing looks wrong. At production volume, it collapses the whole pipeline into a single-file queue.
The fix
I had three options: swap the blocking client for an async one (correct, but doesn’t remove the architectural coupling), move the calls to a thread pool via asyncio.run_in_executor() (faster to ship, same problem underneath), or decouple enrichment from ingestion entirely using a queue and a separate worker pool. I chose the third — an architecture-level fix rather than a patch. The pipeline went from a single synchronous chain (receive → parse → enrich → prioritize → output) to async, queue-based stages: ingestion writes to a Redis queue, a dedicated enrichment worker pool consumes it and writes to a second queue, and prioritization and output run independently of enrichment’s response time. Post-fix throughput: 1,200 items per minute sustained, with ingestion latency staying under 800 milliseconds regardless of how long enrichment takes.
The lesson
Synchronous code inside an async framework doesn’t fail loudly — it degrades silently under load. It looks correct in every test that doesn’t push real volume through it, and then it serializes what was supposed to be parallel the moment production traffic arrives. Load testing isn’t optional validation for a system like this; it’s the only way the failure mode ever becomes visible before a customer does.