Engineering · 6 MIN READ
Reach for a queue before you reach for cleverness
Most throughput problems are solved by separating work into durable units, not by optimising the work itself.
AB19 Innovations
In short
Splitting long-running work into idempotent, durable queued units gives resumability, backpressure and priority isolation — usually delivering more than optimising the processing step itself.
A job takes three hours and fails at the ninety-minute mark. The instinct is to make the processing faster. The better move is usually to make the failure cheap.
Durable units change the economics
Split the work into units small enough that losing one costs seconds, give each an idempotency key, and persist progress as each completes. A crash now costs a single unit rather than the whole run. Nothing about the processing got faster; the cost of failure collapsed.
Backpressure is not optional
Without it, a burst of input is accepted at whatever rate it arrives until memory runs out. With a queue, depth is observable and ingestion can slow down deliberately. Slower is a decision; out of memory at 3am is not.
Separate queues beat one clever scheduler
When two workloads share infrastructure and have different shapes — one latency-sensitive, one throughput-hungry — the temptation is to write a scheduler that balances them. Separate queues with their own worker pools and priorities are simpler, more predictable and far easier to reason about at two in the morning.
Idempotency is what makes retries safe. Without it, a retry is just a second chance to corrupt something.
None of this requires exotic infrastructure. A managed queue, workers that can be killed at any moment without consequence, and persisted progress covers the overwhelming majority of throughput problems people try to solve with cleverness.