Imagine a one-lane bridge, and a thousand drivers who all know about the bridge. Every driver leaves a careful gap. Every driver obeys the limit. And since none of them can see the others, they all arrive at the bridge at once anyway. A quarter of them get turned away.

That was our SMS pipeline. The bridge is a carrier rule called 10DLC (ten-digit long code, the US carrier framework for application-sent SMS): one message-part per second, per phone number. The rule is enforced with a shrug — the API answers 200 OK, and the message quietly doesn't go. On a Tuesday afternoon, that shrug cost us a quarter of a bulk send. There was no retry and no error. The messages were simply gone.

What follows is everything we tried before admitting the drivers could never fix this: concurrency limits, wall-clock pacing, chunk sizing. Three patches, each sensible on its own, each defeated by the same one-lane bridge. Hotel-room announcements on our platform already cross their own bridge through a single toll booth — one queue, one patient consumer — and we spent those three patches hoping SMS would be the exception. Then we built ours.

Fifteen days after the toll booth went up, the production numbers: 14,214 messages attempted. 2,407 worker runs, every one of them finished. 3,616 sends got turned away and made it through on a later attempt. Five exhausted every retry and gave up — 0.035%. The bridge is still one lane. The toll booth turned out to be enough.

# A Thousand Private Pacers, One Neck

On a Tuesday afternoon, Slack filled with Pinpoint throttle alerts: 429 THROTTLED on per-recipient delivery results from bulk BatchAnnouncement sends. 429 is HTTP's Too Many Requests status. The carrier refused the message because we were sending faster than the origination number's rate limit allowed. THROTTLED is Pinpoint's per-recipient status label for exactly that refusal. Same origination number, alert count climbing.

The logs:

• Over a thousand batch-send invocations in a single hour
• One or two recipients per invocation — product fan-out (one SMS per future class registration change), not duplicate sends
• One origination number absorbing most of the traffic
• Roughly a quarter of messages throttled
• Throttled messages not retried; silent drop

Inference, labeled as such: likely a bulk discharge / move-out cascade — message copy like “You have unregistered yourself,” branch names pointing that way. Each registration change fired its own async invoke, individually valid. Under normal interactive load that pattern stays under 1 MPS. Under bulk fan-out it does not.

I had already shipped a stopgap — concurrency down, throttle detection in, delay scaffolding wired up, no new queue. At incident time the inter-message delay was off; we had toggled it during experiments. A4H (Alexa for Hospitality) already had the pattern I needed. Announcements to Alexa Echo devices in resident rooms — spoken in-unit messages, proactive cards on Echo Show — go through a shared Hospitality queue so Amazon's APIs are paced by a single consumer. Batch SMS was still fire-and-forget: each send launched its own worker with no shared drain.

Under bulk load, a thousand private pacers still lose to one neck. A4H had the durable intermediary sitting there the whole time.

# The 429 Inside the 200

Every SMS from our platform flows through a single AWS Pinpoint origination number. From day one our design work targeted the carrier ceiling, not the API ceiling — Pinpoint accepts ~4,000 sendMessages calls/sec, and we were never close to that.

The binding limit is carrier 10DLC throughput (~1 message-part/sec per origination number): "Will the carrier accept delivery?" It surfaces as 429 THROTTLED on the per-recipient result — not necessarily as a thrown exception. The word is borrowed from mechanics — a throttle controls flow by opening or restricting a passage — and computing kept only the restrictive side of it: slowing or limiting to avoid overload. Pinpoint can return HTTP 200 while individual recipients are throttled. That mismatch between API success and delivery outcome was a recurring triage trap.

Seven independent producers call Pinpoint through the same number with no shared pacing:

Batch announcements also have indirect upstream triggers — someone signing up for a class, an event getting deleted mid-schedule, a staff member sending a manual facility announcement — and all of them funnel into producer #1 announcements. Tightening announcements does nothing useful if OTP and check-in still share the neck — the drain has to be global, or you are just rearranging who overflows first.

# Concurrency and Pacing Inside One Invocation

"Concurrency limits how many sends run at once. Pacing limits how fast they complete. You need both — but only inside the invocation you control."

The stopgap in detail: tighten concurrency, detect throttles, and pace inside the handler — still fire-and-forget, still no global drain. After the initial throttling patch, we reduced in-function concurrency from 10 to 1, added throttle detection, and wired a finally block to guarantee minimum wall-clock time between sends within a single invocation. Concurrency and pacing together are the right combination for rate control inside one worker.

The gap is scope. When each SMS is its own fire-and-forget async invocation, the delay inside one invocation does not coordinate a thousand parallel ones. They do not share a finally block or a rate limiter instance.

A4H's announcementHelper.js — the helper module that paces those Echo announcement sends — already used two small npm libraries for exactly this pattern. Bottleneck is a rate limiter: instead of calling an API directly, you hand each call to limiter.schedule(), and the library enforces the knobs for you — a minimum gap between calls, a cap on how many run at once. It is the finally { delay } patch, generalized and done correctly. exponential-backoff is a retry helper: when a call fails or comes back throttled, it waits and tries again with a delay that doubles each attempt — 2s, 4s, 8s — instead of hammering a sender that just said not now. The stopgap I was assembling by hand — concurrency 1, a wall-clock gap, throttle detection — is what those two give you off the shelf. We should have aligned SMS with that earlier instead of reinventing a partial version.

The pattern that eventually mattered:

And for throttle detection:

Concurrency plus wall-clock pacing is the right combination inside one worker. Across parallel invocations you need a shared drain, which is the part I kept postponing.

# Pacing Experiments — Heuristics vs. Documentation

"The docs said 1000ms. I shipped 400ms anyway, because 'observed bursts seemed fine' is the kind of load-bearing assumption you get to test exactly once."

Second patch — still on the stopgap: turn the inter-message delay on. I set it to 400ms (~2.5/sec) because observed bursts seemed to tolerate it, and because a full second per send eats the Lambda 15-minute timeout on larger batches (roughly 900 sends at 1s each; fewer with multi-part SMS). The timer guarantees at least MIN_SEND_INTERVAL_MS of wall time per recipient regardless of API latency — elapsed subtracted, sleep the rest. A private rhythm. Same scope leak as before; 400ms failed the storm for the same reason a thousand private pacers always will.

Third patch: align with documented 1 MPS. Raise to 1000ms. Put Bottleneck in the new worker — same library A4H already used. Legacy paths updated for consistency.

Timeout math drove chunk size: at 1s per send, a 15-minute Lambda caps out around 900 single-part messages — less with retries, backoff, callback work, or multi-part texts. I chose chunk size 300 with margin. Production batches mostly landed 300–450; a few approached 800.

What pacing-only still left open:

• overlapping invokes — N workers hit the gateway alone
• uncoordinated producers — OTP, check-in, and friends share one number
• silent drop — throttled meant gone (pre-queue)
• no durability — async crash loses unsent phones
• retry unit — SQS redelivers a chunk, not each phone; in-loop 429 backoff is a different scope
• blast radius — one giant in-Lambda batch vs bounded chunks

Async invoke with InvocationType: Event is fire-and-forget. If the worker crashes mid-batch, those phone numbers are gone — nothing records them for retry. A queue gives at-least-once delivery, global serialization with a single consumer, chunking, and a dead-letter path. What I shipped still made the chunk the durable unit — one SQS message holds up to 300 phones, one Lambda owns the whole loop. Throttle backoff retries a single 429 inside that loop. Chunk size × interval + retry overhead still has to stay under the timeout ceiling; durability still needs a durable intermediary; global MPS still needs a single drain.

# From Eight Queues to One — The Great Slim-Down

The request for comments I'd circulated before building anything, the RFC, opened from the correct insight — the drain has to be global — though by then it was less insight than surrender. Three patches deep, there was no better patch than finally just accepting the global queue. Its centerpiece was sound: a global FIFO send queue — first in, first out, messages leave in the order they arrive — that every producer writes into and exactly one consumer drains at 1 MPS. That box survived every draft. The other seven queues grew out of reviewing it. Each pass over the diagram asked what about, and every what about became a new box.

None of it felt like over-engineering at the time. Every box answered a question I could defend in review, so I kept answering.

What about the seven producers? They do not arrive with the same semantics. Bulk announcements are fire-eventually; forgot-password codes are fire-now; check-in greetings are scheduled; the other four are low-volume transactional drips. So the RFC grouped them into four track queues — one per message class, each with its own dead-letter queue — so retry policy, alarms, and blast radius could be tuned per class. A poisoned announcements track would never so much as touch the OTP track.

What about ingress? No producer should embed routing logic. The next engineer adding producer #8 should be able to enqueue one message and trust the pipeline to sort it. So a dispatch queue feeding a dispatcher worker: a single uniform entry point that owns the lane decision, keeping producers dumb and routing centralized.

What about latency? A login screen is a person staring at a blank code field — no spinner, no status, no way to tell slow from never. One 300-phone chunk at 1 MPS is five minutes of queue depth; behind a real bulk blast, an OTP code might as well arrive expired. So the priority OTP lane — a separate queue drained ahead of the FIFO for anything time-sensitive.

What about outcomes? Delivery classification and Slack alerts do not belong on the send path; a slow alert webhook should never hold the 1 MPS cadence hostage. So a callback queue with a callback worker, asynchronous by construction.

Matching workers for each. Every queue got a consumer, every queue a DLQ, every DLQ an alarm. Twelve SQS resources if you count the dead letters — eight if you don't.

Read in order, the flow is compelling: every addition is locally justified, every box answers a question the previous box raised, and the diagram only ever grows by one box at a time. Read against the incident, it is embarrassing. The storm needed four things — serialize every producer, make work durable, bound the blast radius with chunks, dead-letter the poison — and the RFC delivered those, then kept designing: imagined producer #8, imagined per-class SLAs, imagined scale, flexibility with no ticket attached. Even the FIFO did not survive review once I admitted a single consumer serializes on its own; ordering was never the requirement, exclusivity was. A post office with four sorting rooms for a building that needed one mailbox.

Then I deleted most of it:

• OTP priority lane — forgot-password almost never waits behind a bulk blast. OTP stays on the direct path (sync bypass).
• Dispatch queue — the existing handler becomes the splitter.
• Callback queue — async invoke the callback worker instead.

What shipped: one send queue, one worker, one callback. That is the migration. OTP and the other low-volume producers stay direct by design. I also deferred a shared token bucket in sendSmsMessage (DynamoDB or Redis counter every caller checks before sending) — synchronous under load, contended across parallel Lambdas, no built-in retry. Mostly, though, it was redundant: a single serial consumer is a global rate limiter, and the only sends it would not pace — the low-volume direct producers — never approach 1 MPS on their own. The global queue was the durable fix A4H already proved.

I deployed the slim version and tore the over-engineered queues out of the stack. The minimum pipeline that closes the global-drain gap was enough.

# The Splitter, the Worker, and the Callback

The splitter chunks phone numbers at 300 or fewer, prefetches gateway configuration once, and enqueues to the send queue. Chunks and prefetches. The worker is the only thing allowed to talk to Pinpoint.

The send queue — standard queue, batch size one, worker reserved concurrency one. Serializes work across all facilities globally — the single drain.

The paced worker uses the same two npm libraries A4H hospitality already relied on. Bottleneck enforces the 1 MPS gap between phones (minTime: 1000, maxConcurrent: 1). exponential-backoff retries only on 429 THROTTLED — up to 5 attempts, starting at 2s, max 256s. Outcomes land in three buckets: success, hard throttle (retries exhausted), and other errors.

SQS retries the chunk on timeout. Backoff retries one phone on 429 only — different scopes, easy to blur when you are tired.

The callback worker — async-invoked, off the critical path. Slack only when something dies. Silent on all-success. Each piece fails in its own lane; a Slack outage does not block SMS delivery.

# One Batch of 607, End to End

The pipeline above is abstract until you run one batch through it. Take a blast of 607 numbers — one facility, one move-out cascade.

One noun needs untangling first, because message works both sides of this street: the queue carries messages, Pinpoint sends messages, and they are not the same thing. The batch is 607 phones. The splitter divides by 300 — 300, 300, 7 — which makes three chunks. Each chunk is wrapped in one SQS message, the queue envelope. Inside the worker, each chunk unwraps into individual SMS — one text per phone.

Each envelope carries its prefetched gateway configuration: twelve reads for the whole batch, not the 2,428 that per-recipient loading would cost. That collapse is its own section, below.

Then the single drain takes over:

Ten minutes is the best case — empty queue besides these three, no throttling. The drain is global, so another facility's chunk can sit between your chunk 2 and chunk 3. And the send queue is standard, not FIFO: with batch size one and concurrency one the chunks almost always land in order, but nothing promises it. Ordering was never the requirement; exclusivity was.

The example earns its keep mid-loop. Say phone 114 comes back 429 THROTTLED. Backoff owns that phone: wait 2s, retry — still throttled — wait 4s, retry — through. Six seconds of sleep, then the loop moves to phone 115. The chunk never noticed. The queue did: maxConcurrent: 1 means a sleeping phone is a paused pipeline, and every facility queued behind it waits too.

A phone that exhausts all five attempts — 2s, 4s, 8s, 16s; thirty seconds of waiting — lands in hard429[] and the loop moves on. SQS retries nothing here. The envelope is still on its single delivery, waiting — deleted only when the loop ends.

Three callbacks fire for this batch, not one — a chunk boundary each. All 607 landed, all three stay silent. Slack speaks only when something dies.

And the arithmetic has teeth. A clean 300-phone chunk drains in ~300s against a 900s function timeout — 600s of slack, and every hard-throttled phone burns 30 of it. Twenty hard 429s in one chunk, under seven percent of it, and the budget is gone: 300s of pacing plus 600s of waiting lands exactly at the wire. Throttle harder than that and the worker dies mid-loop with a tail of phones unsent. What happens to that tail is the next section.

# Timeout, Redelivery, and the Dead Letter Queue

The slim pipeline — splitter → send queue → paced worker → async callback — routes batch announcements through the global drain. OTP and the other producers still call Pinpoint directly; that is intentional.

The durable unit is the chunk, not each phone. One SQS message carries up to 300 numbers; one Lambda runs a serial loop over all of them. There is no per-recipient queue record — hard429 and other errors land in the callback payload, not back on the queue.

My first instinct was catch-everything-in-code: if the worker handles all errors and acknowledges only after the entire chunk finishes, maybe a DLQ is unnecessary. Walking through a timeout mid-chunk closed that idea before I shipped.

A function timeout kills the process before any catch runs. With no DLQ and no maxReceiveCount cap, the queue message becomes visible again and can redrive indefinitely. Each redelivery re-runs the entire phone list — including numbers already sent.

So the migration includes a dead-letter queue: maxReceiveCount: 1, visibility timeout greater than the function timeout. On timeout, the chunk lands in DLQ instead of looping forever.

The DLQ does not erase the half-sent problem. If the worker times out after phones 1–150 of chunk 1, those 150 already went out. The queue has no memory of per-phone progress — phones 151–300 sit in DLQ with the rest of the message. Loss for the unsent tail. Blind replay of that DLQ message would text 1–150 again. Per-recipient queue retry was never in scope for this migration; I traded infinite duplicates for a stuck tail, and I still have to be careful with my hands on the DLQ.

# Prefetch: Four Reads Per Recipient to Four Once

In version one, every recipient triggered four database reads for gateway configuration. 300 recipients: 1,200 reads. That path had already bitten us before the storm.

A two-recipient registration batch failed one of two sends. Same message, two phone numbers, one async CustomSmsHandler invocation. The sequence from timestamps:

1. The API handler fires with a warm database connection, async-invokes CustomSmsHandler, returns 200.
2. A fresh worker container cold-starts (~1.7s init, plus VPC ENI attach — Lambda wiring a network interface into our private network so it can reach Mongo).
3. First recipient: new Mongo client, four parallel DbParameters.Get calls for Pinpoint config. TLS/pool instability on the cold path (MongoNetworkError, pool cleared). Error swallowed upstream.
4. Reported error: "No Parameters found for the given filter" — sends triage to the config table instead of the connection layer.
5. Second recipient, ~650ms later: pool recovered, config loads, Pinpoint SUCCESSFUL.

# The Production Audit

Before the edges — because it mostly worked — the production numbers. Window: 2026-07-29 → 2026-08-14 (15 days).

Worker batch complete:

• 2,407 invocations — START / END / REPORT / batch complete all 2,407; gap 0
• 14,214 phones attempted
• 14,209 worker-reported success (99.965%)
• 5 hard429 — every worker-bucket failure
• 0 otherErrors, 0 timeouts

Pinpoint DeliveryStatus on Message sent! attempt events:

• 13,438 SUCCESSFUL
• 771 PERMANENT_FAILURE (opt-outs and similar)
• 3,616 THROTTLED — retries; do not add them to SUCCESSFUL
• 17,825 events total

Worker success is exactly 771 higher than Pinpoint SUCCESSFUL: 14,209 = 13,438 + 771. The whole classifier is isThrottled — anything that isn't a throttle counts as a win, so the 771 opt-out PERMANENT_FAILUREs landed in success, otherErrors stayed 0, and Slack never fired. The lie is not the count; 14,209 loops really did finish. The lie is the name. Success vouched for a delivery it never checked — it prints the same 99.965% whether the 771 are honored opt-outs, dead numbers, or a failure nobody has named yet. Real accept rate: 13,438 / 14,214 = 94.54%.

Counted honestly, the fifteen days are a win three ways. The loop closed — 2,407 / 2,407 invocations, gap 0, zero timeouts. The opt-out contract held — the 771 refusals are Pinpoint refusing opt-outs and the like, and not one message went. And the throttles recovered: Pinpoint said not now 3,616 times, backoff asked again, and five phones of 14,214 ran out of asks (0.035%). The storm hit the same 1 MPS ceiling and dropped a quarter of the batch, silently, 200 OK all the way down. Same ceiling, different answer to not now. Among people who had asked to be texted: 13,438 / 13,443 = 99.96% accepted.

A shorter or paged Insights view can undercount (stream count, window, UI paging). The log truth for this window is 2,407 invocations. These insights queries were also dirt cheap (~16 MB scanned each). Typical distribution still holds: lots of tiny batches, a few whales.

What matters is throttles recovered, not throttles hit. The ceiling has not moved — 1 MPS bit the storm and bit this window alike — and it will bite again. The difference is the after: the storm's throttles became silent drops; these 3,616 became retry attempts, and all but five phones got their message. Throttle count measures the ceiling. Recovery count measures the pipeline. Watch the second one.

# When to Patch, When to Build, When to Delete

"Patch when the fix's scope matches the failure's scope. Design for incidents you can evidence: scope, data, frequency, logs. Count only what you can name."

The RFC grew one box at a time because every what about could defend itself in review. These are the questions I'll ask instead, in the order that would have caught me:

Does the fix's scope match the failure's scope? Each of the three patches was correct inside its own invocation. The failure lived across a thousand invocations at once, so the patches could only lose. Concurrency and pacing are the right knobs when you own the worker. The storm was a thousand workers nobody owned. When a thousand drivers cannot see each other, the fix is the thing they share — build the toll booth. Ship a stopgap once, to buy time to think. By its third revision, a stopgap is a way of avoiding the decision.

Is this box for the incident, or for an imagination? Define the word first, because the rubric leans on it. An incident is something bad that left evidence. The evidence has an anatomy:

• Scope — which producers, which shared resource, which window. Here: seven producers, one origination number, one Tuesday hour.
• Data — counts per outcome, queryable. 1,204 batch invocations in the storm's worst hour, 1–2 recipients each, roughly a quarter throttled. After the fix: 2,407 worker runs and 14,214 phones in fifteen days, ~160 runs a day, lots of tiny batches and a few whales.
• Frequency — a rate counted over a window. Bulk sends are scheduled human operations: move-out cascades, schedule changes, staff announcements. The ceiling is documented at 1 MPS. Seven producers share one neck, so the exposure is permanent.
• Logs — where the failure surfaces: Slack alerts, per-recipient delivery results. The only place a 429 inside a 200 shows its face.

Then score it like a risk assessment, three axes, each with evidence:

• Severity — a quarter of a batch, silent, no retry. Unrecoverable loss.
• Occurrence — the trigger is routine and the ceiling does not move. Expect it again.
• Detection — weak. 200 OK all the way down. A failure you cannot see keeps happening.

Score every proposed box on the same axes. The storm scored high on all three, with receipts, and everything that shipped answers it. The priority OTP lane had real severity — a person staring at a blank code field — and near-zero occurrence. The logs showed forgot-password almost never waits behind bulk, so the lane died. Producer #8 scored zero on occurrence because there is no producer #8. Per-class SLAs and no-ticket scale failed the same question. The DLQ shipped despite zero occurrences in 2,407 runs, and the reason is worth memorizing: its trigger is a documented mechanism (the fifteen-minute function timeout is as real as the 1 MPS ceiling), its consequence is catastrophic (redrive forever, duplicate SMS), and its guard costs almost nothing. A cheap guard against a documented mechanism is sound engineering. An expensive guard against a scenario nobody can evidence is over-engineering. The eight queues felt defensible in review because every what about argued well. Arguments are cheap. Incidents bring receipts. Put the evidence next to the diagram and ask, for every box, which line it would have prevented. YAGNI is accountability for that answer. The FIFO died on this test. One consumer already gives exclusivity. Ordering was extra.

One note sits underneath all of them. The deliverable is a resident getting the text they asked for, and us knowing, honestly, that it went. The queue, the diagram, the migration: means. Reliability is the product. Trust is the interface. Complexity is a cost — sometimes necessary, invisible to the resident either way. The bridge is still one lane. The ceiling bit 3,616 times this window and will bite again. In the storm, every bite was a silent drop. In this window, every bite was a retry. That difference is the deliverable. So: patch when the scope matches. Stop when the third patch patches the second. Design for incidents with evidence. Delete the boxes that answer nothing. Count only what you can name. That is the whole rubric. The rest is Tuesday.