System design, one failure at a time

A customer sees a timeout, but their payment succeeded. A post is deleted, but its ID is still in a million feeds. A worker restarts, picks up its last job, and sends the same email again.

These problems start with a gap between what happened and what another part of the system knows. Drawing a queue or adding a retry does not close that gap by itself. We have to decide what gets recorded, what can repeat, and how recovery finds the truth.

The scenarios below work through application architectures, a social feed, and the components that connect them. Each asks the same questions: what failed, what survived, what can the user safely do next, and what evidence would prove recovery? The numbers are stated design assumptions, not production measurements. Run the order pipeline, or use the diagrams to inspect each scenario’s normal flow, failure points, and recovery. The diagrams explain design decisions; they do not execute real infrastructure.

Explore the scenarios

Choose a scenario, then move between Normal flow, Failure, and Recovery. Select a component or connection to inspect its role and the evidence needed for recovery. Pan or zoom to follow the route; Escape releases canvas scrolling.

Break the pipeline

Start with Worker outage and press Run. Two workers stop while customers keep submitting orders. Click a service to inspect its state and recent events. Then choose Replay with fixes to compare the same workload and failure schedule.

You can also slow the database or inject a duplicate delivery during a run. Those actions are recorded at their simulation times and replayed in the comparison. Each run is limited to 30 simulated seconds.

A timeout does not undo a write

Choose Slow database. A worker gives up waiting after 1.2 seconds, but the database takes 1.8 seconds to commit during the slowdown. The request has timed out from the worker's perspective; the database is still working.

That is why the playground separates persisted from acknowledged. The first counts distinct orders with a database record. The second counts orders whose worker received a response before its attempt ended. Extra writes count additional records for an already-persisted order.

A timeout describes uncertainty, not rollback. Amazon's guide to timeouts, retries, and backoff discusses this distinction and why retrying a side effect requires care. In this model, pending commits deliberately survive both a worker timeout and a worker crash.

A useful question for a real API is: after a connection failure, can I safely ask again using the same operation identity?

Give the operation a stable identity

Choose Duplicate delivery. Every fourth order enters the queue twice, with the same order ID. Without protection, each delivery can produce a database write. With protection, the second delivery gets the existing result.

Duplicate messages are a documented possibility for systems such as Amazon SQS standard queues. The consumer must account for that possibility instead of treating receipt as proof of new work.

For a database-only operation, a unique key can make the write atomic. This simplified PostgreSQL example assumes order_id has a unique constraint:

INSERT INTO orders (order_id, amount_cents)
VALUES ($1, $2)
ON CONFLICT (order_id) DO NOTHING;

The important part is the constraint, not a prior SELECT. Two workers can both observe “missing” and then race to insert. The database must arbitrate the conflict. The application should also reject reuse of an operation ID with a different payload, and return the existing result when the request is a repeat.

A local transaction does not make an external payment or email atomic. For a remote side effect, carry a stable key to a provider that supports it, and understand its retention and retry semantics. Stripe's idempotent-request documentation is one concrete example. This playground models only database writes; it does not simulate payments.

Retry deliberately, then stop

A retry needs an identity, a delay, and a limit. In the protected run, a failed attempt returns to the queue after increasing delays, with a small per-order offset. The simulation stops retrying after four attempts and records a dead-letter delivery.

The model uses stable jitter so comparisons are repeatable. Its delay calculation is:

const jitter = (orderId * 137 + attempt * 71) % 300;
const delayMs = 400 * 2 ** (attempt - 1) + jitter;

The engine rounds that delay up to its next 100ms tick. This is teaching code, not a recommended production retry configuration. Choose real deadlines, attempt limits, and jitter from the behavior and capacity of the dependency.

A delivery reaching dead letter is different from proving the operation never happened. In the slow-database scenario, a late commit can exist even when its worker has given up. The event log preserves that distinction.

The retry policy does not repair invalid input, create capacity, or guarantee eventual success. Keep the workers down for the entire run and work remains unresolved. The comparison reports that work rather than calling it completed.

The database committed. The event did not arrive.

Scenario: the API saves a post and then publishes a fanout event. The process crashes between those operations. The post exists, but followers never receive it. Reversing the order creates the opposite problem: an event for a transaction that later rolls back.

In a transactional outbox design, the domain change, the request's saved result, and a pending event commit together in one database transaction. A relay publishes pending events later. This closes the missing-event gap; it still permits duplicate publication. See the transactional outbox pattern.

AppLab's outbox adds another failure to this sequence: a relay claims an event, pauses long enough for its lease to expire, then resumes after another relay takes over. Each claim has a new token, and marking completion checks that token. The old claimant cannot overwrite the new claimant's database progress. It can still have published to the broker, so consumer deduplication remains necessary.

Keep attempts and failed events for inspection. An exhausted event needs explicit recovery, not silent deletion. A broker acknowledgment proves acceptance at that boundary; it does not prove a downstream order was fulfilled. An inbox transaction also cannot atomically cover a remote payment or email.

One worker, every subscriber, or a lost history?

Scenario: an order event needs fulfillment, analytics, and a notification. Putting three workers in one competing-consumer group does not deliver one copy to each responsibility. They compete for the same work.

Use a work queue when one worker should handle a delivery. Use independent durable subscriptions when several consumers need their own progress. Workers can then compete within each subscription. The fanout worker itself is just a background program; SQS is a queue, while Lambda or an ECS/Fargate container can run that program. Lambda's SQS integration documents the need to handle repeated processing.

Temporary failure, exhausted retries, and cancellation are different outcomes. Retry a transient error within a budget. Isolate an incompatible or invalid message so it cannot block an entire partition forever. Cancellation asks running work to stop; it does not roll back an effect already committed. A stalled worker can also cause redelivery outside its ordinary exception-retry path.

Scenario: analytics is offline while other subscribers continue. Retention eventually removes events that analytics has not read. In AppLab's durable-subscription design, this becomes an explicit history gap and delivery stops for that subscription. Advancing to the oldest retained event is an acknowledgment of loss, not successful catch-up. Rebuild from another authoritative source if the product requires the missing history.

An inspection replay should not secretly advance live progress. A redrive is a new delivery with its original cause recorded. Keep those actions distinct, and measure oldest unprocessed age, retry exhaustion, history gaps, and duplicate effects separately.

A queue buys time, not infinite capacity

The queue has eight waiting slots and there are two concurrent workers. A worker outage reduces service capacity to zero while incoming orders continue. Once the buffer fills, something must give.

The baseline rejects overflow. The protected run defers admission at the API and feeds waiting deliveries into the queue as slots become available. Watch the queue-depth trace and the API's deferred count during recovery.

This demonstration has a fixed workload of twenty orders, so deferred work is bounded by construction. A production API needs an explicit limit and expiry for its own waiting work. Depending on the contract, that may mean rejecting with a retry hint, applying an admission budget, or returning an accepted job ID after durable enqueueing.

Moving an unlimited backlog from a queue into application memory would only relocate the problem. The meaningful outcome is controlled load and a clear response to callers, not a graph that never turns red.

Calculate the pressure before adding servers

Feed reads: suppose Home receives 70,000 requests per second and each page needs 30 post bodies. That creates 2.1 million logical lookups per second. A 90% body-cache hit rate leaves 210,000 misses before batching and request coalescing. Those are not automatically 210,000 network round trips: a batch query changes the number of calls, but the records still have to be fetched and processed.

Fanout backlog: arrivals of 120,000 recipient updates per second against capacity of 100,000 grow the queue by 20,000 per second, or 1.2 million per minute. Raising completion capacity to 160,000 leaves 40,000 per second for catch-up. A 12-million-update backlog then takes five minutes to drain, assuming arrivals and downstream capacity stay constant. Scaling workers without checking the feed store can make recovery slower.

Storage: at one billion posts per day and 150 bytes per record, 90 days is 13.5 TB of logical data, using decimal units. Adding a hypothetical 30% index allowance gives 17.55 TB per complete copy. A primary, two replicas, and one full backup total 70.2 TB under these simplified assumptions. Thirty days instead of ninety reduces that total to 23.4 TB; it does not reduce new write traffic. Real compression, log retention, backup strategy, and index layouts need measurement.

Media: 100 million uploads at 50 KB each produce 5 TB of uploaded bytes. One billion downloads of the same-sized objects deliver 50 TB to readers. Reducing CDN byte-hit rate from 90% to 60% raises origin traffic from 5 TB to 20 TB. Upload volume did not change. Request-hit rate is insufficient when object sizes differ.

These calculations identify pressure points, not an instance count. Check CPU, memory, connections, disk, and request cost under load. Google SRE's overload guidance explains why raw request rate is an incomplete capacity signal and why graceful degradation still needs limits.

The same failure in four application shapes

Modular monolith: a successful save followed by a lost background job still needs a durable handoff. Keeping modules in one process simplifies deployment and local transactions; it does not make the process crash-proof. Use ownership boundaries inside the application, observe expensive queries and jobs, and extract a service when independent scaling or ownership justifies the new failure modes.

Scalable web application: more API replicas exhaust a shared database connection budget. Limit total connections across replicas, shorten expensive transactions, and shed work before adding another layer of waiting. If a session cache disappears, either reconstruct a session through a verified bounded path or fail closed. Treating a cache failure as permission to skip authentication is not a fallback.

Serverless API: a burst creates more concurrent functions than the database can support. Cap concurrency against the dependency's budget and account for cold starts, runtime limits, and retries. Uploading an object and saving metadata are separate operations: stage the upload, validate it, then finalize its state. Repeated media jobs should produce the same variant identity, while permanently broken inputs stop retrying.

Service platform: a slow inventory service holds open checkout requests, which consume API connections, which trigger client retries. Give the user request one overall deadline, pass the remaining budget downstream, bound concurrency, and avoid retrying at every layer. Three attempts at each of three nested layers can create 27 calls to the deepest dependency. gRPC's deadline guidance describes propagation and cancellation; the same budgeting question applies to HTTP calls.

A gateway, service mesh, or telemetry collector also has a capacity and failure boundary. Add those components for a specific responsibility. A diagram with more services does not establish independent operation, compatible contracts, or safe deployment.

A post reaches 200 followers—or 20 million

Ordinary author: Alice publishes to 200 followers. Save the authoritative post, then add its ID to bounded prepared feed lists in the background. Bob's Home request reads candidate IDs, loads bodies in batches, checks visibility, and orders the result. The feed list is a derived view; losing it should not delete Alice's post.

Celebrity spike: pushing one post to 20 million followers can monopolize fanout. A hybrid design pushes ordinary authors and merges recent posts from very large authors at read time. The threshold comes from posting frequency, follower activity, write cost, and read cost—not a universal follower count. Break large fanouts into bounded recipient batches and isolate them from ordinary authors.

The awkward cases: an inactive reader returns with no prepared list; an author crosses the push/pull threshold; a reader follows thousands of high-volume accounts. Rebuild only a bounded recent window, overlap old and new strategies during a transition with ID deduplication, and cap candidate collection and merge work. Otherwise the optimization creates an unbounded read path.

Duplicate fanout: the same event arrives twice. Inserting a post ID into a recipient's feed must be repeat-safe. Lagging fanout: readers can see older content while the system catches up, with a separately defined read-your-writes path for the author. Replica lag: a prepared feed references a post that its body replica cannot yet read. Use a chosen freshness path or bounded retry; do not turn every missing record into an unlimited origin query.

Partition from access patterns: author timelines by author, prepared feeds by recipient, and follow relationships in the directions needed for lookup. Bucket an enormous follower list when necessary. Hashing many keys does not split the load on one exceptionally hot key.

A late post is acceptable. A blocked post is not.

Scenario: Bob blocks Alice, but Alice's IDs remain in his prepared feed. Waiting for the fanout queue to remove them would make the block depend on unrelated background progress.

The next authorized feed read must enforce current visibility before returning content. Persist the block, invalidate or version relevant permission caches, and check the relationship during hydration. The block endpoint's success contract must account for stale replicas and caches; simply moving the check into another stale service does not solve the problem. The client should remove already-rendered items immediately, but client filtering is not access control.

Deletion has the same stale-reference problem. Keep a tombstone or authoritative deleted state long enough for delayed consumers, and hide the record on reads while cleanup reaches feeds, search, media, caches, notifications, and derived copies. Replaying an old event must not resurrect the post. A reply or repost also cannot bypass the original post's visibility.

Unfollowing, private-account changes, moderation decisions, and mutual-follow queries need explicit semantics too. A counter may lag; permission to view private content cannot be inferred from an old count or a previously cached page. When permission cannot be established, exclude the content. Measure stale exposure and cleanup lag separately from ordinary feed freshness.

The cache returns, but the old value returns with it

Scenario: a cache miss starts a slow loader. While it runs, another request updates the database and invalidates the cache. The old loader finishes and repopulates the cache with obsolete data.

AppLab's cache adapter uses a fill marker and a conditional write to stop that old loader from storing after invalidation or replacement. That protects the cache entry. The original caller may still receive the old loaded value, so this is not a claim of linearizable reads. A stricter product contract needs an additional version or freshness check.

Cache misses must also be distinct from legitimate values such as null, false, and zero. Bound entries with TTLs, coalesce concurrent loads within the adapter's actual scope, and discard fills that complete after their deadline. A local single-flight map does not coordinate every process in a cluster.

Cache outage: forwarding every miss to the database can move the outage downstream. Use bounded fallback, selective stale data where allowed, reduced work, or explicit errors. Permission decisions require a different fallback from a public trending list. Design and exercise the cold-cache path, as discussed in AWS's caching guidance.

Page two changes while the reader is on page one

Scenario: a chronological list contains (10:00:05, 105), (10:00:05, 104), (10:00:04, 103), (10:00:04, 102), and (10:00:03, 101). Page one returns 105 and 104. Then post 106 arrives and post 103 is deleted.

Offset pagination can shift under those changes. A cursor carrying the initial cutoff and the last (created_at, id) tuple lets the next request continue below 104 and return 102 and 101. The tie-breaker matters because timestamps repeat. This is an illustrative query; authorization and cursor validation still belong in the request path:

SELECT id, author_id, body, created_at
FROM posts
WHERE created_at <= $1
  AND (created_at, id) < ($2, $3)
  AND deleted_at IS NULL
ORDER BY created_at DESC, id DESC
LIMIT $4;

PostgreSQL row comparisons provide the tuple comparison used here. Match indexes to the real filters and ordering, and bind the cursor to its viewer and feed mode. A timestamp cutoff is not a perfect snapshot: a later-arriving record with an older timestamp can still enter it. A ranked feed may need a stored candidate ordering or feed session instead.

Large IDs should cross a JavaScript API as strings when they can exceed safe integer precision. A Snowflake-style generator also needs unique worker assignments and a deliberate response to clock rollback; chronological-looking IDs do not remove those operational requirements.

A correct server can still produce the wrong screen

Stale response: the reader switches feed mode or account while a request is in flight. Cancel the obsolete request where possible, and check a request-generation identity before applying its result. Key cached pages by viewer and feed mode; clear sensitive state on logout. Cancellation alone cannot guarantee that no completed response callback will run.

Uncertain publish: the user submits “Hello” with key publish-17. The server saves post 9001, but its response is lost. Meanwhile the user changes the editor to “Hello again.” Retrying must use the original key and original text. On confirmation, replace the pending item with post 9001 and preserve the new draft. Reusing the key for different text is a conflict; creating a new key for the old text can duplicate the post. If the key has expired, reconcile before blindly submitting again.

Pagination and live updates: allow one request for a cursor at a time, deduplicate IDs across pages, and retain existing items if the next page fails. Surface new posts with an explicit control so they do not move the reader's position. A socket or server-sent event is a hint to reconcile authoritative state, not permission to trust every reconnect payload.

Reconnect gap: retain a resume cursor, reconnect with jitter, and detect when the server no longer has the missing interval. Fetch a fresh bounded snapshot instead of silently joining incompatible history. Preserve focus through loading and reconciliation, keep keyboard access when virtualizing, offer explicit load-more controls, and respect reduced motion.

Uploads, search, ranking, and counters have separate clocks

Partial upload: issue an authorized, bounded upload destination and keep metadata in a staged state. Verify ownership, size, type, and processing results before marking media ready. An object in storage is not proof that a post can safely reference it. Clean up abandoned uploads; use deterministic variant identities so repeated resize or transcode jobs do not create duplicates. Private media needs authorization at delivery too.

Search outage or indexing lag: publishing and Home should not depend on the search tier being healthy. Track indexing lag, replay source events, and apply current visibility when returning matches. A deleted document can still exist in an old index until cleanup; search results must not expose it. An inverted index finds candidates; relevance ordering is another decision.

Ranking failure: collect a bounded candidate set, then score and filter it. If the scoring service fails, a defined chronological fallback can preserve a usable feed. Evaluate ranking on its own objectives instead of assuming more model complexity improves the user experience.

Trending topic: compare recent activity with a baseline in bounded time windows and regions. Limit repeated contributions from one actor, and avoid concentrating every update for a popular topic on one worker. Distinguish an already-large topic from one whose activity is rising quickly.

Likes and notifications: keep the authoritative unique user/post relationship separate from a derived count. An approximate count can catch up asynchronously. Delivery preferences, duplicate prevention, and private-content checks still apply before a notification leaves the system.

A reconnect storm is also a history problem

Scenario: a real-time messaging deployment restarts and thousands of clients reconnect together. They all authenticate, load conversation history, and subscribe at once. The recovery traffic can overwhelm a service that handles steady-state traffic comfortably.

Jitter reconnects, drain old connections deliberately, and bound catch-up work. Store durable conversation history separately from transient socket delivery and presence. Presence can use an expiring heartbeat; it should not be the source of truth for whether a message exists.

Stable message IDs let clients deduplicate. A per-conversation sequence or equivalent cursor lets them detect gaps and order the relevant history. Receiving message 43 before 42 should trigger reconciliation according to the conversation's contract, not an assumption that delivery order is universal.

When the resume point is too old, say so and fetch a bounded history window. Monitor catch-up latency, authentication load, connection churn, and oldest missing message—not just the number of open sockets.

Payment succeeded after the checkout timed out

Scenario: inventory is reserved, payment authorization succeeds remotely, and the response is lost. Retrying checkout as a new operation may charge twice. Releasing inventory immediately may make a paid order impossible to fulfill.

Represent reservation, authorization, capture, fulfillment, expiry, and compensation as explicit states with stable identities. Reconcile the payment provider using its documented idempotency and lookup behavior. Stripe's idempotency documentation is one concrete provider contract, not a guarantee that every remote API works the same way.

Reservation expiry: if stock expires while payment is uncertain, recovery must choose a valid next action: obtain another reservation, reconcile and cancel, or enter an exception state that a person can resolve. Avoid duplicate fulfillment while reconciliation is pending.

Compensation is another operation that can fail. A refund is not a database rollback, and its status needs durable tracking. Test the state transitions around lost responses, repeated callbacks, expired reservations, and failed compensation. The useful outcome is one explainable order state, not every service showing a green health check.

The report is published before the data is complete

Batch pipeline: a source changes schema halfway through an import. Validate and quarantine incompatible rows, preserve raw inputs where permitted, and stop publication of an invalid output. Give the batch and its partitions stable identities. After a crash, resume from a checkpoint and use a repeat-safe merge or partition replacement. Publish a completed version through a commit marker or atomic pointer change only after required partitions pass validation.

Duplicate batch: running the same input twice must not double the totals. Record input version, transformation version, and completion evidence. Retention and deletion rules must cover raw, intermediate, and published copies, not just the visible report.

Streaming analytics: an event arrives after its reporting window appeared complete. Decide how long results remain provisional, what watermark closes a window, and whether later arrivals correct previous output or enter a separate adjustment path. Processing time and event time answer different questions.

Hot partition: one customer or topic dominates a stream key. Track lag per partition, and consider splitting work or computing partial aggregates before merging. Any split must preserve the ordering actually required by the calculation. Replay and duplicate handling also need stable event identities. A broker's transaction feature alone does not make an unrelated external database effect exactly once.

The primary region is gone. Is the replica safe to promote?

Scenario: the writer region disappears while a replica is behind. Promoting it restores availability, but acknowledged writes missing from that replica may be lost. Recovery time and tolerated data loss are separate targets: RTO describes how long restoration may take; RPO describes the tolerated loss window.

Detect failure through more than one signal, inspect replication position, and fence the old writer before allowing a replacement to accept writes. Otherwise a network partition can leave two writers believing they own the same data. A second region on a diagram does not define a write-conflict policy.

The region returns: reconcile its state before admitting traffic. Restore current permissions and deletion state as well as posts. A replica is also not a backup: a bad deletion can replicate successfully. Test restoration and the time required to rebuild derived feeds and indexes.

For a single-writer design, route reads that need fresh confirmation to an appropriate consistency path. Other reads can accept bounded staleness under an explicit contract. Stronger cross-region write guarantees usually carry latency and availability tradeoffs that must be justified by the operation.

Retries at an external boundary need a narrower contract

AppLab's integration fixtures exercise more than “success” and “error.” Each outcome asks for a different response:

The HTTP adapter's selected read retries do not imply that writes are safe to replay. Its total deadline also includes credential lookup. Logs should preserve status, timing, and attempt identity without leaking credentials or raw private payloads.

Webhook receiver: verify the signature against exact raw bytes, event ID, and timestamp before accepting the event. Then durably record the ID and payload fingerprint. An identical repeat can return a duplicate result; the same ID with different content is a conflict. Standard Webhooks specifies the signing fields. Signature age tolerance and deduplication retention solve different problems: retain identity long enough to cover the sender's retry horizon.

Webhook sender: preserve event identity and serialized bytes across attempts, refreshing the signature timestamp as required. A timeout may mean the receiver already accepted the event. Retry only against a receiver contract that can deduplicate it. If terminal delivery fails, persist the failure before reporting it as stored for recovery. Failure to persist that record is its own error. A callable sender cannot recover a process crash before completion unless outgoing intent was durable before the call.

Notification delivery: use (event, recipient, channel) identity, check preferences at enqueue and before each attempt, and fail closed if preferences cannot be established. An opt-out while waiting must suppress later delivery. Record attempts before invoking the transport; a crash may still leave an uncertain remote result. When retained identity storage reaches its bound, reject new work or apply an explicit retention policy instead of silently forgetting deduplication history.

A scheduled job was running when the process stopped

Overlap: a five-minute schedule launches a job that takes seven minutes. Choose whether to skip an overlapping invocation or retain one queued invocation. AppLab's queue-one policy retains the newest pending invocation and records replacement of the older one. This is a policy choice, not a universal property of cron.

Missed runs: after downtime, choose between skipping and coalescing missed occurrences. A bounded lookback prevents a long outage from creating an unlimited catch-up storm. Record when that bound truncates history.

Clock changes: daylight saving can remove a local time in spring or repeat it in autumn. The studied scheduler skips a nonexistent local time and chooses the first occurrence of a repeated time. Persisting invocation identity prevents a wall-clock rollback from quietly repeating the same scheduled work. Other schedulers may choose different semantics.

Interrupted execution: a job marked running before the crash has an uncertain effect afterward. Do not automatically convert that into “failed, safe to rerun.” Reconcile its external result or require a documented idempotent handler before replaying it. Preserve immutable attempts and distinguish failed, cancelled, interrupted, and completed states.

A local single-process ledger is not a distributed scheduling lease. Multiple schedulers need coordination and fencing. A preview of a future invocation is also not evidence that it ran; completion requires the handler's actual result and durable record.

The assistant retrieves a document the user can no longer read

Scenario: a document is removed from a user's permissions, but its old chunks remain in a vector index or response cache. Retrieval finds those chunks and sends them to the model. Hiding the citation afterward is too late: private content already entered the context.

Check current access before constructing model context, scope retrieval to authorized candidates, and propagate removals through indexes and caches. If authorization cannot be verified, exclude the document. Keep the source of truth separate from an asynchronously refreshed vector representation.

Irrelevant evidence: a confident answer cites a document that does not support it. Inspect chunk boundaries, retrieval recall, ranking, and citation support using a labeled evaluation set. Increasing the context window does not fix missing or irrelevant evidence. Allow the system to decline an answer when the authorized material does not support one.

Measure retrieval quality, index freshness, permission failures, and grounded answer quality separately. A quick response with a citation is not sufficient proof that the answer is correct or authorized.

The release passed tests and still broke production

Scenario: a new artifact passes its tests but changes latency or error rates under production traffic. Build once, promote an identifiable artifact, release to a bounded cohort, and compare version-specific health before expansion. Pause or roll back when the chosen checks fail. Keep configuration identity visible too; the same binary with a different configuration can behave differently.

Schema rollback: the new application deletes a column that the old application needs. Rolling back the binary now cannot restore service. Use an expand-and-contract migration: add compatible structure, deploy code that can coexist, migrate and verify data, then remove the old structure after the rollback window closes.

Contract change: one service deploys a payload that an older consumer cannot parse. Version and validate contracts, test compatibility, and preserve failed messages for deliberate recovery. A successful deployment command is not evidence that dependent workflows work.

Rehearse the failure as well as the release. Measure user-path health, migration progress, and recovery time. Rollback is a tested operation with prerequisites, not merely a button on a deployment page.

Compare outcomes, not just green boxes

Replay with fixes runs the same twenty order IDs and the same timed faults through both policies. The comparison table is the completed result at 30 seconds; the diagram then replays the protected run so you can inspect how it got there.

Look at the measures together:

• Persisted and acknowledged: distinguish durable state from confirmation received by a worker.
• Extra writes: reveal duplicated side effects even if throughput looks healthy.
• Rejected deliveries and dead letters: show different ways an attempt can stop progressing. A duplicate delivery can fail while another copy succeeds.
• Unresolved deliveries: include queued work, deferred admissions, active attempts, and scheduled retries.
• Retries: show additional load created by recovery itself.

The protected policy is a bundle of bounded retries, idempotent writes, and admission control. Each solves a different problem. The default outage demonstrates recovery; the duplicate scenario isolates the effect of the write identity; the latency scenario exposes the gap between a commit and its acknowledgment.

What the interactive model does—and does not—prove

This is a deterministic, single-browser simulation, not a cluster benchmark. It advances in 100ms ticks, has one database, two workers, and a finite arrival schedule. The diagram's packets illustrate deliveries and responses; their travel animation is not a measured network trace.

It omits replication, consensus, process memory, durable broker internals, transaction isolation, and arbitrary network partitions. The worker's retry policy approximates redelivery rather than implementing a particular queue product's visibility timeout. Idempotency is atomic within the simulated database, and its record does not expire during the run.

Those boundaries keep the experiment small enough to inspect. Its central question is practical: when an attempt fails, what evidence tells us whether the operation failed too?

The feed, cache, region, payment, and other diagrams are staged design walkthroughs, separate from the timed pipeline simulator. They describe where a guarantee has to live and how to test it. Implementations still need verification against their actual database, broker, provider, and deployment behavior.

For any of these scenarios, keep a record of the operation identity, authoritative state, delivery attempts, permission decisions, and unresolved work. Recovery is complete when that evidence supports the product's promise—not simply when retries stop.