Test replay, slow dependencies and changing data before adding load

TL;DR: Inspect what happens when work repeats, dependencies consume time and data violates its contract. Test the relevant failure and recovery paths; a key, timeout or validator is useful only with the surrounding semantics it needs.

Where you are. You have a trace and boundary register. Use them to choose failure tests before the next module develops the production code at those boundaries.

Choose failures the ordinary demonstration misses

Replay, slow dependencies and changed data are useful investigation categories. They are not an exhaustive list or a measured ranking of every customer's incidents. They can be tested with controlled fixtures; they are not inherently invisible until production.

Start with the actual write paths, resource limits and input contracts. A successful request under clean data and low load leaves these questions unanswered. Ask what evidence would establish the behavior under the failure, including what the operator sees and how recovery works.

Repeated delivery and repeated intent are different

A queue may redeliver under its delivery contract; an operator may rerun a batch; a caller may retry after losing a response. The intended business operation can be the same even when the transport attempt is new. Conversely, two legitimate operations may use identical payloads.

Choose identity for the logical operation, with appropriate tenant/source scope and payload consistency. A shipment ID alone cannot distinguish “tag shipment” from “send update” or successive valid events for that shipment. A timestamp generated anew per retry defeats deduplication; a timestamp that is part of a source's genuine unique event identity may be meaningful. Do not reduce key design to “timestamps bad, record IDs good.”

Checking a key before a write is insufficient if the check and effect are not coordinated. Run this deliberately broken, single-threaded simulation of two interleaved callers. It demonstrates the race without requiring threads or touching a real service.

seen = set()
effects = []
key = ('tenant-a', 'source-x', 'event-7', 'review-tag')

# Both callers check before either commits.
a_should_write = key not in seen
b_should_write = key not in seen
if a_should_write:
    effects.append('tag applied')
    seen.add(key)
if b_should_write:
    effects.append('tag applied')
    seen.add(key)
print('business effects:', len(effects))
print('remembered keys:', len(seen))
assert len(effects) == 2 and len(seen) == 1
# business effects: 2
# remembered keys: 1

The set contains one key while two effects occurred. A durable unique operation record coordinated atomically with a local state change can address this class of race. Remote effects need the receiver's idempotency contract or explicit reconciliation; a local database transaction cannot roll back an email already sent elsewhere.

Test duplicate delivery, concurrent attempts, a crash before and after the effect, conflicting payloads under one key, changed authorization and expired deduplication retention. The later replay and action lessons provide fuller implementation exercises. Here, record the operation contract and the failure evidence you still need.

Bound the time and the work

A dependency can fail quickly, slowly or never return. A “dead” endpoint may also hang until a network timeout. The dangerous behavior is consuming shared resources beyond what the system can tolerate, regardless of the label.

Inspect connection, request and overall deadlines, including what the client actually covers. A timeout limits waiting; it does not prove remote cancellation or prevent all accumulated work. Bound concurrency, queue size and retries, and decide how overload is rejected or deferred. Protect unrelated work where dependencies share a pool.

Marc Brooker's AWS article on timeouts and retries documents an instructive case: a low timeout encountered secure-connection setup after deployments, while reused connections usually succeeded. The team first adjusted the handling and later established connections before taking traffic. The lesson is to inspect the work included in a timeout rather than copying a number from another service.

Use this separate synthetic deadline calculation. The request has a 1,000 ms total budget, 100 ms of local work and three permitted attempts, each bounded at 250 ms. The two backoff waits are 50 and 100 ms. These are illustrative allocations, not recommended defaults.

budget_ms = 1000
local_ms = 100
attempt_limits_ms = [250, 250, 250]
backoff_ms = [50, 100]
planned_ms = local_ms + sum(attempt_limits_ms) + sum(backoff_ms)
print('planned bound:', planned_ms)
assert planned_ms == budget_ms

# If local work actually consumes 300 ms and the first attempt uses 250,
# only 450 ms remain. After 50 ms backoff, at most 400 remain for all work.
remaining_ms = budget_ms - 300 - 250
print('remaining before next backoff:', remaining_ms)
assert remaining_ms == 450
assert remaining_ms - 50 == 400
# planned bound: 1000
# remaining before next backoff: 450

An implementation must compute the remaining monotonic deadline before each wait and attempt, leaving room for required response handling. It must not launch the entire original retry schedule after local work overruns. Actual timeout coverage and cancellation behavior require testing; arithmetic is only the plan.

Retries at multiple layers can multiply load. If three layers each make up to three total attempts, one top-level request can cause up to 27 deepest calls when all retries are exercised. Coordinate retry ownership, bound attempts and use an appropriate backoff/jitter policy. A retry is also inappropriate when the operation's outcome is unknown and no valid repeat-safe contract protects it.

Use the circuit-breaker model with its limits

The existing widget shows closed, open and half-open behavior. Trigger failures, observe calls being blocked, then watch limited probing before normal calls resume.

A breaker can reduce calls to a failing dependency. It does not automatically cancel in-flight work, bound every queue, isolate all resources or establish business-effect safety. Its thresholds, scope, probe behavior and recovery interaction need tests. It is one possible control, not a mandatory second step after every timeout.

Validate the agreed meaning, not only the shape

An added field may be backward-compatible under one contract and forbidden under another. A status changing from ACTIVE to Active may be valid if the documented contract is case-insensitive. If the contract requires exact enum values, treating it as active without review is an unsupported assumption.

Write the accepted representations and canonical result explicitly. Preserve the original input and transformation provenance where required. Do not silently convert unknown values into a plausible default. A structurally valid value can still be semantically wrong, such as a valid-looking customer ID belonging to another tenant.

For this small exercise, the agreed input contract accepts exactly ACTIVE and Active as aliases for the canonical value active; INACTIVE is canonicalized to inactive. Other values are rejected. This is a supplied contract, not a general recommendation to accept mixed case.

aliases = {'ACTIVE': 'active', 'Active': 'active', 'INACTIVE': 'inactive'}

def canonical_status(value):
    if not isinstance(value, str) or value not in aliases:
        raise ValueError('Unsupported status representation')
    return aliases[value]

assert canonical_status('ACTIVE') == 'active'
assert canonical_status('Active') == 'active'
assert canonical_status('INACTIVE') == 'inactive'
for value in ['active', ' ActIve ', None, 1]:
    try:
        canonical_status(value)
    except ValueError:
        print('rejected:', repr(value))
    else:
        raise AssertionError('Unexpected acceptance')

The lowercase string is rejected because this particular input contract did not include it, even though it equals the canonical output. If that representation should be accepted, change the contract and tests deliberately. Do not infer the input language from the output format.

Quarantine needs a reason, owner, retention/access policy and replay path. Some invalid records can be isolated; some failures require holding the whole batch or affected workflow. Alert on actionable conditions without copying sensitive input into broad logs. Reconcile received, accepted, rejected and unresolved work so failures cannot disappear from the count.

The spine below runs all three families as one test sequence, because they share a starting point: the real write paths, limits and contracts rather than the demonstration path.

Do this before moving on

Run all three fixtures. Explain why one remembered key did not prevent two effects; calculate 27 deepest attempts from the three-layer plan; then change the status contract to accept lowercase explicitly and add its test.

For your trace, write one failure test and one recovery check for each category. Include a lost response after a write, a slow call that consumes the remaining deadline, and a valid-looking value with the wrong domain meaning. State which controls belong to the producer, consumer and operating owner.

Your inventory passes when it identifies exact effects, deadlines and contracts, and distinguishes tested behavior from proposals. Finding no defect in a small fixture is not evidence that all three categories are handled in the real deployment.

Go deeper

• Idempotency develops operation identity and repeat-safe effects in depth.
• Circuit breaker covers what to do once you have timeouts and still have a degrading dependency.
• Exponential backoff is the retry behavior that makes the twice problem common in the first place.
• Data quality is the discipline behind validating at the boundary rather than coercing.
• At-least-once versus exactly-once delivery is the interview question that starts from the same place this lesson does.

Key takeaways

• Identify logical operations and coordinate deduplication with effects; a prior key check can race.
• Timeouts limit waiting, while concurrency, queues and retry budgets bound accumulated work.
• Circuit breakers need explicit scope and recovery tests rather than automatic adoption.
• Accept normalization only under a defined contract and reject unknown meanings visibly.
• Pair every failure test with reconciliation and a usable recovery path.