Build an evidence map of an unfamiliar system

TL;DR: Follow a representative path, verify what revision and configuration actually run, and compare logs, data, tests, code and owner explanations. Each source answers different questions; preserve uncertainty instead of turning an incomplete trace into a complete architecture.

Where you are. Discovery identified a candidate workflow and its prerequisites. You now need enough understanding of the existing system to judge an integration or change without guessing about hidden behavior.

Start from the decision you need to make

You may have a repository, a warehouse, operating dashboards and documents of different ages. Begin with the question: can this source support the required freshness; where does identity change; how is a retry handled; or what can be deployed safely within the environment?

A short, targeted investigation is more useful than trying to read every file before forming a question. That does not make code a last resort. Entry points, tests and schemas can be the quickest route to the relevant behavior, especially when runtime access is unavailable.

Before removing an unexplained component, seek its purpose, dependencies and evidence. It may enforce an important constraint, or it may be obsolete. Neither conclusion follows from its age or from the absence of a person who remembers it. Record a testable removal hypothesis and the required checks instead of treating every old component as sacred.

Follow one record and mark the coverage limit

Use an approved synthetic or appropriately authorized record. Track its business identity, operation ID, transformations, stores and consumers. Avoid using a customer ID alone as a universal join key when it is only unique within a tenant or source.

A representative path helps identify participating components, but one successful record does not exercise alternate branches, rare failures, retries or other tenants. Missing telemetry can also make a component invisible. Add traces for the important variations after establishing the ordinary path.

Asynchronous work can time out too: publishing, processing deadlines, lease expiry and polling all have time semantics. “Async” means the caller does not wait for the complete downstream workflow in the same interaction; it does not mean delivery or completion is guaranteed.

OpenTelemetry's trace concepts describe spans and their relationships, including links for related asynchronous work. Instrumentation can help connect a path, but a missing span is not by itself proof that no effect occurred.

Read a supplied timeline

This complete Python fixture filters a small synthetic event log by tenant, order and operation. Times are offsets from one shared exercise clock, not wall clocks from separate machines. The fixture does not establish that real distributed clocks are synchronized.

events = [
    (0, 'A', 'O1', 'op-7', 'request accepted'),
    (100, 'A', 'O1', 'op-7', 'queue accepted message'),
    (200, 'A', 'O1', 'op-7', 'client timeout'),
    (300, 'B', 'O1', 'op-8', 'unrelated tenant operation'),
    (500, 'A', 'O1', 'op-7', 'worker started'),
    (550, 'A', 'O1', 'op-7', 'effect committed with receipt'),
    (900, 'A', 'O1', 'op-7', 'redelivery returned existing receipt'),
]
selected = [row for row in events if row[1:4] == ('A', 'O1', 'op-7')]
for offset, tenant, order, operation, message in sorted(selected):
    print(offset, message)
assert len(selected) == 6
assert sum(row[4] == 'effect committed with receipt' for row in selected) == 1
# 0 request accepted
# 100 queue accepted message
# 200 client timeout
# 500 worker started
# 550 effect committed with receipt
# 900 redelivery returned existing receipt

The supplied log records an effect after the client stopped waiting. It also records a duplicate delivery that returned the receipt. That supports this scenario's one-effect claim, assuming the log is complete and truthful; it does not prove the implementation is correct for all crashes or concurrent calls.

The queue wait is 400 ms between acceptance and worker start. Worker start to recorded commit is 50 ms. Adding every duration in a real trace can double-count overlapping or nested work, so use the path and timestamp semantics rather than summing arbitrary span lengths.

Remove the commit and receipt entries from the fixture. The remaining client timeout does not establish failure without effect. Record the outcome as unresolved and seek source state or another authoritative operation record. Do not replay a write merely because its success log is absent.

Verify what actually runs

Compare the repository revision with the deployed artifact, configuration, scheduled jobs and enabled routes. A code path in main may not be deployed; a deployed feature may be disabled; a scheduled worker may run from a different image.

Suppose the repository contains a timeout added in revision R2 while the deployment manifest identifies R1. You cannot claim production has the R2 timeout. Verify the actual release and effective configuration through authorized read paths. Conversely, a manifest alone may be stale; connect it to the running workload or trusted release record.

Inspect configuration names and provenance without copying secret values into your investigation notes. Distinguish source build, artifact promotion and runtime configuration, and record which evidence was available. This is enough to identify a gap without pretending you have inspected production when you have only read source.

Use workload and cost as clues

High-volume endpoints, large tables and expensive jobs can identify important investigation targets. They do not define importance by themselves. A low-frequency emergency path or a small access-control table may be critical, and an alert may be obsolete or poorly chosen.

Ask what the workload serves, who relies on its output and what failure costs. Compare average and peak behavior, backlog and freshness. Do not optimize a costly query before confirming its semantics; making the wrong report cheaper does not make it correct.

Read tests as executable claims

A test states an expectation for the exercised inputs and implementation. It might have been written from a requirement, a hypothetical failure, a bug report or a production incident. A specific date or ticket is a lead to investigate, not proof of an incident or its severity.

Read relevant tests early when they clarify the contract. Run them only in an appropriate environment after checking dependencies and effects. Passing tests do not prove the deployed version passes, and mocks may omit the integration boundary you need to assess.

Use tests, issue history, owner explanations and code comments together to understand reasoning. Where they disagree, write down the conflict and the next verification step. Do not assume one source always outranks all others.

The spine below is the investigation in order, with the coverage limit sitting as its own step because a trace that forgets it becomes an architecture nobody verified.

Do this before moving on

Run the timeline and remove the two final outcome records. Write what remains known and unknown. Then filter by order alone and show why the unrelated tenant event contaminates the trace. Explain why business identity and operation identity both matter.

Create an evidence map with five columns: claim, source, revision or observation time, coverage limit and next check. Include the R1/R2 deployment mismatch and a specific test whose historical origin is unknown.

For a permitted project of your choice, trace one request and one important failure branch. Compare your map with the architecture documentation. Record actual differences if present; do not invent a contradiction because the exercise expects old documentation to be wrong.

Go deeper

• Parsing messy data explains the transformations you may encounter while following a record.
• Observability for AI covers instrumenting a system so the next person does not have to do this archaeology.
• Idempotency is the property you will be checking for as you trace a record through retries.
• Change data capture explains one mechanism that maintains derived copies across systems.
• Debugging a production incident in a customer environment collects the questions that test this skill directly.

Key takeaways

• Orient around a concrete decision using multiple evidence sources.
• One trace covers one path; asynchronous work still has timeouts and uncertain outcomes.
• Verify deployed revision and effective configuration before claiming source behavior is live.
• Cost and traffic help prioritize, but critical low-volume paths still matter.
• Tests demonstrate exercised expectations, not automatic production history or complete safety.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1. 1The client times out at 200 ms; the worker commits at 550 ms. Did timeout prevent the effect?

   Yes, a timeout cancels all downstream workNo, this supplied timeline records a later effectYes, because the queue is asynchronous

2. 2Main contains an R2 timeout fix, but the deployment record says R1. What can you claim?

   Answer it, then reveal

3. 3A test names a ticket and a date. What does that prove about its origin?

   It definitely followed a severe production outageNothing definitive; inspect the linked historyIt must be obsolete