Choose the integration path that meets the workflow and can be operated

TL;DR: Choose a source path that can meet the agreed workflow, then verify permission, freshness and replay behavior on that path. An adapter isolates parsing and transport details; it does not make stale snapshots equivalent to a live feed.

Where you are. Module two. The freight pilot has a decision and dependency list. You now need an approved, testable route from the transport system to the exception workflow.

Four shapes to investigate

The fictional freight transport system predates its customer portal. Its owner offers several ways to export information, with different guarantees. Do not infer those guarantees from the system's age or from a box labeled “integration” on the architecture slide.

Conway's 1968 paper connects a system's design to the communication structure of its designers. Use that as a reason to ask who owns each boundary. It does not prove that every fragmented dataset comes from an old reporting line, or that technical changes always require reorganizing teams.

The four paths below organize the questions to ask. None is automatically best, universally available or exempt from security review.

File drop. A producer exports records to an agreed location. Confirm the schedule, whether the file is a snapshot or a delta, how completion is signalled and who can read it. Publish through an atomic rename or a completion manifest where the storage supports that pattern, so the consumer does not mistake a partially written file for a complete export. Check both expected arrival and source watermark; a newly downloaded old file is still old data.

Request-response interface. An API or RPC endpoint may provide current records, cached records or an asynchronous job result. Confirm response semantics, pagination, rate limits and timeout behavior. Some interfaces carry an application error inside a successful transport response; parse the documented error structure rather than searching arbitrary payload text for the word “error.” A customer's business record may legitimately contain that word.

Direct database read. Agree a supported view or schema contract with the owner and use a read-only account. A replica can reduce primary load but may lag; it is not automatically current or harmless to overload. Establish query budgets and timeouts, and test schema changes before relying on the result. A bounded approved primary read can be appropriate where its owner has assessed the load and consistency requirement; do not treat a replica as a substitute for that decision.

Hyrum's Law describes how clients can depend on observable behavior beyond an interface's stated promises. Direct table reads create exactly that risk if you rely on undocumented status values or join semantics. Write those dependencies down and get an owner to review them. A contract test detects some changes; it cannot guarantee that a field's business meaning stayed constant.

Change feed. A feed can support incremental freshness, but you need its ordering scope, retention, duplicate behavior and recovery procedure. A resume token that expires during an outage is an operating constraint. Ask how to combine an initial snapshot with the feed without losing changes between them, and how deletions are represented. A low-latency feed you cannot restart is an incomplete integration.

Choose against the actual pilot question

Use this synthetic option packet. The pilot tests historical shipment identification, with a separate possible phase for live fifteen-minute handling. All approval states and timings are invented for the exercise.

Choose the export plus portal messages for a historical lookup pilot, after confirming the dataset and identity mapping. Keep live freshness out of its claim. The portal API cannot make transport data fresher, and a replica query returning quickly measures response time rather than the age of its values.

If the buyer requires live fifteen-minute handling as the pilot's sole decision, none of the current evidence establishes a compliant end-to-end path. Escalate the replica/feed checks or renegotiate the question. Calling an available export “good enough for phase one” does not solve a requirement the funded pilot must test now.

Test a file's age and completeness separately

Suppose the exception demo uses a historical snapshot with a 26-hour maximum age agreed for this exercise. At April 11, 02:00 UTC, you receive this supplied manifest:

The file arrived five minutes ago, but the data is 50 hours old. Its age exceeds the 26-hour requirement by 24 hours. A fresh transfer timestamp cannot repair that gap.

The row ledger balances: 998 parsed + 2 rejected = 1,000 declared. That establishes accounting for this manifest, not complete usable data. Retain the rejected records in an access-controlled quarantine with reasons, and decide whether the pilot can proceed with those exclusions. Do not publish “1,000 records loaded” when two could not be parsed. The manifest's own count also needs a source-side guarantee; a balanced consumer ledger cannot detect rows the exporter never included.

If the completion marker were missing, wait or investigate according to the producer contract before processing. An empty file may be a legitimate zero-record export; absence of the expected file is a different state. Give each a distinct status so the operator knows whether to retry a transfer, ask the producer or inspect rejected records.

Build an adapter without erasing the source contract

Put transport and parsing behind a small adapter. Downstream code should consume a normalized record plus the source version, extraction time and relevant watermark. Preserve missing, deleted and unknown states rather than flattening them into ordinary values.

Recorded fixtures let you develop without continuous customer access, subject to permission to retain them and appropriate redaction. Maintain at least a happy response, a missing-field response, an application-error response and a replayed input. Prefer synthetic fixtures where real customer records are unnecessary; never put secrets or unapproved personal data into the repository.

The adapter makes a future switch more localized, but the workflow still needs a migration review. A full snapshot has different deletion and completeness semantics from a stream of changes. Run both paths on comparable records, reconcile their watermarks and counts, and test restart before changing the source of truth. Reusing the same normalized fields does not establish semantic equivalence.

Write the operating agreement

For the chosen path, name the producer, consumer, approval owner and failure contact. Record expected arrival, freshness limit, retention, credentials, source schema version and the response to a missed interval. Include the recovery method and the largest replay the source owner permits during business hours.

Use a measurable statement: “The export watermark must be no older than 26 hours when the historical demo starts; otherwise the demo reports stale input and pauses.” Avoid “the export is daily,” which leaves both its cutoff and the consequence of a missed run undefined. If a stale dataset can safely support a narrower demonstration, label its age visibly and obtain agreement on that narrower claim.

The spine below is the choice in order, and it separates the two verdicts a single manifest check tends to blur: how old the data is, and how much of it is there.

Do this before moving on

Choose a path from the option packet and write its approval dependency, freshness claim and recovery question. Diagnose the supplied manifest with separate freshness and row-accounting verdicts. Then move its source watermark to April 10 at midnight, keeping the April 11 02:00 check time and the two rejected rows.

Worked review. The original snapshot is 50 hours old and fails the 26-hour age limit. The changed snapshot is exactly 26 hours old and passes an inclusive maximum-age policy. Both ledgers still contain two rejected rows, so neither proves 1,000 usable records. For the historical pilot, the approved export is a feasible path if those data-quality limits are handled. The live fifteen-minute question remains unproven by every available path in this packet.

Go deeper

• Change data capture is the fourth shape in depth, including the ordering problems it hands you.
• Data quality is what you need at the seam when the source is a nightly file nobody validates.
• Idempotent pipelines matter more here, because file drops get redelivered and feeds get replayed.
• Testability and dependency injection is how the seam lets you develop against fixtures.
• Integrating an undocumented system is the Foundations method this builds on.
• REST API design is what you hand their engineers once the legacy system is behind you.
• Consistency and CAP decides how stale the reads coming out of that integration are allowed to be.

Key takeaways

• Check the source contract, permissions and recovery path for each integration option; age alone does not determine them.
• Distinguish transfer time, query latency and the source watermark.
• An export can balance its row ledger while containing unusable or stale data.
• An adapter localizes transport details but must preserve freshness, deletion and missing-value semantics.
• Keep the funded question visible when choosing a fallback; historical lookup is not proof of live handling.

Check yourself

Answer before you look. Recalling it is what makes it stick; recognising it does not.

1. 1A file downloaded five minutes ago has a 50-hour-old source watermark. What is its data age?

   Five minutesFifty hoursZero, because download completed

2. 2A manifest declares 1,000 rows; 998 parse and two are quarantined. What can you claim?

   Answer it, then reveal

3. 3Why does a common normalized record not make a snapshot and change feed interchangeable?

   Answer it, then reveal