Our Agent Chain Silently Routed Around a Broken Data Contract

We run a GTM motion on a ten-skill agent chain inside Claude Code - discovery, qualification, scraping, company signals, signal scoring, email resolution. In testing, we found that three of its writer stages had been violating the chain’s shared data contract, in one case for as long as the contract had existed. Nothing crashed, the outputs looked fine. Every run finished green.

This is the postmortem: what broke, why every run still passed, what we changed, and the regression gate that stops this.

The gate in action: a healthy baseline, a one-line edit that removes a contract cap, the commit refused with the violation named, revert, green.

The system

Each stage of the chain is a skill wrapping a vendor API, and the stages compose: discovery finds companies, qualification filters them, scraping and signals gather evidence, scoring ranks it, resolution finds the verified email. The thing that makes them a chain rather than six tools is the record contract: every stage writes a run folder - runs/<run-id>/ holding records.jsonl, tracker.json, meta.json - and appends its fields to the records the previous stage wrote. One record per company, growing as it moves downstream. The contract lives in a single shared conventions file that every skill references.

That contract matters more now than it did a year ago, for a reason we wrote up in The Post-Fable Repo: a weaker model needed a human between stages, and the human noticed the weird files. A Fable-class model runs the chain end to end, and the contract is the only thing keeping stage five compatible with stage two. Nobody is watching the seam.

The failure

A July audit graded every skill against the conventions file - verified in code rather than the skill docs. Findings:

• One discovery stage wrote records.jsonl to the parent directory of whatever filters file it was handed. In practice: /tmp/records.jsonl. And only on its export path, which was opt-in - a search-only run produced no chain record at all.
• The local-business discovery stage emitted no records, ever. Flat files, different shape. The documented chain through it only ran because a human handed the next stage a domain list. Nothing structural connected them.
• The company-signals stage split its output across two directories, neither matching the convention.
• Three writers emitted only {company, domain, person, filters_matched}. The documented per-stage fields - firmographics, scraped page content, funding, recent hires - were flattened into display strings inside filters_matched.

The conventions file described a chain that did not exist.

Why every run still passed

The scoring stage - the judgment layer that ranks signals and picks the outreach approach - accepted --scrape-run and --crustdata-run flags and read the upstream run folders directly. It reconstructed the evidence it needed from raw stage output, bypassing the records entirely. So task success measured the reconstruction, not the contract. The chain worked, but not in the way we intended.

That is the part worth generalizing. The failure mode of a capable model is not stopping at a broken seam - it is routing around the seam and finishing the task, invisibly. The drift would have surfaced the first time anything composed the chain a new way: a new consumer of the records, a new stage, or the public release we were preparing. By then the break would be weeks of commits away from its cause.

Root cause was ordinary: each writer was built against what the next stage happened to need that week, not against the document. Nothing executable enforced the document, so it drifted into fiction. A contract that isn’t enforced by code or verified by an eval is a comment.

The change

Four changes, all landed before the chain went public as Headless GTM:

1. The richer record shape won. Writers now emit the documented per-stage fields as structured data, not display strings.
2. Every writer lands on runs/<run-id>/ with records.jsonl + tracker.json + meta.json. No more /tmp, no more split directories. The local-business stage writes a real run folder the next stage consumes directly - it is structurally in the chain now.
3. Upstream inheritance. Downstream stages accept the previous stage’s records.jsonl, save it as upstream.jsonl, and inherited fields must survive to the output.
4. deviations[] in every run’s meta.json, plus a one-line command to append one. When a run has to depart from the contract, the departure is recorded in the run folder, not in someone’s memory.

The regression gate

The enforcement is a single offline evaluator. It imports each of the six writer stages, runs it against synthetic fixtures, and validates the output: folder layout, base fields with normalized domains, the per-stage field table, upstream inheritance, content caps, the deviations list. It also validates any real run folder after the fact.

It runs as a pre-commit guard. A staged change touching any chain script or the conventions file triggers the eval, and drift blocks the commit with the exact field and writer named. The hook itself is version-controlled with an installer.

We tested the gate by deleting one field from one writer and trying to commit. It refused and named the field. The recording at the top is the same loop on a subtler edit - removing a 15,000-character page cap, framed as “keep full page content for the model.” It reads like an improvement. It would have silently changed what every downstream stage receives. The gate catches it in the diff, at the commit, before it exists anywhere but the working tree.

What this proves, and what it doesn’t

The evaluator runs on synthetic fixtures and tests system invariants: shape, layout, inheritance, caps. It proves the contract holds - that the chain’s stages stay composable and that the seam we stopped watching is now watched by code. It does not prove the chain’s judgment is good, and it says nothing about reply rates or revenue. Judgment quality is a separate eval program with its own methodology - that work looks like our model-panel eval on AI-washing, and it is graded against human labels, not fixtures.

In an agent system, task success is not evidence that the seams hold. A capable model will paper over a broken interface and hand you a green run. Write the contract down, enforce it with code that runs at the commit, and log every deviation where the next person will find it.