Turning Legacy Code and a Cabinet of Documents into a Domain Agent That Actually Works: The Six-Step Path

I took nine legacy algorithms from a vertical domain (water resources) that had been running for years, plus a pile of domain documents that existed only as docx / pdf / xlsx, and turned them into a Q&A agent I actually use every day. Ask it "how does this indicator get banded in document No. X" and it greps, locates, and reads the original document in full, returning a sourced answer in seconds to tens of seconds. Ask it a computation question and it calls one of those nine algorithms.

This is not a retrospective — it is a tutorial: six steps in the order you would walk them from zero, and each step covers only three things — what to do, how to know it worked, and where the trap is. Every number here is measured, not estimated.

One reassurance and one warning up front. The reassurance: there is no vector database, no fine-tuning, no training anywhere in this chain — the main retrieval path ended up as just three moves: grep, locate, read the whole document; after cutting the fancy layer, quality went up. The warning: the most expensive failures in this chain were not code bugs but verifiers that lied — the same acceptance question went red four times, and in three of those four runs the thing that was wrong was the grading gate, not the answer.

The hard part of building an agent is not wiring up the LLM — it is that every layer of verification can go falsely green, and each layer must be asked: what exactly makes you green?

---

Step 0: Rank your goals first, or every later step fights itself

What to do: Before writing code, answer one question: who is this for? My first draft said "dual goals: daily real use + job-hunting showcase," side by side. Side by side is wrong. I changed it to a ranking:

Why this order: a thing you use every day demos itself with real usage records, which is the most convincing story there is; a thing designed for demos that you don't actually use falls apart at the first detailed question. And "distributable" means other people's machines need the corpus, installation, maintenance — productization costs that shouldn't be paid before the first line of code.

The ranking hardens into one iron rule: every architecture decision is judged only by "I use this myself, every day, for real work." Every later trade-off (cut the vector layer or not, fix broken algorithms or not, build multi-tenancy or not) gets settled by this one sentence, no meetings required.

How to know it worked: Any "should we build X" debate can be settled in ten seconds with that one test. If it can't, your goals aren't ranked yet.

The trap: The real cost of parallel goals isn't abstract friction — it's concrete rework. I once considered a dedicated fake-data mode "for demo polish" and later scrapped it entirely: the moment a demo mode forks from the real mode you maintain two behaviors, and interviewers love asking about exactly the seam between them.

---

Step 1: Inventory the estate — tag legacy code, don't fix it

What to do: Not all nine legacy algorithms work. After actually running each one, wrap each as a tool, register it in one table, and hang a four-color status tag on it:

Three key moves:

1. Don't fix the broken ones. The goal is an agent that works, not paying down years-old code debt. Fixing a broken algorithm is a bottomless pit; tagging it takes an hour.
2. Every tag must carry evidence: a reproduction command, the actual observed output, and a remediation path — all three, or you don't get to hang "broken." "I think it's broken" doesn't count.
3. Make the tags visible to the agent. When a user asks about a broken algorithm, the model sees the tag and says "this is currently unavailable, because —" instead of running it anyway and crashing in the user's face.

The final inventory: 22 tools in 10 groups (the document-retrieval tools registered as peers of the nine algorithms): green 12, degraded 4, blocked 3, broken 3. More than a quarter of the estate has problems — admit it first, then start.

How to know it worked: Call the tool-list endpoint once and check that counts and colors match what you got by hand; then deliberately ask the agent a question that lands on a broken algorithm and see whether it declines with a reason — that, not the happy path, is the evidence the tags actually work.

The trap: Never hand-copy the tag table into your docs. The day I copied it into the project docs it had drifted from reality in 7 places (measured). The right way: the table is computed from the registration code, and the copy in the docs carries a "hand edits are void" check that goes red if anyone edits it manually. Anything where "the same fact must appear in two places" deserves this treatment.

---

Step 2: The retrieval verdict — agentic search replaces chunk RAG

What to do: How does the model get to read the domain documents? The textbook answer is RAG: slice documents into chunks, embed them, stuff the top-k most similar chunks into the model. I walked that road first, then falsified it with one class of real questions, and switched to agentic search: let the model work like a person — grep / locate the document by structured clues first, then read it in full, and try another term if that fails.

The falsifying experiment was "comparison" questions — "what changed between the old and new editions of this standard" — one of the most common question types in professional work. Under chunk retrieval it is unsolvable, structurally:

• It is a capacity problem, not a ranking problem. The two editions together get at most 4 chunks of top-k — about 3.8% of the text. A better embedding model or a reranker only changes which 3.8%, never the only 3.8%.
• The darker layer: similarity retrieval systematically prefers the passages where the two documents use the same wording — while the differences live precisely where the wording differs. The retrieval paradigm is semantically opposed to this question type.
• Flip it around and look at discriminative power: the short strings that best pin down a document in this corpus — document numbers, place names, indicator names — are exactly the information embeddings are worst at preserving. grep hits them exactly.

So the main path is three moves, with chunk retrieval demoted to a fallback (only when all you have is a vague description):

Two money-saving details: ① retire the vector layer "off first, delete later" — first confirm the production dependency list contains no vector packages at all (and write a test asserting it never will), so cutting the layer is a zero-change for production, an order of magnitude less risky than it feels; ② office formats (docx/pdf) are binary — grepping the source files will never find Chinese text; extract the text into an index first and grep that, or "I searched, it's not there" is a lie.

How to know it worked: Use that comparison question as the acceptance test: the per-dimension differences that chunk retrieval couldn't answer become, under locate + two full reads, a dimension-by-dimension comparison table where every cell can be traced to the source text. The question going from "structurally unsolvable" to "seconds, fully correct" is the proof the verdict was right.

The trap: Don't parrot "even Claude Code abandoned RAG" as a slogan. It rests on a premise — the files are on local disk and greppable. That premise happened to hold for me (the corpus is all local); if your corpus lives in someone else's database or is too big to enumerate, this verdict does not transfer as-is. Check the premise before copying the conclusion.

---

Step 3: Build the eval gates first — and expect the gates themselves to have bugs

What to do: Before optimizing anything, stand up three layers of deterministic evals, all runnable with one command, red/green unambiguous:

• L0, the loop layer: the agent loop's critical paths (declines what it should decline, restates what it should restate), replayed against recorded fixed scripts, no live model;
• L1, the tool layer: all 22 tools actually invoked, outputs reconciled against golden values, fully deterministic, zero LLM;
• L2, the retrieval layer: given a question, retrieval must hit the documents it should hit;
• End to end: a few real questions against the live model; the answer is hard-graded on numbers that must appear and pairings that must appear together.

Two more disciplines: golden files must never be self-approved — changes require an independent second party to actually run and countersign; and grading scripts stay hostile to empty sets — a missing scan root or an empty enumeration is an automatic red. Refuse to report green on an empty set.

How to know it worked: All three layers plus end-to-end are green, and you put a known bug back in on purpose and the gate actually turns red. A gate that has never passed this reverse experiment has a green light that means nothing — it may never have been looking at all.

The trap: Worth bolding — the grading gate has its own bugs, and they are sneakier than the answer's bugs. The same end-to-end acceptance question went red on me four times. The post-mortem:

1. Run 1: the "-90/-80" in the answer were band lower-bounds; the grader captured them as negative numbers and falsely flagged missing values;
2. Run 2: the model rewrote the source's "60≤score<80" as "60–79" — 79 appears nowhere in the source; the gate rejected it correctly, and the fix belonged on the generation side ("quote ranges in the source's form; no paraphrasing");
3. Run 3: the context budget cap and the max-step cap fought each other — evidence fully gathered, then truncated by budget; the answer began "results incomplete";
4. Run 4: pass.

Three of the four runs needed fixes to the gate and the configuration, not the answer. If you treat every red light as "the answer is wrong" and hack at the generation side, you will break things that were right. The first move after a red light is an itemized audit: who exactly is red this time.

---

Step 4: Latency — 418 seconds to 23, itemize before you cut

What to do: Once it was correct, the same acceptance question took 418 seconds — unusable. Governance came in three cuts, and the order matters; the first cut is not an optimization, it is an itemized account: per-segment timing on the production trace to see where time actually goes. The account was startling:

Sixty percent of the time went to "generated, then thrown away," while all the actual work combined took under half a second. Caching, model swaps, and concurrency tuning without this account would all have been blind.

The three cuts, in order:

1. Streaming + a synthesis contract: forbid finished prose inside loop steps (it gets discarded anyway); the finished answer is generated exactly once at the end, and streamed to the frontend as it generates — the user goes from "blank screen for 418 seconds" to "first character early, prose flowing." 418 → 249s, and the frontend went from nothing to 44 streamed increments.
2. Locate-then-read-directly: start with one deterministic, zero-LLM reconnaissance pass (12 candidate documents by document number / year / keywords); if a document number in the question matches a filename exactly, skip the whole multi-step loop and read-then-answer. Document-number questions dropped from 72–126s to 21.5s, one model call.
3. Switch to a faster model provider: with the structural cost gone, the remaining wall-clock is mostly raw generation speed — only now does a provider swap mean anything (earlier it would only have masked the structural disease). Same question: 122.9 → 32.5s, and 21.5s measured in the browser with 14.4s to first character.

How to know it worked: Not averages — the same question under the same grading gate, before and after: 418.3 → 248.9 → 122.9 → 32.5 → 23.1 seconds, every step with a trace on file; and track time-to-first-character separately (streaming is about the first character; total duration can't fake the feeling of waiting).

The trap: Optimizing in the wrong order hides diseases. Had I swapped in the fast model first, the "generate and discard" structural disease would have been masked by model speed — 418 seconds becomes roughly 80, it looks like a win, and sixty percent waste survives with nobody ever hunting it again. Itemize first, cut structural waste first, hardware and models last.

---

Step 5: The repo-wide hunt, going live — and the acceptance checklist for the whole chain

What to do: Before going live, run one systematic bug hunt. The method matters more than the tooling; it has four parts:

1. Hunt by partition: split the repo into seven non-overlapping partitions (agent loop / API layer / retrieval / model layer / tool layer / frontend / ops scripts) and hunt each independently;
2. Adversarial verification: every finding gets a verifier whose stance is "falsify this" and who must reproduce it by hand — no reproduction, no finding. The first round returned 21/21 confirmed, zero rejections — zero rejections is itself suspicious — so every fix was additionally verified with a "red before the fix / green after" reverse case;
3. Every fix ships with a reverse test: one case that goes red if the bug is put back — a fix without its reverse test doesn't count as fixed;
4. Re-review the diff: another adversarial round over everything changed. This round: 21 new findings, 4 shot down by verifiers, 3 severe and real — the most dangerous being: the deploy script wrote a runtime config the server never reads, and step 12 of the post-deploy verification read exactly the file the script wrote — twelve green steps, and chat would be dead in production. The textbook "verification path ≠ execution path" false green.

Two rounds, 42 findings total, all closed (fixed, or a recorded reason not to fix). Then ship: the deploy script itself is a 12-step acceptance chain — dependency reconciliation, secret probes, index counts, decline-shape checks, one algorithm actually run in production against golden values, edge-network reachability — any red step aborts.

How to know it worked: Twelve green deploy steps are not the finish line — the last step is always asking the live system, as a real user, one real question with a known answer. My post-launch acceptance: the same document-number question, run live — every per-dimension score correct, 5 citations, seconds to answer. That is "deployed successfully."

The trap: Two, both cases of "the reconciliation tool is congenitally blind to one error class": ① file-level sha reconciliation catches "a file's contents drifted" but is blind to "a whole package never collected" — missing files aren't on the manifest, so reconciliation stays green forever; the fix is an import smoke test after reconciliation (actually load the entry module once). ② "upload/sync succeeded" only proves bytes left your machine, not that the other end received something runnable; deploy verification must reach the remote runtime (run one algorithm live, answer one real question live), never stop at the transport layer's receipt.

Finally, fold the whole chain's "don't trust / go look at" into one table. This table is the article's acceptance checklist:

Looking back, the most valuable thing this project taught was not any single technical choice but one move that runs through all six steps: whenever something reports green, first ask who feeds its enumeration, and whether its verification path is the same path execution takes. Gates err, reconciliation is blind, receipts lie — audit the verifier as suspect number one, and everything else is manual labor. Do that, and wiring up the LLM really is the easiest part of the chain.