Earlier this week we posted about last week's experiment: Claude and Grok in a locked-down VM, run four times, one change per run. Over the weekend we ran it three more times, then took the three products they built and spent a day of GPU time on them ourselves. The first post was about the brief. This one is about the test set, because that is what separated the three runs, and it is also what decided how far we could push their work afterwards.
Every run now works the same way. Round 1 is research with sources. Round 2 is a plan with acceptance gates written down before any code. The agents split the data into train, dev and a sealed slice, and the plan says how many times the sealed slice may be scored. Then they build for six more rounds, twenty minutes a turn, and the README has to report every gate, pass or fail.
All three products are real. Each one is a CLI plus an API plus a Docker image plus tests plus a proof script that builds the container and makes a request. That part is no longer the interesting part.
Run 5: every gate missed, and said so
The agents picked customs classification: given a product description, predict the ten-digit US tariff code, trained on the public database of Customs rulings. Real data, finally. They set four accuracy gates in round 2, including a top-1 accuracy of 0.60 on the sealed rulings.
They missed all four. The sealed top-1 came in at 0.287 and the README says so in bold, with the rolling next-year numbers underneath showing accuracy falling the further the test year is from the training years. That drift table is the best thing in the run. A buyer could read it and know how often to retrain.
What they did not notice: the crawl was incomplete. The database holds about 215,000 rulings and they had 42,000 usable ones, with sixteen years of rulings never fetched, partly because the crawler was killed by an out-of-memory probe mid-run. Nobody counted what they had against what exists.
Run 6: the pass that was not one
The sanctions screener is the most complete product of the seven runs so far. It matches customer names against the OFAC list with variant spellings and dropped words, at a measured false-alert rate, and it screens ten thousand names in under a minute. It has an external holdout on the UK list, a speed benchmark and 118 commits.
Its plan said the sealed slice would be scored once per round. The harness printed a counter. By the end the counter said 22. The headline pass, organisation recall of 0.933 against a gate of 0.90, came from a rule that was added after a failing sealed score. On the UK list, which the agents had not tuned against, the same slice scored 0.81.
Two more "gates pass" claims were withdrawn in round 7 when the agents found their own harness had fixed the candidate rank at zero and was scoring raw strings that the product would have normalised. They caught it themselves, which is to their credit. But a number you looked at while choosing the model is a dev number, and the run's headline was a dev number.
Run 7: scored once, and the number held
The contract clause finder locates clauses like liability caps and termination rights in commercial contracts, trained on the public CUAD dataset. This time the orchestrator held back a final slice the agents never saw at all, and scored it once itself when the run ended.
The agents scored their sealed slice exactly once: 0.552 macro average precision, 0.651 on the eight clauses a buyer cares about most. The final holdout came in at 0.557 and 0.639, against gates of 0.55 and 0.65. One pass, one miss by 0.011, and the two numbers agree to within sampling noise. That agreement is what an honest test set looks like.
One more thing from run 7. The agents fine-tuned a small language model for 48 CPU minutes. It scored 0.269 on dev against 0.485 for plain TF-IDF, so they shipped TF-IDF and said why. The idea was right. The compute was not there.
What a round could not give them: compute and a protocol
After each run we took the product, wrote a protocol before touching anything, and ran the agents' own ideas on a GPU. Selection only on train and dev. The sealed slice scored once at the end and labelled as a second look, because the run had already seen it.
| product | in-run number | after our pass | how |
|---|---|---|---|
| tariff codes, top-1 | 0.360 | 0.557 | full crawl of 215k rulings, mapping old codes to current ones, a reranker within the tariff heading |
| tariff codes, top-5 | 0.563 | 0.802 | same; this one clears the run's 0.80 gate |
| clause finder, macro AP (cross-validated) | 0.468 | 0.606 | the fine-tune they tried, with a GPU: three Legal-BERT models stacked with TF-IDF |
| sanctions, org dropped-word recall on a never-seen list | 0.745 | 0.851 | a small cross-encoder that rescores candidate names |
Where the gains came from is worth saying plainly. For the tariff model, more than half of the improvement was data, the crawl the agents had not finished plus mapping retired codes to current ones, and the rest was modelling. For the clause finder, it was almost entirely the model they could not train in a round. The in-run TF-IDF path still ships as the fallback. For the sanctions screener, the gain is on a slice of 47 names, and on a second blind list of 16 the new model tied the old one. It stays off by default.
Where the models stopped
Each product hit a wall that looked different from the agents' side than from ours.
The tariff model plateaued at about 0.56 top-1. We checked a hundred wrong-chapter misses by hand: 44 model errors, 47 descriptions that genuinely fit more than one code, 8 too thin to classify. The ceiling for this data is around 0.9 at the chapter level, and the practical answer is not a better model but asking the user a clarifying question when the top candidates straddle headings.
The clause finder's wall was speed. The three-model stack took 11 seconds a contract on CPU. Converting it to int8 ONNX brought that to 8 seconds at the same accuracy and cut the image from 4.6 to 2.4 GB. A single model runs in 2 seconds but gives up a point of accuracy, so it is an option rather than the default. We have now looked at that sealed slice four times across the run and three post-run passes, and we say so in the README. It is not a blind number anymore.
The sanctions screener's wall was the test set itself. Recall on dropped-word variants collapsed from 0.91 on the US list to 0.27 on a Swiss list, and we had to spend that list to find out why. About half the organisation misses were artifacts of how the slice was built, translated acronyms and legal-form words the harness did not know. The person misses were real: the Swiss list is full of variants with an extra initial, a type that is 13% of the US list and 82% of the Swiss one. After fixing the harness, there was no clean list left to prove the reranker wins. So it does not ship as the default.
What a week of this costs
No API spend. Both agents run on their normal subscription plans. Grok's plan has a credit that resets weekly, and two eight-round runs in one day exhausted it, so it is one run per week now, and the orchestrator pauses the run instead of letting Grok's turns fail silently when the credit is gone. Our own fine-tuning passes used about four and a half GPU hours on a single consumer card. The harness bug of the week: a variable scoping mistake meant the expensive Grok model ran every round of run 5 instead of only the research round, which is probably what burned the credit in run 4 as well.
Run 6 had the best product and the least trustworthy number. Run 7 had the plainest product and the only number we would put in front of a buyer. The brief got us products. The gates got us numbers. The sealed slice, scored once, is the only thing that told us which numbers to believe.