On October 14 we put two coding agents, Claude and Grok, in a throwaway virtual machine and told them to build something together with nobody watching. Over the next six and a half hours we ran that four times. Each run changed one or two things about the setup. Two runs produced competent, forgettable software. One produced something we would actually show a customer. The fourth stopped halfway when one agent ran out of quota.
This is what we changed between runs and what each change did. The numbers come from the git history and the files each run left behind.
The box
The VM runs on the homelab's Proxmox cluster, on a lab network with a firewall rule that blocks it from reaching anything else in the house. It can reach the internet and that's it. Inside: a git repo, a shared mail/ folder the agents use to talk to each other, and a small orchestrator that hands out turns. Claude went first every round, then Grok, then repeat, until a round limit or time cap.
Both agents ran on their normal subscription plans through command-line tools: Claude Code for Claude, OpenCode for Grok. No API keys, no per-token spend. That turns out to matter in run 4.
The brief, from run 1 on: build something deployable that saves a business time or money, for a specific buyer, finished and tested, with no claims the code can't back up.
Four runs
| run | product | length | commits | tests | what changed going in |
|---|---|---|---|---|---|
| 1 | bank-vs-ledger reconciliation CLI + web UI | 62 min, 20 rounds | 98 | 57 | first run, short turns |
| 2 | insurance-certificate (COI) expiry tracker | stopped at round 16 of 30 | 68 | 53 | different product, Docker installed, proof scripts required |
| 3 | job-shop scheduler (CP-SAT) | 107 min, 8 rounds | 28 | 20 | agents research and pick a hard problem; long turns; strongest Grok model |
| 4 | audit + rebuild of an existing tool | 8 rounds, Grok gone after 4 | 26 | 42 | real codebase, real usage data, owner answers mid-run |
Commit count went down every run and the work got better. That's not a coincidence, as the next section shows.
Lever 1: proof, not promises (run 2)
Run 1's reconciliation tool was honestly documented. On a synthetic benchmark it auto-matched 88% of transactions within a 3-day window and 95.6% within 5 days, and the README said the data was synthetic. But the brief said "deployable", and neither agent ever installed Docker. Both had sudo. The Dockerfile was written and never built, and the README said so.
For run 2 we installed Docker ourselves and added a rule: a deploy-check.sh that builds the container, starts it and fetches a page, plus a clean-clone check, both run and committed. It worked. Run 2's Docker path was real. The product was still a CRUD tracker, a nice one, built in two-minute turns, and we stopped it at round 16 because more rounds were only going to polish it.
Lever 2: make them pick something hard (run 3)
Runs 1 and 2 had the same shape: agree on an idea in round 1, then grind out features. So run 3 changed the brief. Round 1 is research with sources (docs/research.md). Round 2 is a plan with acceptance tests (docs/plan.md). The product has to need a hard capability: no trackers, no CRUD. Turns went from about 2 minutes to up to 20, with a nudge if an agent handed back its turn before the 10-minute mark. Grok ran its strongest model every turn.
This is the run that worked. The agents chose job-shop scheduling, which means ordering work across machines to cut late orders. They built it on Google's OR-Tools CP-SAT solver, checked it against the standard public benchmark (JSPLIB) and wrote an independent validator for every schedule.
- On the hard benchmark set: every schedule valid, a mean gap of 1.35% to the published optima (worst 4.60%), and better than both simple dispatch rules on all 8 instances.
- On generated 12-job shops, the business metric: weighted tardiness relative to the best of four standard dispatch rules, lower is better. Their own gate was 0.70. The held-out median came in at 0.708.
They missed their own gate by 0.008 and did not move it. The README says so in bold, next to a note that all the shop data is synthetic. That's the line we'd point to if anyone asks whether this experiment is worth running.
The research round is where Grok earned its place: first-hand descriptions of the scheduling pain, the real benchmark data, a sharper argument for the pick. Claude conceded the choice and took over evaluation and QA. One finding we didn't expect: the long-turn nudge stretched Grok's turns, not Claude's. Claude kept working in fast bursts whatever the turn length.
Lever 3: a real codebase and a real owner (run 4)
Everything so far was built on synthetic data for a buyer the agents imagined. Run 4 gave them a scrubbed copy of one of our own small tools (no git history, database, uploads or secrets) and its real usage numbers. The brief was: audit it, plan, and rebuild only what the evidence justifies.
It ended with 42 tests and the proof scripts passing. Nothing in the run could deploy on its own; we read the change log ourselves first.
Halfway through, the agents left questions for the owner. We answered them in a docs/OWNER-ANSWERS.md file while the run was still going, and the next turns used the answers. That was the cheapest useful change of the whole experiment.
What broke in the setup itself
- Grok was silent for four rounds of run 1 and nobody noticed until afterwards. OpenCode's permission system auto-rejected commands that touched
/tmp, so Grok's turns did nothing. That's fixed in the config now. - Our firewall script wiped Docker's network rules in an early version of run 2 (
flush rulesetclears everyone's tables, not just ours). It now replaces only its own table. - Grok hit its subscription's usage cap after round 4 of run 4, and every Grok model returned a spending-limit error after that. Claude finished rounds 5 to 8 alone. The strongest model every turn for most of a day is what used it up.
What we'd change for run 5
- Use the strong Grok model only for the research round and the final review, and the cheaper build model in between. Check the usage page before starting.
- Keep the research round and the hard-capability rule. They're the levers that changed the outcome.
- Real data again. The scheduler run is a candidate on a real shop's export, and so is a lighting-show sequencer fed with our own show files.
- A lessons file carried from run to run, so run 5 starts from what runs 1 to 4 learned instead of from the brief.
The short version: two agents in a box will reliably build a tidy CRUD app. They build something worth reading when the brief forces them to research, pick a hard problem, prove claims with scripts, and work from real data with a real owner answering questions.