A totally unscientific comparison of Spec Kit building an RTS twice
The same game built twice, one run led by Product Forge and one by a richer PRD. Each run had its own strengths, the two retrospectives disagreed on the basic figures, and what I actually learned was the value of iteration.
Before we get too carried away with this post I must tell you that whilst I’m a fan of spec-driven development, I’m no Spec Kit expert. It’s something I’ve been eager to have more of a play with to understand and get to grips with how to make the most of it.
Last week someone I used to work with (Hi Sandra 👋) reached out on a slightly random post I’d written about creating a PRD for Mine Sweeper, asking me how my hobby of making games had evolved in the age of AI. To be honest, I’ve not really applied any of the things I’ve learned for web development onto games development. The most I’ve done is to augment how I use Godot to help me understand some of the concepts in games development a little more - understanding the patterns that exist to create particular systems.
But this weekend I thought, why not give Spec Kit a whirl at being the backbone of making a game - especially after I couldn’t find a simple RTS to play to while away a few minutes in the evening. I was a huge Command and Conquer fan back in the day.
I ended up running a completely unscientific experiment involving creating the same game twice. Both with Spec Kit, but the first attempt used Product Forge to help me round out the idea. It’s a plugin that wraps Spec Kit in a much longer product lifecycle, putting problem discovery, research, a product spec and wireframes ahead of the planning and implementation you’d get on your own, with an approval gate at the end of each phase. Whilst the second led with a more in depth PRD. Both runs drew their art from Kenney’s Sci-Fi RTS pack, which is CC0 and saved me from having to think about art at all.
The short version of this is that I feel the key thing I learned in this process was the value of iteration. That, whilst I tried to keep the two experiments independent from each other I couldn’t help but have the second attempt benefit from the first. And that each run had its own strengths - first run had stronger specifications, the second run had a better vision for what the game should be which led it to be more fun - that would greatly benefit from a third run combining those learnings.
If you want to jump in and play the two games here’s the links RTS 1 and RTS 2. The code, the specs and the write-ups the two Claudes produced along the way are all in the repos, dp-lewis/rts and dp-lewis/rts-2, if you’d rather poke around in those.
What I tried, and how it went
Run one, with Product Forge
| What I tried | What I expected | How it went |
|---|---|---|
| Letting Product Forge do the product thinking | It would round out a half-formed idea better than I could on my own | Thorough specs, solid foundations, and a game I didn’t enjoy |
| Reviewing the specs at each phase gate | This would be where I steered it | Too much to get through. I approved most of it on faith |
| Waiting until it was playable before forming a view | The natural point to look | Eighty per cent of the tasks were already done |
| Changing my mind once the spec was locked | A quick tweak | Three formal change requests against requirements marked Must |
Run two, with a PRD
| What I tried | What I expected | How it went |
|---|---|---|
Writing an AGENTS.md from scratch first | Shorter replies. I mainly wanted less to read | It did that, and then turned out to be the most useful artefact of the weekend |
| Building the PRD as an HTML page with the real sprites in it | A better-looking spec | The diagrams and images made the direction clear. The first thing in either run I could have an opinion about before there was code |
| Telling it to proceed unless genuinely unsure | Fewer interruptions | Seven slices, two questions, both about how the game feels |
| Banning tests on rendering, input and UI | Less brittle noise slowing us down | Faster. Both Claudes later marked the repo down for it |
| Playtesting each slice rather than at the end | A sanity check | Four of five real bugs came from playing. None came from the test suite |
Run one, and the faith problem
I finished the first run disappointed.
Not because nothing happened. A great deal happened, and most of it is work I’d struggle to justify a weekend on by hand: A* pathfinding, a deterministic simulation with a replay corpus running across four CI runners, a balance pass tuned to a measured six to ten minute match. It’s the better-engineered of the two by some distance.
The problem was that I couldn’t keep up with it. Problem discovery, research, a product spec, wireframes, journeys, two architecture decision records, 82 tasks across ten milestones. There was simply more to review than I could get through, so at every gate the honest answer was that I was approving it on trust. I could see the documents agreed with each other. I had no way of telling whether they described a game worth playing.
The first moment I could actually apply judgement was when there was something on screen to play, and by then about eighty per cent of the tasks were done. So the three things I wanted changed all came through as change requests: a tech tree, because the Factory I’d spent 200 ore on turned out to do nothing at all; scattered ore, because two nodes in the middle of the map meant nobody ever needed to move; and a sudden death rule, because when the ore ran out everything just stopped without anybody winning.
Somewhere in the middle of that I said the thing I keep coming back to:
I think I specced it too soon.
What I wrote down before starting again
Before the second run I wrote an AGENTS.md from scratch. Not from a template, and not with any grand intent behind it. I mainly wanted Claude to say less, so that I stood a chance of keeping up with what it was telling me.
So it starts there. Default to concise, under ten lines a turn, tables rather than paragraphs, decisions at the top. With the failure mode named directly:
layers of generated documents where the real decision is lost in paragraph four of section three
Default to proceeding, and stop only when genuinely unsure, with anything that changes how the game feels reserved as my call.
And then the line I’d now argue is the most valuable in either repo, sitting in the definition of done alongside the tests and the typecheck:
It runs. Launch the game and confirm the change works in the real app. “It compiles” is not done.
Reading it back, that document is basically run one’s post-mortem. Every rule in it is there because something in the first weekend got on my nerves.
Run two
The PRD came next, and building it was the part I enjoyed most.
We went through the Kenney pack properly rather than glancing at a contact sheet, working out what the art would and wouldn’t let us do. The units are drawn side-on and can’t rotate, so facing tells you nothing about what a unit is shooting at, which means readability has to come from target lines and muzzle flashes. The structures only ship in one colourway, so ownership needs a second cue beyond a tint. Neither of those had occurred to me before we looked.
It ended up as a generated HTML page illustrated with the actual sprites. Rosters, a structure dependency graph, the counter cycle, a picture of the opening position. I liked having something I could look at and fine-tune with Claude, but the bigger thing was that the diagrams and images let me see where we were heading. With a markdown spec I’d been assembling that picture in my head from a list of requirements, and doing it badly.
Then seven slices, each one plan, tasks, implement, each ending in something playable.
The scoreboard, and why I don’t lean on it
| RTS 1 | RTS 2 | |
|---|---|---|
| Source | 5,646 lines | 3,991 |
| Tests | 5,755 lines | 1,844 |
| Markdown | ~8,300 lines | ~6,100 |
| Determinism | Hash, replay corpus, CI | None |
| Pathfinding | A* | Straight-line |
| Balance | Tuned to a target | Never attempted |
| Fog of war, camera, minimap | Cut early | All present |
I asked each Claude to review the other repo, which was more interesting than I expected. Both of them preferred RTS 1 on the engineering, because determinism, pathfinding, test depth and a balance pass are all legible from inside a repository. Whether a game is fun isn’t.
The two retrospectives also disagreed with each other on the basic figures, and when I counted I got a third answer again. Very much in the spirit of the title.
The thing both runs found independently
Neither test suite caught its own worst bug. Both times I found it by playing.
In RTS 1 the Factory was specified in phase two to train units, and nothing ever routed a command to it. It survived seven milestones and two of the agent’s own code reviews as a 200 ore ornament. I found it in my first session, and the fix I asked for turned out to be what the product spec had said all along.
In RTS 2 three harvesters sat next to a refinery with full cargo forever. One slice set an arrival distance of four pixels. A later slice added unit separation with a radius of twenty-two. Three harvesters converging on one building shove each other about twelve pixels apart, so an arrival test needing four can never pass. 205 tests never saw it. I found it watching a crystal counter sit at exactly 50 for four minutes.
Across both repos the agent caught anything that came down to one number disagreeing with another. The bugs it missed were the ones you’d only find by wanting to play the game.
Where that leaves me
I said at the top that I couldn’t keep the two runs independent, and I want to be specific about why, because I think that’s the interesting part.
The second run had a better PRD, yes. It also had a working agreement written specifically to fix the first run’s problems, and it had me, having already been through the entire cycle once and knowing which questions actually mattered. Three variables, one confound, one participant, and the participant already knew which answer he was hoping for.
As a build, run one didn’t fail. Where it came apart was the collaboration: how much there was to review, how late I could first apply any judgement, how expensive it got to change my mind. Fail fast and learn is hardly a new idea, but it reads differently when the thing that failed is the working relationship rather than the code.
Looking back at the changes I made for the second run, they’ve all got something in common. Shorter replies because there was too much to read. A picture of the game because I’d been assembling one in my head from a list of requirements. Playing it every slice instead of at the end. On their own, none of them improved the work. They just kept me in the loop while it happened. Duller than it sounds, and probably the bit that matters.
Most teams can’t afford to build the same product twice. But the thing worth keeping from run one was a 132 line document about how to work together, and that came out of a day and a half of being mildly irritated.
I went back to RTS 1 after finishing the second game and liked it a bit more than I had on the Saturday night. Not enough to change which one I’d rather play, but enough to notice I’d judged it at the exact moment I was most sick of it.
If I were to do a run three, I’d take the PRD and the working agreement from the second run, the problem discovery and the balance target from the first, and playtest at milestone one rather than milestone nine.