Cole Medin's AI Software Factory turns GitHub issues into merged pull requests

Cole Medin has published AI Software Factory, a repository that installs into an existing project and turns filed GitHub issues into merged pull requests with no human involved. It reached 112 stars in its first seven days on GitHub.

What the factory actually does

The install writes a runtime, a workflow pack, skills, and a governance template into a target repository. It changes no application code. A scheduled loop starts on a timer, reads the state of open issues, plans the change, has an agent build it, and runs an evaluator that did not write the code. When every structural check passes, a script reads a verdict file and merges the pull request. The README calls the automation the easy half; everything else in the repository exists so that a merge nobody read can be justified.

The three files a user writes

Every install starts with three files the user has to author. MISSION.md states what the product is and what it must never become. The out-of-scope list is what the agent uses to recognise a plausible request that moves away from the mission. harness/END-TO-END.md is two to five user journeys, written in plain English, that an agent runs against the running app on every validation. HOLDOUT.md is the same product described a second time, in a directory the builder cannot read, which the author calls the only honest reason to merge code no human reviewed.

The autonomy levels

Levels run from zero, where workflows exist but a person runs them by hand, up to five, where the factory writes its own issues from the mission. Level three is the first level where code merges without a human reading it. A doctor script refuses to raise the level to three until the evidence supports it: real journeys, a holdout, a mutation set shown to detect errors, and a notification channel that can reach a human.

What is enforced in code

The merge itself is a script that reads a verdict file, so an ordinary if statement makes the decision. Two strings must appear in the output before anything counts as a pass: proof the app started and proof end-to-end tests ran. A pull request touching governance, the harness, or the holdout is auto-rejected, and the validator reads its rules from the base branch so a PR cannot weaken the rules it will be judged by. An earlier version once replayed one rejected pull request 68 times in three and a half hours; the current watchdog stops the run on seven kinds of stuck behaviour.

Cost, in one comparison

The README cites one comparison on one task. A solo agent produced a non-functional result in about twenty minutes for single-digit dollars. A planner, generator, and evaluator harness where the evaluator drove the live page produced a working result in about six hours for roughly twenty times the cost. The advice is to measure token spend from the first day, since projections tend to be wrong by ten to twenty times.

The evidence each merge depends on

Most autonomous coding setups released in the last year stop at generating a pull request and rely on a person to press merge. This one says that step leaves a person as the slowest part of the process. The interesting change is moving the design work from writing a better agent to writing the specific evidence a merge has to be based on: a mission with real exclusions, a holdout the builder cannot see, and a mutation set the assertion counts have to detect. A team that adopts it gets that as a checklist.

Source

Primary source: coleam00/ai-software-factory on GitHub.