When LLMs Pass Tests but Fail the Process: Lessons from Governing a Multi-Agent Software Development Project

Abstract

Software teams increasingly use large language models (LLMs) to plan, implement, and verify changes, but a passing test suite does not establish that the team built what was intended. This experience report describes an independent Ruby on Rails benchmark project in which a human principal investigator worked with an LLM Architect and an LLM Coder. Across 28 prompt-driven cycles covering nine algorithm families, the project retained prompts, results, errors, corrections, and commits. The most consequential failures were not syntax errors and did not necessarily cause tests to fail. Incidents included an unapproved substitution of a heuristic for an exact objective, a comparison that checked the wrong property, reference assumptions that failed when executed, context loss, and a user-interface regression missed by automated tests. The project responded with nine operational controls covering decision authority, stop-and-ask checkpoints, independent challenge, executable reference checks, session initialization, completeness evidence, visual verification, and commit attribution. Some later algorithm families completed in one cycle, while other problems were detected during premise checking, implementation, or visual review. Task differences, learning effects, one human evaluator, and one model pairing prevent causal or general claims. The practical contribution is a reusable governance playbook: define who may decide what, require evidence before implementation, match verification to the artifact, preserve state outside chat, and make consequential checkpoints observable. Because the setting was an independent project rather than an industrial deployment, these lessons are operating hypotheses for practitioners to adapt and evaluate.