Keeping Score: what a year of recording our own defects taught us about evidence for automated decisions

Abstract

An operator applying serious effort to checking his own work, with tooling built for the purpose, found fewer than half of his own defects without help. This paper reports that record: ninety-five numbered items over a year, a taxonomy of ten species of defect, and the share found by other people — published as a range, 55.8% to 56.8%, and as a floor rather than a measurement, because the three-way classification the figure is computed over does not cleanly partition the population. One item's origin is not established and is drawn rather than absorbed into a neighbouring class. The paper is offered as a hypothesis a reader can test against their own record. One operator's record cannot establish a general fact about self-attestation, and this paper does not claim to. Six contributors are named with consent recorded for this surface specifically, on the principle that consent is bound to a surface and does not travel between documents. Three of them narrowed a criticism of this project in this project's favour. The unifying principle of section 5 is quoted from the reviewing body rather than the author, and the chronology — taxonomy first, principle afterwards — is stated so the principle is not made to look as though it organised the work from the beginning. Stated limits Shango MID is pre-revenue: no customers, no delivered engagements, no production deployments. Counter-signing is not running in the current build. The counter-signed corpus is historical. The connector is a reference implementation, not a live integration with a system of record. The implementation is closed. A third party cannot presently re-execute the conformance results independently. Nothing here is a certification, endorsement, regulatory approval, source-code audit, security assessment or production validation. The external governed finding referenced is PARTIALLY SUPPORTED — UNCHANGED, bounded to one frozen claim on one date. Registration is not certification. Integrity Source SHA-256, over LF-normalised markdown: bbd9aff0f0d39f6abd5e6d55caa712cb05efba75c6c38d857b8307ec59afcdbf The PDF digest is a SHA-256 over the raw deposited file with no normalisation, and is printed in the deposit notes. The two are not interchangeable.