Attribution Slows Model Collapse Whether or Not It Is True

Abstract

Language models trained recursively on their own output degrade. Every published mitigation works by importing information from outside the loop — blending in real human data, relabelling with a frozen external model, or detecting and filtering synthetic text. This paper tests an intervention that imports nothing: a source tag inside the model's own generated text. Thirty GPT-2 models were trained across five recursive generations in three lineages — sentences carrying their true source title, a false title, or no title at all — replicated across two independent seeds. Each generation trains only on the previous generation's output; no human data is reintroduced at any point. THE RESULT Every lineage degraded monotonically. None stabilised at any generation, including the one with attribution fully intact. total drift over five generations (nats, lower is better) provenance 0.4892 scrambled 0.5096 control 0.6481 Tagged lineages drifted 25% less than untagged and lost half as much lexical diversity. Seed-to-seed variation in total drift is 0.001-0.005, so these differences are several times the observed noise. THE TAG DOES NOT HAVE TO BE TRUE A lineage anchored entirely to false sources resisted collapse nearly as well as one anchored to correct sources. Decomposing the 0.1589 nats of total protection: structure — having any tag at all 0.1385 87% truth — the tag being correct 0.0204 13% The truth component is small but consistent in direction across both seeds and roughly four times the seed noise, so it is real rather than absent. The likely mechanism is structural: emitting a title before a sentence forces a commitment to a topic before generating, and that commitment keeps the output distribution from narrowing. Whether the topic is the correct one contributes little, because the scaffolding does not depend on correctness to function. WHERE ACCURACY MATTERS, AND IT COMPOUNDS Correct attribution does not protect the prose. It protects the sourcing. The share of emitted titles that actually exist in the source corpus, at generation five, was 65.2% and 67.5% for the truthfully-anchored lineage against 55.7% and 58.6% for the falsely-anchored one across the two seeds. That advantage is absent at generation one and widens monotonically — roughly +9 points by generation five in both seeds independently. Unlike the structural effect, this one is a property of recursion depth rather than of the tag. ATTRIBUTION INFLATES RATHER THAN COLLAPSING Naive collapse predicts narrowing. Attribution does the opposite. Generation zero contains 2,073 real source titles; generated corpora contain between 6,700 and 13,171 distinct titles, roughly a third of them fabricated by generation five. The models do not forget where content came from — they begin confidently citing sources that never existed, and invent more each round. A corpus that looks better-sourced every generation while its sourcing becomes fiction. Output also lengthens and stops closing: at a matched 128-token budget, well-formed single-sentence output falls from 98.3% to 51.6% while mean length grows from 170 to 312 characters. The failure mode is run-on text, not noise. An earlier analysis at a 64-token budget overstated this collapse by roughly fourteen points; the corrected figures are reported here. RELATION TO PRIOR WORK A structurally similar dissociation is established at inference time: replacing in-context demonstration labels with random ones degrades performance far less than expected, with format mattering more than label correctness (Min et al., 2022). This is a training-time analogue under recursive self-training. The 87/13 decomposition is, to our knowledge, the first quantitative split of the structural and veridical contributions in a recursive setting. The hypotheses tested here originate in a prior theoretical framework (Sweeney, 2025) holding that attribution is structural substrate rather than metadata. That framework's central thesis is supported, and in a stronger form than its own axioms state — its recursion axiom, which predicts that meaning strengthens through cycles when attribution is preserved, is falsified: nothing strengthened under any condition tested. LIMITATIONS All thirty fine-tunes use GPT-2 small; the recursion experiment was not repeated at larger scale, though the underlying single-pass conditioning effect strengthens from 124M to 774M (companion work). The truth component is established as non-zero but not precisely characterised. One corpus, one language, sentence granularity. Wikipedia lies within GPT-2's pretraining distribution. Tags are not re-corrupted between generations, so this measures the fate of a lineage descended from broken attribution rather than the effect of continuous corruption. Code and evaluation harness are included for full reproduction.