Augmentation Helps but Single-Run Metrics Mislead: A Paired, Job-Level Audit of an Automated LLM Fine-Tuning Platform

Abstract

Hosted platforms now fine-tune large language models on a user's data and report success through single numbers: a dataset quality grade and a pairwise win rate against the un-adapted base model. We audit one such platform, anonymised, using its own per-job records for all 384 fine-tuning jobs (371 succeeded) that we ran on 21 datasets in July and August 2026. In a paired within-seed design that holds dataset, base model and recipe fixed, adding 2,000 synthetic domain rows raised the held-out win rate on 10 of 10 seeds, by a mean of 13.4 points (seed-level bootstrap 95% CI 11.7 to 15.1; exact sign test p = 0.0020). The numbers a user sees by default are less trustworthy. Runs with identical configurations vary with a pooled standard deviation of 2.75 points, so two single runs must differ by about 7.6 points before the difference exceeds noise, and reporting the best of 8 to 16 identical runs adds 2.1 to 6.3 points. The displayed on-dataset win rate exceeded the held-out rate in 159 of 187 jobs and ordered 44.2% of same-dataset job pairs differently from it. The reported quality improvement was largest (+32% to +38%) on the 6 datasets whose prompts referred to a chart they did not contain, where the platform had rewritten 35.9% of target completions into statements that no chart was provided. We correct nine statements of our earlier preprint against these records and give an audit protocol for such platforms. Version 2 (11 October 2026) replaces version 1 (14 August 2026, titled "Silent Failure in Automated Model Adaptation"). Every statistic is now recomputed from the platform's own per-job records for all 384 jobs, and nine statements of version 1 are corrected, three of them substantially. Section 8 of the manuscript lists each change. The main ones: Paired augmentation effect: 10 of 10 seeds, mean +13.4 points (seed-level bootstrap 95% CI 11.7 to 15.1; exact sign test p = 0.0020). Version 1 reported 11 of 11 seeds, +14.4 (12.4 to 16.4), p = 0.0010; two seeds' augmented arms had been assembled from other doses or pooled configurations. Scope: 384 jobs on 21 datasets, not 176 jobs on 13. Refusals: on the six datasets whose prompts referred to a chart they did not contain (a defect in the author's own seed data, disclosed), the platform rewrote 35.9% of target completions into statements that no chart was provided; elsewhere 0.05%. Version 1 said about one in ten. Run-to-run variation: pooled standard deviation 2.75 points over 148 identical-configuration runs. Held-out evaluation fired on 57.7% of the 324 retrieved jobs, not 70%. The displayed win rate was below the held-out rate in 28 of 187 jobs (version 1 said never) and ordered 44.2% of same-dataset job pairs differently from it. Files. The manuscript (PDF), and an archive holding the anonymised per-job records (job census, held-out and displayed win rates, evaluation domains, launch configurations), the export audit, the refusal-validation sample and labels, the generated statistics, the analysis code, a README and SHA-256 checksums. The platform is not named; its vendor received the findings, with job identifiers, before version 1 was posted. The exported datasets are not redistributed because their names identify the platform; they are available from the author on reasonable request.