Does the Router Know? Mixture-of-Experts Routing Signals for Detecting Silent Errors in Text-to-SQL over University Data Warehouses

Abstract

Natural-language assistants can open institutional data warehouses to non-technical staff, but their most harmful failures are silent: a well-formed query that runs and returns a plausible, wrong number. We study these errors on an Italian university's live data warehouse, using an on-premises research prototype built on the open-weight Mixture-of-Experts model gpt-oss-120b, and ask whether the model's routing decisions reveal them. We built a benchmark of 1000 Italian questions from 51 templates, with references checked against official reports and semantic conventions recorded explicitly. The assistant answered 70-77% of evaluable questions correctly and returned a silent error on 19-23%; hand-written guards did not reduce silent errors, and many depended on undocumented institutional conventions rather than on the model. From the accepted queries we extracted per-layer router statistics and trained probes validated across question templates. Token-level confidence was at chance (AUROC 0.51), and routing features averaged over layers (0.56) were no better than a control that only recognises the question; kept per layer, routing reached 0.75 and added information to question metadata (0.77 combined). Delivering only the least suspicious half of the answers reduced silent errors from 21% to 10%. Routing signals offer a cheap, internal basis for triaging answers on institutional data. Preprint of a manuscript submitted to Knowledge-Based Systems (Elsevier) on 8 October 2026. Supporting data and code are deposited separately: https://doi.org/10.5281/zenodo.23292481