It Ran Is Not It Was Right: A Policy-Conditioned Benchmark of Ten Text-to-SQL Architectures on an Enterprise Database

Abstract

Enterprise text-to-SQL assistants are usually judged by whether the SQL runs. On a realistic company database that is misleading: a query can run and still be wrong, return rows the asker may not see, ignore a rule in a policy document, or answer a question that should be refused. We build a testbed (59-table oil-and-gas database with role-based access, an approval workflow, nine policy documents) and a gold set of 94 questions labelled with the correct answer, the asking role, and whether to answer or refuse. Scoring ten published architectures on answer correctness, data leakage, policy grounding, and refusal overturns the execution-success ranking: the declared winner is ninth of ten on correctness, every architecture leaks restricted data (37 to 52 percent), almost none refuse, and the best is right about half the time. Results are stable over three runs, and the winner's validator gate does not fix any of these failures. Data is synthetic. We release the testbed, gold set, and scorer.