A Golden-Set Evaluation Harness for Multimodal LLM Document Extraction Pipelines: Design and Failure Analysis in a Public Procurement System

Abstract

Multimodal large language model (LLM) pipelines are increasingly used to extract structured information from unstructured regulatory documents, such as public tender files (dossiers d'appel d'offres, DAO). In production, engineering teams need a repeatable way to judge whether a change to a prompt or to the extraction pipeline improves or regresses output quality, rather than relying on ad hoc manual review of one or two documents. This paper describes a golden-set evaluation harness built for an LLM-based DAO extraction pipeline operated by EBIDDER AI, a public procurement assistanceplatform in Benin. The harness compares pipeline output against hand-validated expected results on a field-by-field basis, applying strict equality rules to business-critical fields (lot counts, guarantee presence, validity-period nullity, cross-category contamination) and tolerant, normalized comparison to free-text fields. We report results from three real cases in the golden set. Two cases matched the expected output closely, with one revealing a subtle truncation in a free-text field that manual review would likely have missed. We present the harness design, the real failures it caught, and an honest account of what remains unverified, and argue that distinguishing "genuinely absent" from "extraction failed" in pipeline output is a concrete next step for reliability.