Beyond LLM-as-a-Judge: Independence-Aware Heterogeneous Evaluation with Conflict-Gated Verification

Abstract

Recent empirical studies reveal that large language model (LLM) evaluator ensembles exhibit substantial correlated errors, demonstrating that the independence assumptions underlying classical Condorcet aggregation do not hold for LLM panels. A diverse nine-judge frontier panel has been shown to provide information equivalent to approximately 2.2 independent votes. Furthermore, highly self-consistent frontier models often exhibit weak consistency-correctness correlation, and instruction-tuned judges collapse toward random-guessing on objective benchmarks like JudgeBench when stylistic confounds are controlled. Building on these findings, we propose a structural shift from monolithic semantic scoring to heterogeneous, multi-paradigm evidence aggregation. We formulate an independence-aware Evaluator Conflict Score (ECS) that dynamically weights epistemic disagreement across probabilistic semantic, deterministic symbolic, programmatic, and retrieval-based verification mechanisms. We present a conflict-gated cascading architecture that treats disagreement as a trigger for investigative escalation, ensuring that evaluation rigor scales proportionally to uncertainty.