Code-MUE: Measuring Code LLMs’ Uncertainty through Execution-Based Semantic Interaction Graphs

Abstract

As Code Large Language Models (LLMs) become central to modern software engineering, their inherent stochasticity poses significant real-world risks, where even minor errors can lead to severe functional, security, or safety consequences. Reliable automation, therefore, demands the ability to distinguish between confident, well-supported predictions and stochastic guessing. However, existing uncertainty estimation methods face a critical gap: white-box and grey-box techniques are often inapplicable to closed-source models, while standard black-box text metrics fail to capture the unique fragility of code, where syntactic variation does not necessarily imply semantic divergence. To bridge this syntax-semantics gap, we introduce Code-MUE, a purely black-box framework that measures uncertainty through execution-based Semantic Interaction Graphs. Unlike prior approaches that rely on superficial textual similarity, Code-MUE grounds uncertainty in observable runtime behavior and calculates the von Neumann entropy of the solution space to quantify global semantic diversity. A large-scale empirical study across eight state-of-the-art LLMs demonstrates that Code-MUE achieves a strong negative correlation with functional correctness, with Spearman’s correlation reaching up to −0.98. It significantly outperforms lexical and embedding-based baselines while enabling robust risk detection and selective prediction in practical workflows.