From Tool Selection to Autonomous Execution: Process Templates, Sandboxed Code, and Modular Orchestration for Reliable LLM Agents at Scale

Abstract

Large language model agents must route and execute tools from large, overlapping libraries. Prior work reported routing degradation under flat catalogs; we study both routing and execution under scale, noise, and multi-step control.Routing. Expert-disambiguating descriptions improve selection for some models and hurt others. The mean nearest-neighbor cosine similarity of target skills increases with catalog size N (Bare peak ≈0.51 at N=500 with MiniLM embeddings); detailed expert text remains more separable than short bare text. Live selection stays near-perfect for strong models on clean queries, while weaker models degrade at large N.Execution. On atomic tasks, bare prompts often fail structured output; expert process templates (SOPs) largely remove schema failures, and sandboxed Python execution removes arithmetic failures, producing large end-to-end (E2E) gains versus prompt-only math. On production file-based tasks, selection can remain high while execution E2E stays modest without sandbox support.Robustness. Under a noise gradient (L0–L5), mild noise is tolerated; adversarial injection collapses E2E for several models, and an AST script-integrity guardrail recovers a substantial fraction of L5 failures (e.g., ∼0%→∼47% on DeepSeek V4 Flash and GPT-4o-mini). A 2×2 query matrix (phrasing novelty × semantic ambiguity) shows high routing accuracy for strong models, with model-dependent weaknesses in harder cells. Orchestration. In a 30-task live multi-step Data Scientist stress test, partitioned Expert stages (N≤15 tools per stage) achieve 93%–100% E2E versus 47%–77% for a flat Bare catalog, with higher loopbacks under Bare. Volume projections for adjacent roles are calibrated simulations, not live API results. Together, these results support a practical recipe—expert templates, sandbox execution, stage-wise tool partitions, and light guardrails—without claiming a single universal cosine threshold or unlimited scaling as a theorem.