Combining Formal Reasoning and LLMs for Scenario-Based Educational STEM Exercises

Abstract

Abstract Large language models (LLMs) enable scalable generation of educational exercises for science, technology, engineering, and mathematics (STEM) but often produce mathematically invalid or physically inconsistent problems, while manual authoring remains accurate but hard to scale. This paper introduces a neuro-symbolic framework for scenario-based educational STEM exercises, demonstrated through applications in introductory thermodynamics. We encode domain knowledge in first-order logic and use a satisfiability-modulo-theories (SMT) solver to enumerate all physically and pedagogically admissible exercise scenarios. We derive scenario-based knowledge graphs and employ symbolic reasoning to select solution paths and assign consistent numerical values before using an LLM solely for natural-language realization. In our case study, the framework reduces 512 naive attribute combinations to 12 valid thermodynamic scenarios and generates correct, solvable exercises per scenario with diverse solution paths. Compared to direct LLM-based generation, in which roughly half of the generated exercises were physically invalid or implausible, our pipeline avoids such inconsistencies by design. This suggests that scenario-centric approaches provide a promising foundation for transparent, human-centric AI in STEM education.