Does a Synthetic Aircraft Design-Space Simulator Predict Real Competition Outcomes? A Case Study on the DARPA Lift Challenge

Abstract

This paper evaluates whether a synthetic aircraft design-space generator and mission simulator, built for the DARPA Lift Challenge (a $6.5M competition requiring vertical-takeoff aircraft to lift at least four times their own weight), agrees with what happened when roughly 100 real teams attempted the challenge. Only five teams completed a scored course; most of the rest crashed or failed to finish. A sixth aircraft, DefendTex, is also analyzed as the one crash with enough public data to reconstruct. All six real aircraft are reconstructed from public photographs, video, and social media as inputs to the simulator's own, unmodified mission-physics engine. Naive point-estimate reconstructions produced a striking but spurious finding: a perfect inverse correlation between simulated reliability and real placement. This is traced, via controlled ablations and a Monte Carlo sensitivity analysis, to inconsistent treatment of battery energy capacity across the reconstructions, not to the simulator itself. After correcting this, fine-grained ranking agreement with real placement does not hold (Spearman correlation -0.30, n=5, not significant). A more modest result instead emerges: benchmarked against an unfiltered random sample from the simulator's design population, all five real, scored competitors land in the 70th-77th percentile, solidly above average. The simulator further supports design improvement: comparing high- and low-performing simulated designs identifies landing-gear robustness relative to descent rate as a real, controlled-verified reliability lever, while a promising cruise-speed correlation does not replicate under controlled testing. Methodological defects are identified and reported rather than concealed. The simulator is a defensible screening tool for design viability and a source of testable design guidance, but not a predictor of fine-grained competition placement.