Optimizing 3D Scene Reconstruction with Differentiable Ray Tracing

Abstract

A fundamental limitation of classical rendering pipelines is that their discrete, non-differentiable structure makes them unsuitable for gradient-based parameter recovery. This paper presents a differentiable ray tracing framework that directly addresses this gap, enabling end-to-end optimization of object appearance, illumination, and viewpoint parameters from observed images. The framework is built around automatic differentiation, specifically the source-to-source AD mechanism provided by Zygote in Julia. In our experiments, AD-based gradient computation achieves a 60% reduction in wall-clock time compared to central finite differences for scenes with more than 50 optimized parameters, while reducing gradient approximation error by approximately two orders of magnitude. We validate the approach through controlled inverse rendering experiments. These results open practical pathways for applications in neural rendering, differentiable simulation, and broader inverse graphics research.