MRIo3DS-Net: a mutually reinforcing images to 3D surface RNN-like framework for model-adaptation indoor 3D reconstruction

Abstract

Currently, deep neural networks (DNNs) have made some progresses in local region feature extraction, image matching under difficult conditions and point cloud surface optimization in photogrammetry and computer vision. However, there have been being no reports on an end-to-end high-accuracy 3D reconstruction network that integrated point cloud generation (i.e., dense image matching) and 3D surface optimization. Moreover, deep-learning-based 3D reconstruction networks are often difficult to apply to new dataset tasks. Therefore, this paper is the first to propose an end-to-end framework of mutually reinforcing images to 3D surface recurrent neural network (RNN)-like (MRIo3DS-Net) for model-adaptation indoor 3D reconstruction, where multi-view dense matching and point cloud surface optimization are mutually reinforced by a RNN-like structure rather than being treated as a separate issue. The characteristics and advantages of this framework are as follows: (1) In the multi-view dense matching module (MVDMM), the model-adaptation strategy is used to fine-tune and optimize a Transformer-based multi-view dense matching DNN, so that it has the higher image feature for matching and detail expression capabilities; (2) In the point cloud surface optimization module (PCSOM), the 3D surface reconstruction network based on 3D implicit field is optimized by using model-adaptation strategy, which solves the problem of point cloud surface optimization without knowing normal vector of 3D surface. Especially, to improve and finely reconstruct 3D surfaces from point cloud, smooth loss function with an edge-preserving strategy is proposed and added to this module; (3) The proposed MRIo3DS-Net is a RNN-like framework, which utilizes the finely optimized 3D surface obtained by PCSOM to recursively reinforce the differentiable warping for optimizing MVDMM. This refinement leads to achieving better dense matching results, and better dense matching results leads to achieving better 3D surface results recursively and mutually. Hence, model-adaptation strategy can better collaborate the differences between the two network modules, so that they complement each other to achieve the better effect; (4) To accelerate the transfer learning and training convergence from source domain to target domain, a multi-task loss function based on Bayesian uncertainty is used to adaptively adjust the weights between the two networks loss functions of MVDMM and PCSOM; (5) In this multi-task cascade network framework, theoretically, the two aforementioned modules possess the inherent potential to be replaced by arbitrary corresponding state-of-the-art (SOTA) networks to achieve better 3D reconstruction results. We tested performance of MRIo3DS-Net in the DTU, Tanks and Temples datasets by comparison experiments and ablation experiments with SOTA networks. The results show that MRIo3DS-Net has achieved the best performance and can reconstruct various indoor scenes reliably, completely and accurately. MRIo3DS-Net can provide a new end-to-end framework for high-accuracy indoor 3D reconstruction and has certain potential for the new target domain task.