Scene graph generation from point cloud panoptic segmentation for indoor scene understanding

Abstract

Abstract Indoor scene understanding enables machines to comprehend 3D indoor environments by capturing object relationships and supporting spatial reasoning. Scene graphs are commonly used to represent objects and their relationships in a node–edge structure describing how neighboring objects are connected within a scene. The current scene graph generation methods often suffer from ambiguous object-edge labeling caused by imbalanced training samples and unknown multi-objects identification. Hence, a panoptic segmentation derived from an unified transformer segmentation framework is proposed to retrieve objects and their attributes. This information is then used to generate the nodes and edges of the scene graph. Subsequently, semantic labels for predicted instances are obtained using a majority label voting mechanism, while an edge completion network predicts relationships among neighboring object instances. The proposed network is evaluated using the ScanNet and ScanNet++ datasets. The results show that using a unified transformer network to generate object label predictions prior to scene graph generation yields an improvement in Panoptic Quality of +29.74% on the ScanNet dataset and +22.87% on the ScanNet++ dataset.