Classic 3D scene graph generation approaches fail to work in real-time due to the heavy computational cost of environment mapping and the need to generate intermediate point-cloud representations. To alleviate this issue, a recent work eschews point clouds in favor of a lightweight Gaussian distribution for each object. This approximation drastically speeds up inference and enables real-time 3D scene graph generation. However, the representation has two key weaknesses. 1) Each object is approximated by a single 3D Gaussian, which causes a severe loss of 3D geometric detail. 2) The discrepancy between this approximation and the true object geometry exacerbates the inaccurate merging of object candidates during online inference. To address these issues, we propose NoPA, which represents each object as a separate non-parametric distribution. This formulation retains 3D geometric information while preserving real-time inference of the parametric Gaussian formulation. To build upon our novel object representation, we propose a tailored merging strategy to recover coherent object instances. Specifically, we leverage maximum mean discrepancy on kernel density estimates to enable robust merging of object candidates during online exploration while minimizing added computational complexity. The key is to maintain a fixed particle set per object. Furthermore, to rectify the relation loss caused by misclassified objects, NoPA propagates relationships between objects with high affinity. Experiments show that NoPA substantially outperforms current methods without sacrificing real-time inference speed.
NoPA builds a 3D scene graph from a sequence of streamed RGB images. We used a pretrained 2D SSG model, RT-DETR-EGTR, to predict the initial 2D scene graph for each frame. We then lift the 2D SSG to 3D to obtain the local 3D scene graph for each frame. We represent each object as a non-parametric distribution of particles, which retains more geometric information than the previous parametric Gaussian representation. The local 3D scene graph is merged into the global 3D scene graph via a two stage merging process. We leverage fast and cheap pre-filtering from calculating the Hellinger distance between Gaussian approximations of the 3D object candidates for unambiguous cases. For ambiguous cases, we use the maximum mean discrepancy (MMD) between kernel density estimates (KDE) of the 3D object candidates to determine if they should be merged. To recover lost relationships due to misclassified or filtered objects, we propagate relationships between objects with high affinity.
FROSS generally struggles to correctly identify the wall class instances. FROSS still tends to omit relationships between the wall instance and other object instances during classification, even when the wall instance is correctly classified. Because of the initial incorrect classification, all the relationships that are predicted with the wall instance are misclassified or missing. Our method correctly classifies all objects while correctly predicting the majority of predicate relationships.
Demonstration of the qualitative results of our method. The videos show the 3D scene graph being built in real-time as the camera moves through the scene. Our method is able to correctly classify and merge object instances while maintaining some level of geometric detail.
@misc{yeo2026nopa,
title={{NoPA}: Non-Parametric Online 3D Scene Graph Generation},
author={Qi Xun Yeo and Seungjun Lee and Yan Li and Gim Hee Lee},
year={2026},
eprint={2607.00529},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2607.00529},
}