Exploring the Possibilities of (Near) Real-Time Semantic Segmentation with 3D Point Cloud Data and Effective User-Centric Visualizations for First Responders

Abstract

First responders operate in complex and dynamic indoor environments where accurate real-time spatial information is crucial for situational awareness and decision making. The aim of this study is to explore the possibilities of (near) real-time segmentation of 3D point cloud data using state-of-the-art deep learning models and evaluate different visualization techniques to improve the situational awareness of first responders in indoor environments.

The study first evaluates nine segmentation models, considering key factors such as accuracy and inference speed. Although models like Point Transformer V3 + PPT and Point-SAM achieve high segmentation accuracy, real-time performance remains a challenge, particularly on personal devices. Applying these models to self-acquired point cloud data revealed not just preprocessing needs, but deeper compatibility issues. In practice, the entire point cloud was labeled as “clutter,” likely due to a combination of model limitations and the author’s limited programming experience, highlighting key barriers to real-world deployment.

Beyond segmentation, this research applies cartographic principles and cognitive theories to develop visualization techniques for effectively communicating the segmented point cloud data. Although the concept of (near) real-time segmentation serves as a guiding principle for this research, the achievement of (near) real-time segmentation of self-acquired point cloud data was not achieved. To address this, the self-acquired point clouds were manually segmented to approximate the expected model output, allowing the evaluation of different visualization techniques. Several proof-of-concept visualizations were created, testing different color schemes and levels of detail to assess their impact on interpretability and situational awareness. The findings indicate that structured visualizations, particularly those using functional color schemes, which assign colors according to their functional significance (e.g., green for floors, yellow for doors, red for hazards, dark gray for barriers), improve situational awareness and decision-making, whereas excessive complexity hinders usability in high-pressure scenarios.

The results highlight the progress of deep learning in indoor segmentation while also emphasizing the need for improved data integration and processing workflows. Furthermore, a single structured visualization approach was preferred over role-specific adaptations, reinforcing the importance of clarity, simplicity, and user-centered cartographic design in operational contexts. In the future, optimizing realtime processing and refining visualization techniques will be essential to improve situational awareness for first responders in critical situations.