Spatialsense AI (Echomap): Persistent 3d Spatial Memory and AI Voice Navigation for Visually Impaired Users

Author

Ms. Hemalatha, Divya, Baranisha, Akshaya, Bheeshma Sri

Abstract

 SpatialSense AI (EchoMap) is an assistive software system that helps blind and visually impaired people understand their surroundings. Most existing vision applications identify objects in a single camera frame and then forget them. The proposed system detects objects with YOLOv8, tracks them across frames with ByteTrack/DeepSORT, places them in a lightweight room map using a mini-SLAM module, stores what it has seen as a persistent scene memory, and reports changes through a voice assistant that answers in the user’s own language.

The system answers questions such as “What is in front of me?” and “Where is my bag?”, detects objects that are new, moved or missing since the last scan, and clearly separates what is visible now from what was only seen earlier. Voice interaction is supported in Tamil, English, Telugu and Hindi. SpatialSense AI is intended as a supplement to, and not a replacement for, a white cane, guide dog or other established mobility aids.

Keywords

assistive technology, visual impairment, object detection, YOLOv8, object tracking, SLAM, scene memory, voice assistant, multilingual speech.

Full Text:

Download Paper PDF

References

[1] Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. You Only Look Once: Unified, Real-Time Object Detection. Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

[2] Jocher, G., Chaurasia, A., & Qiu, J. Ultralytics YOLOv8. Software, 2023. https://github.com/ultralytics/ultralytics

[3] Bewley, A., Ge, Z., Ott, L., Ramos, F., & Upcroft, B. Simple Online and Realtime Tracking. Proc. IEEE International Conference on Image Processing (ICIP), 2016.

[4] Wojke, N., Bewley, A., & Paulus, D. Simple Online and Realtime Tracking with a Deep Association Metric. Proc. IEEE ICIP, 2017.

[5] Zhang, Y., Sun, P., Jiang, Y., et al. ByteTrack: Multi-Object Tracking by Associating Every Detection Box. Proc. European Conference on Computer Vision (ECCV), 2022.

[6] Mur-Artal, R., Montiel, J. M. M., & Tardós, J. D. ORB-SLAM: A Versatile and Accurate Monocular SLAM System. IEEE Transactions on Robotics, 31(5), 2015.

[7] Cadena, C., Carlone, L., Carrillo, H., et al. Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age. IEEE Transactions on Robotics, 32(6), 2016.

[8] Kuriakose, B., Shrestha, R., & Sandnes, F. E. Tools and Technologies for Blind and Visually Impaired Navigation Support: A Review. IETE Technical Review, 39(1), 2022.

[9] Messaoudi, M. D., Menelas, B.-A. J., & Mcheick, H. Review of Navigation Assistive Tools and Technologies for the Visually Impaired. Sensors, 22(20), 2022.

Share your valuable work from Social Media Buttons