Re-imagining sensory substitution through gestural control: Point-To-Tell 2. - Ruan, Ligao; Hamilton-Fletcher, Giles; Beheshti, Mahya; Hudson, Todd E.; Porfiri, Maurizio; Rizzo, John-Ross (Disability & Rehabilitation: Assistive Technology, 2026)
Aim: Sensory substitution devices (SSDs) can convert environmental information into an accessible format for people who are blind or have low vision (pBLV). Yet, current SSDs often passively deliver all of the information available with limited user control, potentially leading to confusion and/or cognitive overload. To address this issue, this work proposes a selective, gesture-controlled system intended to improve information relevance and reduce cognitive overload. Methods: We present Point-To-Tell 2, a system that enables pBLV to privately and efficiently select which information to convey through simple pointing-based gestural control. By integrating a monocular camera with AI-driven pipelines for depth estimation, hand pose tracking, and object detection/segmentation, the system identifies the users' 3D pointing direction and announces the names and distances of objects as they are pointed at, thereby connecting an object's spatial position and identity through hand proprioception. Results: Validation tests in controlled indoor environments show high hand pose tracking accuracy, ensuring reliable ray-casting and object selection despite declining object recognition at longer distances. Distance estimates are stable at close range, though a systematic bias is present. Conclusion: This work introduces and technically validates an assistive system designed to improve the usability of assistive technologies by focusing system feedback—potentially reducing users' cognitive load and enhancing their spatial comprehension by leveraging concurrent hand proprioception. Future work will involve user testing and expanding system features to further enhance its practicality across more diverse scenarios. IMPLICATIONS FOR REHABILITATION: Enhanced Spatial Awareness: By leveraging hand proprioception and accurate 3D ray-casting, the system is expected to help users develop more robust mental maps of the environment by building off an egocentric representation of space. Pending further validation, this potential improvement in spatial awareness may enhance visual rehabilitation for orientation, mobility, and awareness of key objects in the environment. Reduced Cognitive Load: This gesture-based method allows the user to focus the AI feedback on describing specific elements of a scene. This potentially allows the user to focus on relevant information to them, while minimising extraneous information. This could reduce the cognitive burden associated with systems that provide too much irrelevant information that the user must then parse through. Increased Independence in Daily Activities: The hands-free design of the technology should allow the user to be able to focus on interactions with the environment, rather than on manipulating hardware. This should make users more efficient, in being able to call on additional information through brief mid-air gestures, even in the middle of completing daily tasks. Integration with Personalised Rehabilitation Programs: The adaptive and context-aware nature of the system makes it a promising candidate for inclusion in comprehensive rehabilitation protocols. Further optimisation and experimental testing of the system will aim to tailor sensory feedback specifically to support user-centric learning and functional improvements while ensuring privacy safeguards and confirming efficacy through testing with persons who are visually impaired.