Re-imagining sensory substitution through gestural control: Point-To-Tell 2.
Aim: Sensory substitution devices (SSDs) can convert environmental information into an accessible format for people who are blind or have low vision (pBLV). Yet, current SSDs often passively deliver all of the information available with limited user control, potentially leading to confusion and/or c...
| Publicado en: | Disability & Rehabilitation: Assistive Technology Vol. 21; no. 4; pp. 1454 - 1473 |
|---|---|
| Autores principales: | , , , , , |
| Formato: | algorithm equations & formulas pictorial research tables/charts Journal Article |
| Publicado: |
Taylor & Francis Ltd
May2026
|
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ccm&AN=194897753&site=ehost-live header: @attributes: shortDbName: ccm uiTerm: 194897753 longDbName: CINAHL Complete uiTag: AN controlInfo: bkinfo: dissinfo: jinfo: jid: 17483107 1X04 jtl: Disability & Rehabilitation: Assistive Technology issn: 17483107 maglogo: Y pubinfo: dt: May2026 vid: 21 iid: 4 pid: 377 pub: Taylor & Francis Ltd place: Philadelphia, Pennsylvania artinfo: ui: 194897753 189439422 194897753 194897753 10.1080/17483107.2025.2590679 194897753 ppf: 1454 ppct: 19 formats: tig: atl: Re-imagining sensory substitution through gestural control: Point-To-Tell 2. aug: au: Ruan, Ligao Hamilton-Fletcher, Giles Beheshti, Mahya Hudson, Todd E. Porfiri, Maurizio Rizzo, John-Ross affil: Department of Mechanical and Aerospace Engineering, NYU Tandon School of Engineering, Brooklyn, NY, USA sug: subj: Vision Disorders Rehabilitation of Persons with Vision Loss Sensory Aids Utilization Body Language Cognition User-Computer Interface Task Performance and Analysis Human Funding Source Male Adult Descriptive Statistics Conceptual Framework Photography Data Analysis Software Feedback Hand Physiology Proprioception Validation Studies Detection Algorithms Adult: 19-44 years Male ab: Aim: Sensory substitution devices (SSDs) can convert environmental information into an accessible format for people who are blind or have low vision (pBLV). Yet, current SSDs often passively deliver all of the information available with limited user control, potentially leading to confusion and/or cognitive overload. To address this issue, this work proposes a selective, gesture-controlled system intended to improve information relevance and reduce cognitive overload. Methods: We present Point-To-Tell 2, a system that enables pBLV to privately and efficiently select which information to convey through simple pointing-based gestural control. By integrating a monocular camera with AI-driven pipelines for depth estimation, hand pose tracking, and object detection/segmentation, the system identifies the users' 3D pointing direction and announces the names and distances of objects as they are pointed at, thereby connecting an object's spatial position and identity through hand proprioception. Results: Validation tests in controlled indoor environments show high hand pose tracking accuracy, ensuring reliable ray-casting and object selection despite declining object recognition at longer distances. Distance estimates are stable at close range, though a systematic bias is present. Conclusion: This work introduces and technically validates an assistive system designed to improve the usability of assistive technologies by focusing system feedback—potentially reducing users' cognitive load and enhancing their spatial comprehension by leveraging concurrent hand proprioception. Future work will involve user testing and expanding system features to further enhance its practicality across more diverse scenarios. IMPLICATIONS FOR REHABILITATION: Enhanced Spatial Awareness: By leveraging hand proprioception and accurate 3D ray-casting, the system is expected to help users develop more robust mental maps of the environment by building off an egocentric representation of space. Pending further validation, this potential improvement in spatial awareness may enhance visual rehabilitation for orientation, mobility, and awareness of key objects in the environment. Reduced Cognitive Load: This gesture-based method allows the user to focus the AI feedback on describing specific elements of a scene. This potentially allows the user to focus on relevant information to them, while minimising extraneous information. This could reduce the cognitive burden associated with systems that provide too much irrelevant information that the user must then parse through. Increased Independence in Daily Activities: The hands-free design of the technology should allow the user to be able to focus on interactions with the environment, rather than on manipulating hardware. This should make users more efficient, in being able to call on additional information through brief mid-air gestures, even in the middle of completing daily tasks. Integration with Personalised Rehabilitation Programs: The adaptive and context-aware nature of the system makes it a promising candidate for inclusion in comprehensive rehabilitation protocols. Further optimisation and experimental testing of the system will aim to tailor sensory feedback specifically to support user-centric learning and functional improvements while ensuring privacy safeguards and confirming efficacy through testing with persons who are visually impaired. pubtype: Academic Journal doctype: algorithm equations & formulas pictorial research tables/charts Journal Article ougenre: Article language: English refInfo: holdings: @attributes: islocal: N |
|---|