Gesture Smart
Gesture Smart is an accessibility mobile app that allows disabled users to operate Android devices using eye movements, facial tracks, and voice commands.
The Problem
Disabled users struggle to tap touchscreen interfaces, limiting their mobile device accessibility.
Business Goal
Build an offline facial gesture detector mapping eye winks and voice commands to device touch events.
Our Solution
We designed a React Native app utilizing Kotlin native modules for camera processing and AssemblyAI scripts.
Development Phases
Coded Kotlin eye tracker landmark mapping classes
Wrote Android system accessibility event triggers
Connected local offline speech recognition hooks
Technical Challenges
Optimizing eye gesture tracking to operate under 5% battery utilization.
Lessons Learned
Processing face landmarks locally instead of uploading video frames saved device bandwidth and battery life.
System Architecture
React Native with custom Java/Kotlin camera filters, connected to a local Express server and MongoDB cache store.
Technology Stack
Delivered Outcomes
Launched a fully working accessibility tool allowing users to navigate websites using winks.
Project FAQs
Is voter/user data uploaded?
No, all camera frames are processed directly on-device and deleted immediately.