SixthSense

An on-device navigation copilot that helps blind and low-vision users move through the world, built entirely without the cloud.

Overview

SixthSense is a wearable navigation system for blind and low-vision users. A chest-mounted Galaxy S25 Ultra watches the path ahead, runs depth estimation and object detection locally, and steers the wearer through a vibrating haptic belt and an offline voice agent. The motivation was a simple thesis: navigation safety should never depend on cloud connectivity, so the entire assistive pipeline runs on the phone and keeps working in airplane mode. Every frame is fused into a single scene state that answers three questions at once: what's around me, how close is it, and is my path clear.

What I built

• Parallel on-device vision pipeline: Depth-Anything-V2 and YOLOv11 run simultaneously on separate threads via ExecuTorch on the Qualcomm QNN backend, so per-frame latency is the max of the two models rather than the sum.
• Scene fusion and haptic mapping: a fusion layer folds depth zones, detected objects, OCR text, and a path-clear verdict into one compact scene state, then maps it to directional belt commands with distinct patterns for caution and curb-ahead warnings, keeping the stronger signal per zone so depth and detections reinforce rather than cancel.
• Offline voice agent: push-to-talk questions answered by keyword intent routing backed by an on-device Qwen LLM running through ExecuTorch, grounded in the live scene and fully functional with zero network.
• Live operator dashboard: a WebSocket streams the scene state and a downscaled camera frame to a React dashboard so sighted operators and judges can see exactly what the system perceives in real time.
• Tested deterministic core: the logic deciding what the user physically feels is pure Kotlin with no Android dependencies, covered by 21 unit tests.

Tech

Kotlin · Android · ExecuTorch · Qualcomm QNN · Depth-Anything-V2 · YOLOv11 · Qwen (on-device LLM) · BLE · ESP32 · React · WebSockets

Outcome

2nd place at the Qualcomm × Meta × ExecuTorch Hackathon 2026. The full assistive path runs in airplane mode on a single phone, from camera frame to belt buzz in tens of milliseconds. Built with a team of four; I led the software side — the vision pipeline, scene fusion, voice agent, and dashboard — while teammates owned the ESP32 belt hardware and firmware.