SignSpeak

A low-cost wearable that translates ASL into speech in real time, and speech back into text, enabling two-way conversation between Deaf and hearing people.

Overview

SignSpeak is a real-time sign language translation device built to break the communication barrier between ASL users and the hearing world without an interpreter or a phone app in someone's hand. A Raspberry Pi with a camera watches the signer and speaks their signs aloud through a speaker, while a microphone picks up the hearing person's reply and displays it as text on an LCD for the Deaf user, making the conversation fully two-way and hands-free. It also supports personalized gesture-to-command mappings, so custom signs can trigger smart home actions. The whole system runs on-device on hardware costing a fraction of commercial alternatives.

What I built

• Static sign recognition: a MobileNetV2 CNN fine-tuned with transfer learning from ImageNet classifies the ASL alphabet and digits across 36 classes, reaching 95% validation accuracy in just 10 epochs.
• Dynamic gesture recognition: a Bidirectional LSTM processes frame sequences to recognize whole words and phrases, hitting 90% validation accuracy, with a TCN with attention pooling extending the vocabulary to 50 words on the ASL Citizen dataset.
• Aggressive data augmentation: fewer than 500 original video sequences were expanded into 14,000+ training samples through frame interpolation, temporal shifting, and frame repetition — which is what made the sequence models trainable at all.
• On-device inference pipeline: models exported to TorchScript and ONNX for the Raspberry Pi, with real-time latency under 2 seconds for static signs, plus offline speech-to-text through Vosk so nothing depends on the cloud.
• Complete wearable build: Pi 4, camera module, LCD, microphone, speaker, and thermal management in a custom enclosure that can be worn as a waist pack or set facing the user.

Tech

Python · PyTorch · MobileNetV2 · BiLSTM · TCN · MediaPipe · OpenCV · Vosk · ONNX · Raspberry Pi

Outcome

2nd place at the 2025 Alameda County Science and Engineering Fair in the Embedded Systems category. Built as a team of two during my junior year of high school, and my first serious end-to-end ML project: dataset construction, training, deployment, and hardware, all the way through.