Iris Smart Glasses 🥽

Voice-controlled smart glasses system with dynamic device discovery and comprehensive mock mode.

Status: ✅ Implementation Complete - Ready for hardware integration and demo deployment.

Overview

Smart glasses that use your phone as eyes, ears, and mouth while a LT acts as the brain and a Pi Zero W drives a tiny OLED display.

Architecture

              Phone (Galaxy S22+ running IP Webcam)
             ┌─────────────────────────────────────┐
             │                                     │
             │  👁️ Camera ──── video stream ──────┼──────▶ ┌────────────┐
             │                                     │        │            │
             │  👂 Mic ──────── audio stream ─────┼──────▶ │      LT     │
             │                                     │        │   (brain)  │
             │  👄 Speaker ◀─── TTS audio ────────┼─────── │            │
             │                                     │        └─────┬──────┘
             └─────────────────────────────────────┘              │
                                                                  │ SSH 
                                                                  ▼
                                                           ┌─────────────┐
                                                           │  Pi Zero W  │
                                                           │   (bridge)  │
                                                           └──────┬──────┘
                                                                  │ I2C
                                                                  ▼
                                                           ┌─────────────┐
                                                           │    OLED     │
                                                           │   128x64    │
                                                           └─────────────┘

Component Roles

Component	Role	What it does
Phone (Galaxy S22+)	Eyes, ears, mouth	Camera captures video. Mic captures voice. Speaker plays TTS audio output.
Pi Zero W	Dumb display terminal	Receives text strings from LT. Draws them on OLED. Zero logic.

Hardware Requirements

OLED: Elegoo 0.96" 128x64, SSD1306 driver, I2C (4 pins), address 0x3c
Pi: Raspberry Pi Zero W
Phone: Galaxy S22+ running IP Webcam app (Thyoni Tech)
Network: Phone WiFi hotspot connects LT + Pi

Project Structure

iris-glasses/
├── .gitignore
├── .env.example                 # API keys and config
├── README.md
├── requirements.txt             # LT dependencies
├── config.yaml                  # Configuration file
├── main.py                      # Entry point (runs on LT)
│
├── core/                        # Core infrastructure (LT)
│   ├── __init__.py
│   ├── feature_base.py          # Abstract base class for features
│   ├── display.py               # Sends text to Pi over SSH
│   ├── audio.py                 # Audio I/O with phone via IP Webcam
│   ├── camera.py                # Video capture from phone
│   └── voice_trigger.py         # Wake word + command parsing
│
├── features/                    # Modular features (LT)
│   ├── __init__.py
│   ├── todo/                    # Voice-controlled todo list
│   ├── directions/              # Turn-by-turn navigation
│   └── translation/             # Live text translation via OCR
│
└── pi/                          # 0w
    ├── display_server.py        # Dumb terminal for OLED display
    ├── requirements.txt         # Pi dependencies
    └── setup.sh                 # Pi setup script

Quick Start

# Setup (one time)
./setup.sh

# Run in full mock mode (no hardware needed)  
python3 main.py --mock

# Run with laptop microphone + terminal display (starts immediately)
python3 main.py --audio-source laptop --display terminal

# Run with ESP32 display (Texas team - starts immediately)
python3 main.py --audio-source laptop --display esp32 --display-ip 192.168.1.100

# Run with Arduino Nano display (starts immediately)
python3 main.py --audio-source laptop --display nano

# Run with Pi Zero W display (waits for Pi discovery)
python3 main.py --audio-source laptop --display pi

# Run hardware diagnostics
python3 main.py --debug-hardware

# Run tests
./run_tests.py --coverage

Architecture

Phone (mic) → Laptop (brain) → Pi Zero W (display) + ESP32 devices (IoT)

State machine voice command processing
mDNS discovery (no hardcoded IPs)
Pluggable audio (laptop mic, IP Webcam, mock)
Persistent SSH display connection
Thread-safe interrupts for motion alerts
Complete mock mode for testing

Setup Instructions

1. LT Setup

# Quick setup with script
./setup.sh

# OR manual setup:
pip3 install -r requirements.txt
cp .env.example .env
# Edit .env with API keys

2. Pi Zero W Setup

# On the Pi, run the setup script
cd ~
git clone <repository-url> iris
cd iris
chmod +x pi/setup.sh
./pi/setup.sh

# Follow the script's instructions to reboot and test

3. Phone Setup

Install IP Webcam app by Thyoni Tech from Google Play Store
Open the app and start the server
Note the IP address shown (e.g., http://192.168.43.1:8080)
Enable WiFi hotspot on your phone
Connect both LT and Pi to the phone's hotspot

4. Configuration

Edit .env on the LT:

# Update with your phone's IP Webcam URL
IP_WEBCAM_URL=http://192.168.43.1:8080

# Update Pi hostname if needed
PI_HOST=raspberrypi.local

# Add API keys for features
GOOGLE_MAPS_API_KEY=your_key_here
DEEPL_API_KEY=your_key_here
OPENAI_API_KEY=your_key_here

Running the System

Start the phone: Open IP Webcam app and start server
Start the Pi: SSH into Pi and run display server (or it can be auto-started)

Start the LT brain:

cd iris-glasses
source venv/bin/activate
python main.py

Voice Commands

"hey iris"              # Wake up → Main menu
"todo"                  # → Todo list
"weather"               # → Weather display  
"translation"           # → Translation mode
"connect"               # → Device list

# In device list:
"connect light"         # Connect to smart light
"connect fan"           # Connect to smart fan

# Device controls:
"on", "off"            # Light/fan control
"low", "high"          # Fan speed
"back"                 # Return to previous screen

Display Constraints

The 0.96" 128x64 OLED is tiny:

Max 4 lines of text
Max 21 characters per line
Uses symbols to save space: ✓ ○ → ↑ ↓ ← ● ■

Network Topology

Phone (Galaxy S22+) hosting WiFi hotspot
   │
   ├─── LT connects to hotspot
   │    └─── Runs main.py (brain)
   │
   └─── Pi Zero W connects to hotspot  
        └─── Runs display_server.py (display only)

LT SSHs to Pi to send display commands

Display Options

The system supports multiple display hardware types:

Display Type	Description	Usage	Startup Speed
`none`	No display output	Testing without hardware	Immediate
`terminal`	ASCII display to stdout	Local development	Immediate
`nano`	Arduino Nano over serial USB	Prototyping	Immediate
`esp32`	ESP32 over HTTP	Texas team glasses	Immediate
`pi`	Pi Zero W over SSH	Main demo glasses	Waits for Pi

Display Examples

# No display (testing only)
python3 main.py --display none

# Terminal ASCII (default in mock mode)
python3 main.py --display terminal

# Arduino Nano (auto-detect USB port)
python3 main.py --display nano

# Arduino Nano (specific port)
python3 main.py --display nano --serial-port /dev/ttyUSB0

# ESP32 display (requires IP)
python3 main.py --display esp32 --display-ip 192.168.1.100

# Pi Zero W (default in real mode)
python3 main.py --display pi

Configuration

Edit config.yaml and .env for:

Audio source preference
Device hostnames for discovery
API keys (OpenWeather, DeepL)
Timeout and polling settings

Hardware Integration

See UNDERSTAND.md for detailed API contracts and integration instructions.

ESP32 Requirements (IoT Devices)

Advertise via mDNS (e.g., "iris-light.local")
Implement HTTP JSON endpoints
Motion sensor must POST alerts to laptop

ESP32 Display Requirements (Texas Team)

// ESP32 display must implement:
// GET /status → 200 OK (health check)
// POST /display with JSON: {"lines": ["line1", "line2", "line3", "line4"]}

Arduino Nano Display Requirements

// Nano display reads from serial (9600 baud):
// Format: "line1|line2|line3|line4\n"
// Parse and display on OLED/LCD

Pi Setup

Run pi/setup.sh
Start pi/advertise.py
Configure SSH keys for passwordless access

Testing

test_parser.py - State machine behavior
test_discovery.py - Device discovery system
test_iot.py - IoT client communication
test_features.py - Todo, weather, translation

./run_tests.py --parser     # Run specific module
./run_tests.py --coverage   # Generate coverage report

Development

All components support mock mode for development without hardware:

python3 main.py --mock --audio-source mock

Use keyboard input instead of voice, ASCII display instead of OLED, simulated device responses.

Troubleshooting

Connection Issues

Verify all devices are on the same WiFi network (phone's hotspot)
Check IP Webcam URL in .env matches what's shown in the app
Test Pi SSH connection: ssh pi@raspberrypi.local
Use i2cdetect -y 1 on Pi to verify OLED at address 0x3c

Display Issues

Ensure I2C is enabled on Pi (sudo raspi-config)
Check OLED wiring: VCC, GND, SDA (pin 3), SCL (pin 5)
Test display server: echo "Test|Display|Working|OK" | python pi/display_server.py

Team Credits

Hardware: Pi Zero W + OLED display integration
Software: Python application architecture
Mobile: IP Webcam integration for camera/audio
AI: Voice recognition and command processing

Full documentation: UNDERSTAND.md
Hardware diagnostics: python3 main.py --debug-hardware

Built for TAMU Hacks Hardware Track 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Iris Smart Glasses 🥽

Overview

Architecture

Component Roles

Hardware Requirements

Project Structure

Quick Start

Architecture

Setup Instructions

1. LT Setup

2. Pi Zero W Setup

3. Phone Setup

4. Configuration

Running the System

Voice Commands

Display Constraints

Network Topology

Display Options

Display Examples

Configuration

Hardware Integration

ESP32 Requirements (IoT Devices)

ESP32 Display Requirements (Texas Team)

Arduino Nano Display Requirements

Pi Setup

Testing

Development

Troubleshooting

Connection Issues

Display Issues

Team Credits

About

Uh oh!

Releases

Packages

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 23 Commits
core		core
features		features
hardware		hardware
pi		pi
tests		tests
.env.example		.env.example
.gitignore		.gitignore
NANO_OLED_SETUP.md		NANO_OLED_SETUP.md
README.md		README.md
TAMU_SETUP.md		TAMU_SETUP.md
UNDERSTAND.md		UNDERSTAND.md
config.yaml		config.yaml
main.py		main.py
requirements.txt		requirements.txt
run_tests.py		run_tests.py
setup.sh		setup.sh
simple_display_test.py		simple_display_test.py
test_display.py		test_display.py
test_nano_display.py		test_nano_display.py
test_optional_pi.py		test_optional_pi.py
test_weather_display.py		test_weather_display.py

ryunzz/iris

Folders and files

Latest commit

History

Repository files navigation

Iris Smart Glasses 🥽

Overview

Architecture

Component Roles

Hardware Requirements

Project Structure

Quick Start

Architecture

Setup Instructions

1. LT Setup

2. Pi Zero W Setup

3. Phone Setup

4. Configuration

Running the System

Voice Commands

Display Constraints

Network Topology

Display Options

Display Examples

Configuration

Hardware Integration

ESP32 Requirements (IoT Devices)

ESP32 Display Requirements (Texas Team)

Arduino Nano Display Requirements

Pi Setup

Testing

Development

Troubleshooting

Connection Issues

Display Issues

Team Credits

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages