┌─────────────────────────────────────────────────────┐
│ MICROCONTROLLER UNIT │
│ │
│ [CAMERA MODULE] │
│ │ │
│ ┌────▼────┐ ┌──────────┐ ┌────────────────┐ │
│ │ Image │───▶│ MCU │───▶│ USB-C Port │──┼──▶ Android Phone
│ │ Sensor │ │ (ESP32-S3│ │ (Power + │ │
│ └─────────┘ │ or RP2040│ │ Data) │ │
│ │ based) │ └────────────────┘ │
│ └────┬─────┘ │
│ │ │
│ ┌────────────┼────────────┐ │
│ │ │ │ │
│ [BTN_MAIN] [BTN_MODE] [BTN_CONTEXT] │
│ (Top Face) (Left Side) (Right Side) │
│ │
│ [LED INDICATOR] │
│ (Status Ring) │
└─────────────────────────────────────────────────────┘
Why 3 buttons instead of 1:
A single button creates complex multi-tap patterns that are unreliable for visually impaired users under stress. Three buttons with distinct physical positions allow muscle memory navigation without looking.
┌─────────────────────────────────────────────────────┐
│ BUTTON RESPONSIBILITY MAP │
├─────────────────┬───────────────────────────────────┤
│ BUTTON │ PHYSICAL LOCATION │
├─────────────────┼───────────────────────────────────┤
│ BTN_MAIN │ Top face, largest, tactile bump │
│ │ (Primary action trigger) │
├─────────────────┼───────────────────────────────────┤
│ BTN_MODE │ Left side, medium, ribbed texture │
│ │ (Mode switching / navigation) │
├─────────────────┼───────────────────────────────────┤
│ BTN_CONTEXT │ Right side, small, smooth │
│ │ (Contextual / interrupt) │
└─────────────────┴───────────────────────────────────┘
Every button supports four gesture types. This creates 12 distinct commands from 3 buttons without complex patterns.
┌─────────────────────────────────────────────────────────────────────┐
│ GESTURE VOCABULARY │
├──────────────┬──────────────────────────────────────────────────────┤
│ GESTURE TYPE │ DEFINITION │
├──────────────┼──────────────────────────────────────────────────────┤
│ TAP │ Press and release under 300ms │
│ HOLD │ Press and hold 300ms–1500ms │
│ DOUBLE TAP │ Two taps within 400ms │
│ HOLD+TAP │ Hold BTN_MODE, then tap BTN_MAIN simultaneously │
│ │ (Chord gesture, unlocks advanced commands) │
└──────────────┴──────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────┐
│ FULL BUTTON COMMAND MAP │
├─────────────────┬──────────────────┬───────────────────────────────────────────┤
│ BUTTON │ GESTURE │ ACTION │
├─────────────────┼──────────────────┼───────────────────────────────────────────┤
│ │ TAP │ Capture frame → Run active mode pipeline │
│ BTN_MAIN │ HOLD │ Start continuous streaming mode │
│ (Top Face) │ DOUBLE TAP │ Repeat last audio output │
│ │ HOLD 3s+ │ Emergency SOS trigger │
├─────────────────┼──────────────────┼───────────────────────────────────────────┤
│ │ TAP │ Cycle forward through Feature Modes │
│ BTN_MODE │ HOLD │ Cycle backward through Feature Modes │
│ (Left Side) │ DOUBLE TAP │ Jump to Home / Default Mode │
│ │ HOLD 3s+ │ Open Settings narration │
├─────────────────┼──────────────────┼───────────────────────────────────────────┤
│ │ TAP │ INTERRUPT — Silence audio immediately │
│ BTN_CONTEXT │ HOLD │ Repeat at slower speech rate │
│ (Right Side) │ DOUBLE TAP │ Ask follow-up ("Tell me more") │
│ │ HOLD 3s+ │ Toggle Caregiver live link │
├─────────────────┼──────────────────┼───────────────────────────────────────────┤
│ CHORD COMBOS │ │ │
├─────────────────┼──────────────────┼───────────────────────────────────────────┤
│ MODE + MAIN │ Simultaneous │ Toggle continuous obstacle detection ON/OFF│
│ MODE + CONTEXT │ Simultaneous │ Switch language / translation target │
│ ALL THREE │ Simultaneous 2s │ Hard reboot MCU + reconnect USB │
└─────────────────┴──────────────────┴───────────────────────────────────────────┘
The device operates in discrete Feature Modes. BTN_MODE cycles through them. The user always knows which mode they are in because the system announces it on each switch.
┌─────────────────────────────────────────────────────────────────────────────┐
│ FEATURE MODE RING │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ MODE 0 │───▶│ MODE 1 │───▶│ MODE 2 │───▶│ MODE 3 │──┐ │
│ │ HOME │ │ READ │ │ NAVIGATE│ │ PEOPLE │ │ │
│ │ (Default)│ │ (OCR) │ │ (Detect)│ │ (Face) │ │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │
│ ▲ │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
│ └──────────│ MODE 6 │◀───│ MODE 5 │◀───│ MODE 4 │◀─┘ │
│ │ DESCRIBE │ │ LEARN │ │ MONEY │ │
│ │ (VLM) │ │ (Edu) │ │ (Currency│ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │
│ BTN_MODE TAP = Move clockwise BTN_MODE HOLD = Move counter-clockwise │
│ BTN_MODE DOUBLE TAP = Jump to MODE 0 │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│ MODE 0: HOME │
│ Announced as: "Home Mode" │
├─────────────────────────────────────────────────────────────────────────────┤
│ BTN_MAIN TAP → Scene snapshot: "Describe what is in front of me" │
│ BTN_MAIN HOLD → Begin continuous scene awareness (10s intervals) │
│ BTN_CONTEXT TAP → Silence │
│ BTN_CONTEXT D.T → "Tell me more about what you just described" │
│ │
│ PIPELINE: │
│ Camera Frame → Scene Classifier → Object Detector → LLM Composer │
│ → TTS Engine → Phone Speaker / Headphones │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│ MODE 1: READ (OCR MODE) │
│ Announced as: "Read Mode. Point at text and press the top button." │
├─────────────────────────────────────────────────────────────────────────────┤
│ BTN_MAIN TAP → Capture → OCR → Read all detected text aloud │
│ BTN_MAIN HOLD → Continuous reading (for long documents, scroll by tilt) │
│ BTN_CONTEXT TAP → Silence mid-read │
│ BTN_CONTEXT HOLD → Re-read last block at half speed │
│ BTN_CONTEXT D.T → Translate last read text to target language │
│ │
│ SUB-MODES within READ (auto-detected by content type): │
│ • Label Mode → Product label detected, reads name + key warnings first │
│ • Document Mode → Detects table/form layout, reads by section │
│ • Currency Mode → Detects currency, announces denomination │
│ │
│ PIPELINE: │
│ Camera Frame → Pre-processor (deskew, contrast boost) │
│ → OCR Engine (on-device Tesseract or ML Kit) │
│ → Layout Classifier (label / document / currency) │
│ → Text Formatter → TTS → Speaker │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│ MODE 2: NAVIGATE (OBSTACLE + SPATIAL DETECTION) │
│ Announced as: "Navigate Mode. Continuous obstacle detection active." │
├─────────────────────────────────────────────────────────────────────────────┤
│ On Mode Entry → Automatically begins continuous frame processing │
│ BTN_MAIN TAP → Snapshot for detailed scene read ("What is near me?") │
│ BTN_MAIN HOLD → Increase detection frequency (from 2fps to 5fps) │
│ BTN_CONTEXT TAP → Silence current alert (obstacle alerts resume next hit) │
│ BTN_CONTEXT D.T → Ask "How many people are in front of me?" │
│ │
│ SPATIAL AUDIO ROUTING: │
│ Objects LEFT of center → Audio panned LEFT channel │
│ Objects RIGHT of center → Audio panned RIGHT channel │
│ Objects CENTER + CLOSE → Alert beep + voice (headphones required for pan) │
│ │
│ PIPELINE: │
│ Continuous Frames → Object Detector (YOLO on-device) │
│ → Bounding Box Extractor → Spatial Mapper (left/center/right zones) │
│ → Proximity Estimator → Audio Panner → Speaker / Headphones │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│ MODE 3: PEOPLE (FACE + SOCIAL LAYER) │
│ Announced as: "People Mode." │
├─────────────────────────────────────────────────────────────────────────────┤
│ BTN_MAIN TAP → Capture → Face detection → Name announcement │
│ BTN_MAIN HOLD → Scan room, list all detected faces │
│ BTN_CONTEXT TAP → Silence │
│ BTN_CONTEXT D.T → "Describe their expression and body language" │
│ BTN_MODE + MAIN → Register new face: "Say the name of this person" │
│ │
│ PIPELINE: │
│ Camera Frame → Face Detector → Face Embedder │
│ → Local Embedding DB Lookup → Name Resolver │
│ [If unknown] → LLM: "Unknown person. Male, approximate 30s, smiling." │
│ → Emotion Classifier (optional, Phase 3) → TTS │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│ MODE 4: MONEY (CURRENCY DETECTION) │
│ Announced as: "Money Mode. Hold a note or coin in front of the camera." │
├─────────────────────────────────────────────────────────────────────────────┤
│ BTN_MAIN TAP → Capture → Identify denomination → Read aloud │
│ BTN_MAIN HOLD → Continuous mode: reads each note as it passes camera │
│ BTN_CONTEXT D.T → "Calculate total from last 3 identified notes" │
│ │
│ PIPELINE: │
│ Camera Frame → Currency Classifier (lightweight CNN) │
│ → Denomination + Currency Type → TTS ("Twenty Dollar Bill") │
│ [Running Total Engine] → TTS on request │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│ MODE 5: LEARN (EDUCATIONAL / REASONING) │
│ Announced as: "Learn Mode. Point at anything to get an explanation." │
├─────────────────────────────────────────────────────────────────────────────┤
│ BTN_MAIN TAP → Capture → Full VLM analysis → Detailed explanation │
│ BTN_CONTEXT D.T → "Go deeper" / step-by-step breakdown │
│ BTN_CONTEXT HOLD → Re-read at half speed │
│ │
│ CONTENT AUTO-ROUTING: │
│ Math/Diagram detected → "This is a bar chart comparing..." │
│ Handwriting detected → OCR → Read → "This appears to be a sticky note" │
│ Whiteboard detected → Summarize key points: "3 main bullet points..." │
│ │
│ PIPELINE: │
│ Camera Frame → Content Type Classifier │
│ → Route: [OCR Path] or [Object Path] or [VLM Direct Path] │
│ → LLM Reasoning Layer → Structured Response Formatter → TTS │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│ MODE 6: DESCRIBE (RICH NARRATION / DELIGHT) │
│ Announced as: "Describe Mode. For art, photos, and rich descriptions." │
├─────────────────────────────────────────────────────────────────────────────┤
│ BTN_MAIN TAP → Capture → Full poetic/detailed scene description │
│ BTN_MAIN HOLD → Photo album mode: narrate continuously as frames change │
│ BTN_CONTEXT D.T → "What colors dominate this scene?" │
│ │
│ LLM PROMPT STYLE: Descriptive, emotional, painterly language │
│ "A warm, sun-drenched kitchen. A wooden table with a blue ceramic bowl..." │
│ │
│ PIPELINE: │
│ Camera Frame → High-res capture → VLM (cloud or on-device) │
│ → Narrative Prompt Template → LLM → TTS (natural, expressive voice) │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────────┐
│ COMPLETE SYSTEM LAYERS │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
│ │ LAYER 6: OUTPUT LAYER │ │
│ │ Phone Speaker (mono) / Headphones (stereo spatial audio) │ │
│ │ TTS Engine → [Google TTS / On-device neural TTS] │ │
│ │ Audio Panner (for directional Navigate Mode alerts) │ │
│ └──────────────────────────────────────────────────────────────────────────┘ │
│ ▲ │
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
│ │ LAYER 5: RESPONSE COMPOSER │ │
│ │ Assembles structured output from pipeline results │ │
│ │ Applies Mode-specific language style (terse vs. descriptive) │ │
│ │ Manages audio queue (priority interrupts, don't stack alerts) │ │
│ └──────────────────────────────────────────────────────────────────────────┘ │
│ ▲ │
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
│ │ LAYER 4: AI / REASONING LAYER │ │
│ │ On-device Models: YOLO (objects), FaceNet (faces), Tesseract (OCR) │ │
│ │ On-device LLM: small quantized model for offline reasoning │ │
│ │ Cloud VLM (optional, for MODE 5/6): GPT-4V or Gemini Vision │ │
│ │ Memory Manager: short-term buffer + long-term SQLite DB │ │
│ └──────────────────────────────────────────────────────────────────────────┘ │
│ ▲ │
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
│ │ LAYER 3: FEATURE MODE ROUTER │ │
│ │ Active Mode State Machine │ │
│ │ Routes incoming frame + button event to correct pipeline │ │
│ │ Manages mode transitions and announcements │ │
│ └──────────────────────────────────────────────────────────────────────────┘ │
│ ▲ │
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
│ │ LAYER 2: USB COMMUNICATION LAYER (Android App Core) │ │
│ │ USB Host Manager → Opens connection to MCU │ │
│ │ Frame Receiver → MJPEG or raw frame stream parsing │ │
│ │ Button Event Parser → Decodes button ID + gesture type from MCU │ │
│ │ Power Monitor → Tracks USB power delivery to MCU │ │
│ └──────────────────────────────────────────────────────────────────────────┘ │
│ ▲ │
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
│ │ LAYER 1: HARDWARE LAYER (Microcontroller) │ │
│ │ Camera Sensor → Frame Buffer → USB Bulk Transfer to Android │ │
│ │ BTN_MAIN, BTN_MODE, BTN_CONTEXT → Debounced → Gesture Classifier │ │
│ │ Gesture events transmitted to Android as USB HID or serial packets │ │
│ │ Receives USB power (5V) from Android via OTG → powers camera + MCU │ │
│ └──────────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────────┘
The MCU communicates two types of data to the Android app over USB:
┌─────────────────────────────────────────────────────────────────────┐
│ USB DATA PACKET TYPES │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ TYPE A: FRAME PACKET (Camera Data) │
│ ┌──────────┬──────────┬──────────────────────────────┐ │
│ │ Header │ Frame ID │ MJPEG compressed frame data │ │
│ │ 0xAA 0xBB│ 4 bytes │ Variable length │ │
│ └──────────┴──────────┴──────────────────────────────┘ │
│ Sent at: 5–15fps depending on active mode │
│ │
│ TYPE B: BUTTON EVENT PACKET │
│ ┌──────────┬───────────┬────────────────┬────────────┐ │
│ │ Header │ Button ID │ Gesture Type │ Timestamp │ │
│ │ 0xCC 0xDD│ 1 byte │ 1 byte │ 4 bytes │ │
│ └──────────┴───────────┴────────────────┴────────────┘ │
│ │
│ Button ID values: 0x01=MAIN, 0x02=MODE, 0x03=CONTEXT │
│ Gesture Type: 0x01=TAP, 0x02=HOLD, 0x03=DOUBLE_TAP │
│ 0x04=LONG_HOLD, 0x05=CHORD │
│ │
└─────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│ ANDROID APP MODULE MAP │
├──────────────────────────────┬──────────────────────────────────────────────┤
│ MODULE │ RESPONSIBILITY │
├──────────────────────────────┼──────────────────────────────────────────────┤
│ UsbHostModule │ Detect MCU connection, open USB interface, │
│ │ manage bulk transfer endpoints │
├──────────────────────────────┼──────────────────────────────────────────────┤
│ FrameReceiverModule │ Parse incoming byte stream into valid │
│ │ JPEG/frame objects, timestamp them │
├──────────────────────────────┼──────────────────────────────────────────────┤
│ ButtonEventModule │ Parse button packets, implement gesture │
│ │ timing logic (tap vs hold threshold), │
│ │ emit named events to Mode Router │
├──────────────────────────────┼──────────────────────────────────────────────┤
│ ModeStateMachine │ Tracks current active mode, handles │
│ │ mode transitions, announces mode changes │
├──────────────────────────────┼──────────────────────────────────────────────┤
│ PipelineOrchestrator │ On frame + command received, selects which │
│ │ AI pipeline to invoke based on active mode │
├──────────────────────────────┼──────────────────────────────────────────────┤
│ OCRPipeline │ Pre-processes frame, runs ML Kit OCR, │
│ │ classifies layout type, formats output │
├──────────────────────────────┼──────────────────────────────────────────────┤
│ ObjectDetectionPipeline │ Runs YOLO inference, maps bounding boxes │
│ │ to spatial zones, generates proximity alerts │
├──────────────────────────────┼──────────────────────────────────────────────┤
│ FaceRecognitionPipeline │ Detects faces, generates embeddings, │
│ │ queries local face DB, returns names │
├──────────────────────────────┼──────────────────────────────────────────────┤
│ CurrencyPipeline │ Runs lightweight currency CNN classifier, │
│ │ maintains running total state │
├──────────────────────────────┼──────────────────────────────────────────────┤
│ VLMPipeline │ Encodes frame to base64, sends to cloud VLM │
│ │ or on-device model, returns description │
├──────────────────────────────┼──────────────────────────────────────────────┤
│ MemoryModule │ Short-term: rolling 15-min visual buffer │
│ │ Long-term: SQLite for faces, routines │
├──────────────────────────────┼──────────────────────────────────────────────┤
│ AudioOutputModule │ Manages TTS queue, priority interrupts, │
│ │ spatial panning for headphone output, │
│ │ speech rate control │
├──────────────────────────────┼──────────────────────────────────────────────┤
│ SettingsModule │ Voice-navigable settings: language, speed, │
│ │ mode defaults, caregiver link toggle │
├──────────────────────────────┼──────────────────────────────────────────────┤
│ CaregiverModule │ Opt-in remote monitoring, SOS trigger, │
│ │ sends compressed frame + location to contact │
└──────────────────────────────┴──────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│ AUDIO PRIORITY QUEUE │
├────────────┬────────────────────────────────────────────────────────────────┤
│ PRIORITY │ EXAMPLE │
├────────────┼────────────────────────────────────────────────────────────────┤
│ P0 URGENT │ "STOP. Obstacle directly ahead." │
│ │ Interrupts everything. Cannot be silenced mid-word. │
├────────────┼────────────────────────────────────────────────────────────────┤
│ P1 ALERT │ "Stair detected on your left." │
│ │ Interrupts P2/P3. BTN_CONTEXT TAP silences it. │
├────────────┼────────────────────────────────────────────────────────────────┤
│ P2 RESPONSE│ Response to user-triggered action (OCR result, face name) │
│ │ Plays after current P2 finishes. TAP to interrupt. │
├────────────┼────────────────────────────────────────────────────────────────┤
│ P3 AMBIENT │ Periodic scene descriptions in continuous mode │
│ │ First to be dropped if queue is full. │
└────────────┴────────────────────────────────────────────────────────────────┘
SPEAKER vs HEADPHONE BEHAVIOR:
• Phone Speaker Only → Mono output, no spatial panning, louder voice
• Headphones Detected → Stereo spatial panning enabled for Navigate Mode
Quieter, more intimate voice style
Navigate Mode becomes fully directional (left/right beeps)
┌─────────────────────────────────────────────────────────────────────────────┐
│ VOICE INPUT FLOW │
│ │
│ User says "Hey Glasses" │
│ │ │
│ ▼ │
│ On-device Wake Word Detector (Porcupine or equivalent) │
│ │ │
│ ▼ │
│ Phone Microphone activates (500ms window) │
│ │ │
│ User speaks command: "What is the expiry date on this?" │
│ │ │
│ ▼ │
│ On-device STT (Speech-to-Text) → Intent Classifier │
│ │ │
│ ├──▶ Detected Mode Switch Intent → ModeStateMachine │
│ ├──▶ Detected Query Intent → PipelineOrchestrator │
│ ├──▶ Detected Settings Intent → SettingsModule │
│ └──▶ Detected Unknown / Unclear → "Sorry, say that again." │
│ │
│ BTN_MAIN HOLD = Alternative to wake word (always reliable, no mic needed) │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────┐
│ SETTINGS MENU STRUCTURE │
│ Activated by: BTN_MODE HOLD 3 seconds │
│ │
│ System announces: "Settings. Say a setting name or press Mode to browse." │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ SETTING 1: Speech Speed → Slow / Normal / Fast │ │
│ │ SETTING 2: Language → English / [other options] │ │
│ │ SETTING 3: Default Mode → Which mode on startup │ │
│ │ SETTING 4: Obstacle Alerts → Always On / Only in Navigate │ │
│ │ SETTING 5: Caregiver Link → Enable / Disable │ │
│ │ SETTING 6: Face Database → Add / Remove / Clear │ │
│ │ SETTING 7: Voice Input → Enable / Disable wake word │ │
│ │ SETTING 8: Exit Settings │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
│ Navigation: BTN_MODE cycles through settings │
│ Activation: BTN_MAIN TAP selects and cycles value │
│ Exit: BTN_MODE DOUBLE TAP returns to active mode │
└─────────────────────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────┐
│ LED STATUS PATTERNS │
├───────────────────────┬──────────────────────────────────────────┤
│ LED PATTERN │ MEANING │
├───────────────────────┼──────────────────────────────────────────┤
│ Slow blue pulse │ Connected to Android, idle │
│ Solid green │ Active, processing frame │
│ Fast white flash │ Button event registered │
│ Solid red │ USB disconnected or power error │
│ Amber pulse │ Low battery (if battery option added) │
│ Purple flash x3 │ Mode changed (matches mode number) │
│ All white spin │ SOS triggered, caregiver being contacted │
└───────────────────────┴──────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────────┐
│ EXAMPLE: User is in READ MODE, presses BTN_MAIN TAP on a medicine bottle │
│ │
│ [1] User presses BTN_MAIN │
│ │ │
│ [2] MCU: Debounce → Gesture Timer → Confirms TAP (released < 300ms) │
│ │ │
│ [3] MCU: Sends Button Event Packet {ID:MAIN, GESTURE:TAP} over USB │
│ │ │
│ [4] MCU: Captures camera frame → Compresses to MJPEG → Sends Frame Packet │
│ │ │
│ [5] Android ButtonEventModule: Parses packet → Emits event MAIN_TAP │
│ │ │
│ [6] ModeStateMachine: Current mode = READ → routes to OCRPipeline │
│ │ │
│ [7] OCRPipeline receives frame: │
│ → Pre-process (deskew, sharpen, contrast) │
│ → Run OCR → Extract text blocks │
│ → Layout Classifier: "This is a product label" │
│ → Priority formatter: Drug name first, then dosage, then warnings │
│ │ │
│ [8] Response Composer: Builds structured audio script │
│ "Paracetamol 500mg. Take 1 to 2 tablets every 4 to 6 hours. │
│ Warning: Do not exceed 8 tablets in 24 hours." │
│ │ │
│ [9] AudioOutputModule: Queues as P2, plays via TTS on speaker/headphones │
│ │ │
│ [10] User taps BTN_CONTEXT → Audio silences immediately │
│ │ │
│ [11] User double-taps BTN_CONTEXT → "Tell me more" │
│ → OCRPipeline re-queries LLM: "Any allergen warnings on this label?" │
│ → Response: "Contains Sodium, not suitable for low-sodium diets." │
│ │
└─────────────────────────────────────────────────────────────────────────────────┘
┌───────────┬────────────────────────────────────────────────────────────────────┐
│ PHASE │ WHAT GETS BUILT │
├───────────┼────────────────────────────────────────────────────────────────────┤
│ PHASE 1 │ UsbHostModule, FrameReceiverModule, ButtonEventModule, │
│ │ ModeStateMachine (modes 0–2 only), OCRPipeline, │
│ │ ObjectDetectionPipeline (basic), AudioOutputModule, │
│ │ CurrencyPipeline, basic VoiceInput │
├───────────┼────────────────────────────────────────────────────────────────────┤
│ PHASE 2 │ Full scene understanding in Mode 0, document layout in Mode 1, │
│ │ MemoryModule (short-term), TranslationModule, Mode 5 skeleton │
├───────────┼────────────────────────────────────────────────────────────────────┤
│ PHASE 3 │ FaceRecognitionPipeline, Mode 3 fully live, EmotionClassifier, │
│ │ BodyLanguageDetector, crowd mapping │
├───────────┼────────────────────────────────────────────────────────────────────┤
│ PHASE 4 │ Full Mode 5 (math, handwriting, whiteboard), VLMPipeline cloud, │
│ │ interactive tutoring conversation loop │
├───────────┼────────────────────────────────────────────────────────────────────┤
│ PHASE 5 │ Mode 6 fully live (poetic narration), photo/art/sports modes, │
│ │ richer audio expressiveness │
├───────────┼────────────────────────────────────────────────────────────────────┤
│ PHASE 6 │ MemoryModule (long-term), PersonalizationProfiles, │
│ │ CaregiverModule, PredictiveAssistance, haptic/smartwatch link │
└───────────┴────────────────────────────────────────────────────────────────────┘