assistant

assistant.md

Voice Email Assistant

A voice-operated email triage assistant. Call a phone number, and an AI walks you through your unread emails one by one — summarizing each, recommending an action, and executing your choice via the Gmail API.

Built with Gemini Multimodal Live API (voice-to-voice), Twilio (phone), and FastAPI (WebSocket bridge).

Architecture

Phone call → Twilio → Cloudflare Tunnel → FastAPI (WebSocket) → Gemini Live API
                                                                       ↕
                                                                  Gmail API

• Twilio receives the call, streams audio over WebSocket as μ-law 8kHz
• FastAPI bridges Twilio and Gemini, handling audio transcoding (μ-law 8kHz ↔ Linear16 PCM 16/24kHz)
• Gemini Live API handles voice-to-voice conversation and calls email tools via function calling
• Gmail API executes actions (archive, delete, unsubscribe, read body)

Prerequisites

• Python 3.12+
• uv (package manager)
• cloudflared (tunnel)
• A Google AI Studio API key
• A Twilio account with a phone number
• Gmail API OAuth credentials (credentials.json from Google Cloud Console)

Setup

# Clone and install dependencies
git clone https://github.com/mattfarkas/assistant.git
cd assistant
uv sync

# Configure environment
cp .env.example .env
# Edit .env with your API keys:
#   GOOGLE_AI_STUDIO_API_KEY=...
#   TWILIO_ACCOUNT_SID=...
#   TWILIO_AUTH_TOKEN=...
#   TWILIO_PHONE_NUMBER=+1...

# Place Gmail OAuth credentials in project root
# (download from Google Cloud Console → APIs & Services → Credentials)
cp ~/Downloads/credentials.json .

# First run will open a browser for Gmail OAuth — follow the prompts
# This creates token.json (cached, won't need to re-auth)

Running

Terminal 1 — start the voice gateway:

uv run test_voice_gateway.py

Terminal 2 — expose to the internet:

cloudflared tunnel --url http://localhost:8000
# Copy the https://*.trycloudflare.com URL

Then configure Twilio:

1. Go to Twilio Console → Phone Numbers → your number
2. Set the Voice webhook URL to https://your-url.trycloudflare.com/voice (HTTP POST)
3. Call your Twilio number

Example Conversation

Assistant: Hi! You have 65 unread emails. Want to go through them?
You:       Yes
Assistant: Eventbrite — local workshop event. I recommend unsubscribe.
You:       Unsubscribe and delete.
Assistant: Unsubscribed and deleted. Delta Airlines — flight reminder. I recommend keep.
You:       Read it.
Assistant: Your flight DL 482 departs LAX February 20th at 8:30 AM,
           arriving JFK at 4:55 PM. Confirmation number XYZ789.
You:       Keep.
Assistant: Daily Newsletter — today's top stories. I recommend unsubscribe.
You:       Delete.
Assistant: Deleted. Mint — weekly spending summary. I recommend keep.
You:       Archive.
Assistant: Archived. Travel Deals Co — weekend getaway sales. I recommend delete.
You:       Delete.
Assistant: Deleted. That's the batch. Want me to fetch more?
You:       Yes.
Assistant: GitHub — PR comment on my-project. I recommend keep.
...

Available Voice Commands

Command	What it does
keep / skip	Leave the email as-is, move to next
archive	Remove from inbox (still in All Mail)
delete	Move to trash
unsubscribe	Extract unsubscribe link from headers
unsubscribe and delete	Both — unsubscribe then trash
read / summarize	Fetch full email body and read a summary

Project Structure

core/           Agent logic and dependency injection
features/email/ Gmail API tools, Pydantic models, tool schemas
gateways/       Twilio voice gateway, Gemini Live client, audio utils
tests/          Unit tests

Development

# Run linting, formatting, and tests
uv run python run_checks.py

# Run tests only
uv run pytest