Problem: When building web apps (React Flow crash course, book library, financial adviser), you often want to chat with an LLM within the app itself — asking it to manipulate app state, query data, provide recommendations, etc. Currently this intelligence lives outside the app (in VS Code chat), disconnected from the running application.
Goal: Build a reusable chatbot module with two parts — a React widget (drop-in chat panel) and a FastAPI backend (LLM proxy + tool orchestration). Each host app provides an adapter that defines what the LLM can see and do within that specific app.
First integration: React Flow crash course — the LLM acts as a tutor that can add/remove/modify nodes on the canvas, navigate to lessons, explain concepts, and help design custom node layouts.
HOST APPLICATION (Browser)
+=============================================================+
| |
| +-------------+ +----------------------------+ |
| | App UI | | ChatPanel (React widget) | |
| | (lessons, | | +-----------------------+ | |
| | canvas, | | | Message list | | |
| | settings) | | | Tool result cards | | |
| +------+------+ | | Input bar + voice btn | | |
| | | +-----------+-----------+ | |
| | +--------------|-------------+ |
| | | |
| +------v----------------------------------v-----------+ |
| | App Adapter (per-app) | |
| | - Tool definitions (schema for LLM) | |
| | - Tool handlers (execute against app state) | |
| | - System prompt (app personality) | |
| | - Context provider (current page, state snapshot) | |
| +------------------------+----------------------------+ |
+===========================-|=================================+
| SSE (streaming) + REST (tool results)
+============================|====================================+
| FASTAPI BACKEND |
| | |
| +--------v--------+ +-------------------+ |
| | /chat/stream | | /chat/tool-result | |
| | (SSE endpoint) | | (resume after | |
| +--------+---------+ | frontend tool) | |
| | +-------------------+ |
| +--------v-------------------------------------------+ |
| | LLM Orchestrator | |
| | - Build context (system + memory + dynamic context) | |
| | - Stream Claude API response | |
| | - Route tool calls (frontend vs backend) | |
| | - Resume after tool results | |
| +-----------+--------------------+--------------------+ |
| | | |
| +-----------v------+ +---------v-----------+ |
| | Claude API | | Backend Tool Exec | |
| | (Anthropic SDK) | | (DB queries, search) | |
| +------------------+ +---------------------+ |
+==================================================================+
The backend reuses the provider abstraction pattern already established across your repos (gmail-intelligent-assistant, font-library-assistant, book-library-assistant).
# Reuse from: gmail-intelligent-assistant/src/ai/base_provider.py
class LLMProvider(ABC):
def analyze(self, content, prompt, **kwargs) -> str: ...
def analyze_image(self, image_path, prompt) -> str: ... # Future: vision
def get_model_name(self) -> str: ...
def validate_availability(self) -> bool: ...
def estimate_cost(self, content, prompt) -> float | None: ...
# Three concrete providers:
class LocalClaudeProvider(LLMProvider): # Zero-cost via Max/Pro subscription
class AnthropicProvider(LLMProvider): # Direct API (ANTHROPIC_API_KEY)
class OpenAIProvider(LLMProvider): # Fallback (OPENAI_API_KEY)# Reuse from: font-library-assistant/scripts/ai/api_provider.py
def get_provider(provider: str = "local", model: str = None) -> LLMProvider:
if provider == "local":
return LocalClaudeProvider(model=model or "sonnet")
if provider == "auto":
if os.environ.get("ANTHROPIC_API_KEY"):
provider = "anthropic"
elif os.environ.get("OPENAI_API_KEY"):
provider = "openai"
else:
provider = "local" # Fall back to CLI
# ... instantiate appropriate providerThe zero-cost local provider runs Claude via subprocess using your Anthropic Max/Pro subscription. For the chatbot backend, this adapts from the subprocess pattern to a streaming subprocess pattern:
# Adapted from: gmail-intelligent-assistant/src/ai/local_provider.py
# Key difference: streaming output instead of batch
class LocalClaudeProvider(LLMProvider):
async def stream_response(self, messages, tools, system_prompt):
env = os.environ.copy()
env.pop('ANTHROPIC_API_KEY', None) # Prevent conflicts
env['CLAUDE_SCRIPT_MODE'] = '1' # Suppress notifications
proc = await asyncio.create_subprocess_exec(
claude_path, '--print', '--no-session-persistence',
'--model', self.model_name,
stdin=asyncio.subprocess.PIPE,
stdout=asyncio.subprocess.PIPE,
env=env,
)
# Stream stdout line by line as SSE eventsNote: The local CLI provider won't support native tool_use initially (CLI input/output is text-only). For v0.1, use it for simple chat; tool execution requires the Anthropic API provider. Tool support for local CLI can be added later by parsing structured output.
# Reuse from: gmail-intelligent-assistant/src/config.py
MODEL_ALIASES = {
"haiku": "claude-haiku-4-5-20251001",
"sonnet": "claude-sonnet-4-6",
"opus": "claude-opus-4-6",
}# config/settings.yaml (same pattern as gmail-intelligent-assistant)
ai:
provider: "auto" # "local" | "anthropic" | "openai" | "auto"
model: "sonnet" # Alias or full model ID
streaming: true
max_tokens: 4096| File | Source Repo | What to Reuse |
|---|---|---|
base_provider.py |
gmail-intelligent-assistant | ABC interface |
local_provider.py |
gmail-intelligent-assistant | CLI subprocess pattern |
api_provider.py |
font-library-assistant | Anthropic SDK integration |
openai_provider.py |
font-library-assistant | OpenAI fallback |
config.py |
gmail-intelligent-assistant | Model aliases, settings |
The ai-agents-workshop (external, attended) has useful patterns to study:
- Pydantic AI agents (lesson_07.py):
@agent.system_prompt,@agent.tooldecorators, structured outputs, async streaming event handlers - Smolagents (lesson_06.py): Alternative agent framework patterns
- Logfire/Phoenix observability: For debugging LLM interactions in production
- Review these for system prompt engineering and memory management ideas
- Anthropic Python SDK is first-class with best streaming support
- FastAPI has native async, SSE via
StreamingResponse, auto OpenAPI docs - Python is the AI/ML lingua franca — tool definitions, prompt engineering, evals
- Pydantic gives typed validation for all payloads
- Already used in your MediaBackupOrchestrator and photo-auto-crop repos
The communication pattern is request → stream → pause → resume, not true bidirectional:
- Frontend sends message via
POST /chat/stream, receives SSE stream back - If LLM calls a frontend tool, backend sends
tool_callSSE event and pauses - Frontend executes the tool, sends result via
POST /chat/tool-result - Backend resumes the SSE stream
SSE is simpler than WebSocket (no upgrade negotiation, auto-reconnect, works through all CDNs/proxies).
MCP is designed for server-to-server tool exposure. This system needs browser-side tool execution (manipulating React state), which MCP has no concept of. A simpler custom tool protocol using Anthropic's native tool format is more appropriate. MCP could be added later behind the backend for connecting to external data sources.
Tools are marked as either execution: 'frontend' or execution: 'backend':
- Frontend tools: Manipulate app state (add_node, navigate, update_collection). Executed in the browser.
- Backend tools: Query databases, search indexes, call external APIs. Executed on the server.
- Both types use the same Anthropic tool schema format.
The chatbot can go beyond read-only queries — it can modify the app (settings, UI, data) with user approval. This mirrors Claude Code's permission model:
+------------------------------------------+
| [bot] I'd like to change the theme to |
| "midnight" to match the dark nodes |
| you're building. Allow? |
| |
| [Allow] [Allow for session] [Deny] |
+------------------------------------------+
Tool permission levels:
auto— Execute without asking (read-only tools likeget_canvas_state,search_lessons)ask— Require user approval each time (app modifications likechange_settings,clear_canvas,delete_collection)session— Ask once, then auto-approve for the rest of the session
Each tool definition in the adapter specifies its permission level:
{
name: 'change_setting',
description: 'Modify an app setting',
execution: 'frontend',
permission: 'ask', // <-- requires user approval
input_schema: { ... }
}This is what makes the chatbot a true active component of the app rather than just a passive Q&A bot.
Browser Backend Claude API
| | |
|-- POST /chat/stream -------->| |
| { message, context, |-- messages.create() -------->|
| tools[], session_id } | |
| |<-- stream: text tokens ------|
|<-- SSE: token "Here's" -----| |
|<-- SSE: token " how..." ----| |
| |<-- stream: tool_use ---------|
|<-- SSE: tool_call{id,name}--| |
| | [pauses via asyncio.Event] |
| [execute tool against | |
| React Flow state] | |
| | |
|-- POST /chat/tool-result --->| |
| { call_id, result } | [resumes stream] |
| |-- continue with result ----->|
| |<-- stream: more tokens ------|
|<-- SSE: token "I've added"--| |
|<-- SSE: done ----------------|<-- end_turn ----------------|
type SSEEvent =
| { type: 'token'; data: { text: string } }
| { type: 'tool_call'; data: { callId: string; name: string; input: object } }
| { type: 'tool_result'; data: { callId: string; name: string; result: object } }
| { type: 'done'; data: {} }
| { type: 'error'; data: { message: string } };chatbot/ # NEW — standalone directory (sibling to crash course)
├── packages/
│ ├── widget/ # @marius/chatbot-widget (React npm package)
│ │ ├── src/
│ │ │ ├── components/
│ │ │ │ ├── ChatPanel.tsx # Main panel (header, messages, input)
│ │ │ │ ├── MessageList.tsx # Scrollable message display
│ │ │ │ ├── MessageBubble.tsx # Individual message (markdown, code blocks)
│ │ │ │ ├── InputBar.tsx # Text input + voice + send button
│ │ │ │ ├── ToolResultCard.tsx # Collapsible tool execution display
│ │ │ │ ├── ChatToggle.tsx # FAB button to open/close panel
│ │ │ │ ├── VoiceButton.tsx # Mic button with waveform
│ │ │ │ └── ThinkingDots.tsx # Typing indicator
│ │ │ ├── core/
│ │ │ │ ├── ChatEngine.ts # State management (useSyncExternalStore)
│ │ │ │ ├── StreamHandler.ts # SSE connection + event parsing
│ │ │ │ ├── ToolRouter.ts # Routes tool_call events to handlers
│ │ │ │ ├── MemoryManager.ts # Sliding window + summarization
│ │ │ │ └── SessionManager.ts # Session CRUD + localStorage
│ │ │ ├── hooks/
│ │ │ │ ├── useChat.ts # Main hook: messages, send, status
│ │ │ │ └── useVoice.ts # Web Speech API wrapper
│ │ │ ├── types/
│ │ │ │ ├── config.ts # ChatbotConfig interface
│ │ │ │ ├── messages.ts # ChatMessage, ToolCall, etc.
│ │ │ │ └── tools.ts # ToolDefinition, ToolHandler
│ │ │ ├── ChatbotProvider.tsx # Context provider (wraps app)
│ │ │ ├── styles.css # Widget CSS (CSS modules, ~5KB)
│ │ │ └── index.ts # Public API exports
│ │ ├── package.json
│ │ └── vite.config.ts # Library mode build
│ │
│ ├── backend/ # FastAPI service
│ │ ├── app/
│ │ │ ├── main.py # FastAPI app, CORS, lifespan
│ │ │ ├── routes/
│ │ │ │ ├── chat.py # /chat/stream (SSE), /chat/tool-result
│ │ │ │ └── sessions.py # /sessions CRUD
│ │ │ ├── services/
│ │ │ │ ├── llm.py # Claude API orchestration loop
│ │ │ │ ├── memory.py # Context window management
│ │ │ │ └── tools.py # Tool registry + backend execution
│ │ │ ├── models/
│ │ │ │ └── schemas.py # Pydantic models (messages, tools, sessions)
│ │ │ └── config.py # Settings from env vars
│ │ ├── pyproject.toml
│ │ └── Dockerfile
│ │
│ └── shared/ # Shared types (optional, for strict typing)
│ └── tool-schema.ts
│
├── adapters/ # Per-app adapter examples
│ ├── reactflow/ # React Flow crash course adapter
│ │ ├── tools.ts # Tool definitions (add_node, get_canvas, etc.)
│ │ ├── handlers.ts # Frontend tool execution handlers
│ │ ├── system-prompt.ts # React Flow tutor personality
│ │ └── context.ts # Dynamic context (current lesson, canvas state)
│ ├── book-library/ # Future: book library adapter
│ └── finance/ # Future: financial adviser adapter
│
├── package.json # Monorepo root (pnpm workspaces)
├── pnpm-workspace.yaml
└── turbo.json # Build orchestration
How a host app integrates the chatbot (3 files):
// 1. src/chatbot/adapter.ts — Define tools and handlers
import type { ChatbotConfig } from '@marius/chatbot-widget';
export const chatbotConfig: ChatbotConfig = {
backendUrl: 'http://localhost:8000',
appId: 'reactflow-tutor',
systemPrompt: `You are a React Flow tutor embedded in a crash course app...`,
contextProvider: () => ({
currentPage: window.location.pathname,
canvasState: { nodeCount: nodes.length, edgeCount: edges.length },
}),
tools: [
{ name: 'add_node', description: '...', execution: 'frontend', input_schema: {...} },
{ name: 'get_canvas_state', description: '...', execution: 'frontend', input_schema: {...} },
{ name: 'search_lessons', description: '...', execution: 'backend', input_schema: {...} },
],
toolHandlers: {
add_node: async (input, { setState }) => {
setState(prev => ({ ...prev, nodes: [...prev.nodes, input] }));
return { success: true, data: { nodeId: input.id } };
},
get_canvas_state: async (_input, { getState }) => {
return { success: true, data: getState() };
},
},
appearance: {
position: 'right',
width: 400,
theme: 'dark',
accentColor: '#ff44cc',
title: 'React Flow Tutor',
welcomeMessage: 'Hi! I can help you build React Flow diagrams. Ask me anything!',
},
memory: { maxMessages: 50, persistSessions: true },
voice: { enabled: true },
};
// 2. src/App.tsx — Mount the provider + toggle
import { ChatbotProvider, ChatToggle } from '@marius/chatbot-widget';
import { chatbotConfig } from './chatbot/adapter';
export default function App() {
return (
<ChatbotProvider config={chatbotConfig}>
<BrowserRouter>
<Routes>...</Routes>
</BrowserRouter>
<ChatToggle />
</ChatbotProvider>
);
}
// 3. Backend — Register backend tools (if any)
// app/adapters/reactflow.py
@tool_registry.register("reactflow-tutor", "search_lessons")
async def search_lessons(query: str) -> dict:
# ... search lesson metadata
return {"results": [...]}The chatbot FAB button appears on every page of the host app. Its capabilities adapt based on context:
| Page | Chatbot Can Do |
|---|---|
| Home / Lesson list | Answer questions, recommend lessons, navigate |
| Lesson page | Explain current lesson, link related concepts, answer questions |
| Sandbox page | Full canvas manipulation — create nodes, edges, styles, layouts on the user's behalf |
| Cheatsheet | Answer API questions, provide examples |
The adapter's contextProvider() tells the chatbot which page the user is on, and the system prompt instructs it to behave accordingly — conversational help everywhere, but active canvas creation only on the sandbox.
The sandbox is a dedicated route (/sandbox) with a blank React Flow canvas where the chatbot becomes a co-creator:
+--------+-----------------------------------------+-----------+
| Sidebar | Sandbox Canvas | Chat Panel|
| (nav) | | (open by |
| | [blank React Flow canvas] | default) |
| | | |
| | User can drag/interact AND the | [messages]|
| | chatbot can inject nodes, edges, | |
| | styles, and layouts via tools | [you]: |
| | | "Create a |
| | +-------+ +--------+ | workflow |
| | | Start |---->| Process| | with 3 |
| | +-------+ +--------+ | steps" |
| | (created by chatbot) | |
| | | [bot]: |
| | | "Done! |
| | | I've..." |
| | | |
| | | [input] |
+--------+-----------------------------------------+-----------+
Sandbox chatbot capabilities (three interaction levels):
Pre-defined tools the adapter exposes. The chatbot calls them like functions:
add_node— Create nodes (default, input, output, custom) at positionsadd_edge— Connect nodes with edges (bezier, step, smoothstep)remove_node/remove_edge— Clean upupdate_node_style— Change colors, borders, shadows, shapesupdate_edge_style— Change type, color, animationset_layout— Apply auto-layout (dagre, elk) to current graphget_canvas_state— Read what's on the canvasclear_canvas— Start freshexport_canvas— Export as PNG/SVG (using existing html-to-image)
Good for: basic node/edge creation, styling, layout. Limited to what tools are pre-defined.
The sandbox defines a node configuration schema — a rich spec that describes node appearance and behavior without writing code:
interface NodeConfig {
shape: 'rectangle' | 'circle' | 'diamond' | 'hexagon' | 'pill';
handles: { id: string; type: 'source' | 'target'; position: Position }[];
content: { label: string; icon?: string; fields?: FieldConfig[] };
style: { bg: string; border: string; shadow: string; borderRadius: number };
behavior: { draggable: boolean; resizable: boolean; rotatable: boolean };
animation?: { type: 'pulse' | 'glow' | 'bounce'; speed: number };
}The chatbot calls create_custom_node_type({ config }) and the sandbox renders it from the schema. This gets you 70-80% of what the crash course lessons demonstrate — custom shapes, handles, styles, animations — without code generation.
For truly complex nodes (custom React JSX with event handlers, state, etc.), the sandbox includes a code editor (Monaco/CodeMirror) + sandboxed live preview:
- Chatbot generates React component code and injects it into the editor
- Sandbox compiles and renders it in a sandboxed iframe (safe execution)
- User can see, edit, and iterate on the code via chat
- This is how the most advanced lesson-level complexity is achieved
This is the most powerful but also most complex approach — essentially what the tutoring-sandbox spec envisions. Deferred to a later version.
The chatbot should never inject raw HTML/JS directly into the DOM:
- Security risk: Arbitrary code execution, XSS vectors
- State management: React won't know about DOM changes, causing sync issues
- No undo/redo: Can't track or reverse injected changes
- Fragile: Breaks when the app re-renders
Instead, all interactions go through React state — either via pre-defined tools (Level 1), configuration schemas (Level 2), or sandboxed code compilation (Level 3).
The sandbox exposes a public API (React context + hooks) that both the user interface AND the chatbot tools use:
// Sandbox API — used by drag handles, toolbar buttons, AND chatbot tools
interface SandboxAPI {
// Canvas state
getNodes(): Node[];
getEdges(): Edge[];
// Manipulation (same methods for user interaction and chatbot)
addNode(config: NodeConfig): string;
addEdge(source: string, target: string, config?: EdgeConfig): string;
removeNode(id: string): void;
updateNodeStyle(id: string, style: Partial<NodeStyle>): void;
// Custom node types
registerNodeType(name: string, config: NodeTypeConfig): void;
// Layout
applyLayout(algorithm: 'dagre' | 'elk', options?: LayoutOptions): void;
// History
undo(): void;
redo(): void;
// Export
exportImage(format: 'png' | 'svg'): Promise<Blob>;
}This way the chatbot's tools are thin wrappers around the same API that powers the sandbox UI. The sandbox is the smart part — the chatbot just talks to it.
Progression: Level 1 in v0.2, Level 2 in v0.3+, Level 3 future. The goal is that the sandbox can eventually recreate the complexity of all 70+ crash course lessons.
Key insight: The sandbox page is where the chatbot truly collaborates — the user describes what they want ("create a workflow diagram with approval gates") and the chatbot builds it step by step, letting the user refine by chatting ("make the nodes blue", "add a reject path from the approval node").
The sandbox concept is app-specific — not all apps need one, and each app's creative space looks different:
- React Flow crash course: Node/edge sandbox — create, style, layout, animate diagrams
- Book library browser: Collection canvas — curate book lists, render topic network graphs, visualize how books connect across topics, perhaps even 3D book rendering and page-flipping to referenced passages
- Financial adviser: Dashboard builder — custom transaction views, chart creation, due-date visualizations
The chatbot widget doesn't own the sandbox — it provides the interaction layer (chat + tools + approval). Each app implements its own creative capabilities and exposes them to the chatbot via the adapter.
The chatbot isn't limited to a sandbox page. With the approval system (Section 5 above), it can modify the app anywhere:
- Change app settings ("switch to midnight theme", "increase font size")
- Navigate between pages ("take me to the custom nodes lesson")
- Modify user preferences ("enable keyboard shortcuts")
- Create/delete data ("add this book to my collection", "delete this transaction category")
The sandbox is the most powerful creative space, but the chatbot is an active component throughout the app — like having Claude Code's agency embedded directly in the application UI.
On lesson pages, home, cheatsheet — the chat panel is a helper, not a co-creator:
+--------+-----------------------------------------+-----------+
| Sidebar | Main Content | Chat Panel|
| (nav) | (lesson canvas + | (slide-in |
| | code panel) | drawer) |
| | | |
| | | [messages]|
| | | |
| | | |
| | | [input] |
+--------+-----------------------------------------+-----------+
^
FAB button (bottom-right)
Chat Panel Component:
+------------------------------------------+
| [icon] React Flow Tutor [-] [X] | Header
+------------------------------------------+
| |
| [bot] Welcome! I can help you build | MessageList
| React Flow diagrams... |
| |
| [you] Add a node with label "Start" |
| |
| [bot] Sure! Let me add that... |
| |
| [tool] add_node — Added "Start" at | ToolResultCard
| position (100, 100) [v] |
| |
| [bot] Done! I've added a "Start" node. |
| Want me to connect it to something? |
| |
+------------------------------------------+
| [mic] [Type a message... ] [send] | InputBar
+------------------------------------------+
Widget states:
- Collapsed: FAB button only (bottom-right corner)
- Open (panel): 400px slide-in from right, main content shrinks
- Open (overlay): On narrow screens, overlays instead of pushing
- Maximized: Full-width for longer conversations
Styling: CSS variables with --chatbot- prefix, falls back to host app's CSS vars. Ships own CSS file (~5KB), no Tailwind dependency in the widget.
CONTEXT WINDOW (~100k tokens)
┌──────────────────────────────────────────────────────────┐
│ System Prompt (per-app, ~2k tokens) │
│ Summary of dropped messages (~500 tokens) │
│ Dynamic app context (current page, state) (~1-5k tokens) │
│ Recent messages (sliding window, last ~50 messages) │
└──────────────────────────────────────────────────────────┘
- Sliding window: Keep last N messages (configurable, default 50)
- Summarization: When messages drop off, summarize the dropped conversation
- Session persistence: Full history saved to localStorage (user sees all messages, but only window is sent to LLM)
- Dynamic context:
contextProvider()called each request — injects current page, canvas state, etc.
Optional, progressive enhancement. Deferred to later version.
- Input: Web Speech API (
SpeechRecognition) for zero-dependency STT. Hold-to-talk button with visual waveform feedback. - Output: Web Speech API (
SpeechSynthesis) for basic TTS. Optional premium voices via local providers. - Auto-play mode: Off by default, configurable.
Existing voice infrastructure to reuse:
~/.claude/plugins/voice/— Voice notification plugin with provider abstraction- Providers: Piper and Kokoro (local TTS engines with Python venvs, zero API cost)
~/.claude/plugins/voice/scripts/tts.sh— Main TTS orchestrator~/.claude/plugins/voice/config/project-voices.json— Per-project voice settings~/.claude/plugins/voice/providers/— Provider implementations (factory pattern)- Reuse this provider abstraction for chatbot TTS output instead of building from scratch
Three modes:
- Hosted (production): Backend holds the API key. Users authenticate via host app.
- BYOK (bring your own key): User enters key in chat settings, stored encrypted in localStorage, sent per-request.
- Direct (dev only): Frontend calls Claude API directly. Never for production.
| Method | Path | Purpose |
|---|---|---|
| POST | /chat/stream |
Send message, receive SSE stream |
| POST | /chat/tool-result |
Return frontend tool execution result |
| GET | /sessions |
List sessions for an app |
| GET | /sessions/{id} |
Load session history |
| POST | /sessions |
Create new session |
| DELETE | /sessions/{id} |
Delete session |
| GET | /health |
Health check |
Goal: Polished chat panel with streaming and one working tool.
- FastAPI backend with
/chat/streamSSE endpoint - Provider abstraction (port from gmail/font-library repos): Anthropic API + local CLI
- Auto-detection of available provider from env vars
- Dark-themed chat UI matching the crash course (CSS vars, slate-900, fuchsia accent)
- Chat panel: MessageList, MessageBubble, InputBar, ThinkingDots
- FAB toggle button (bottom-right), slide-in drawer
- 1-2 basic tools:
navigate_to_lesson(frontend) and/orget_canvas_state(frontend) - Pause-resume SSE pattern for frontend tool execution
- ToolResultCard component (collapsible)
- Hardcoded React Flow tutor system prompt
- No memory management yet (send all messages each request)
- Backend runs locally (uvicorn), deploy to Railway/Fly.io later
Goal: Dedicated sandbox page where the LLM co-creates React Flow diagrams.
- Sandbox page (
/sandbox) with blank React Flow canvas + chat open by default - Expand frontend tools: add_node, remove_node, add_edge, remove_edge, update_node_style, update_edge_style, set_layout, clear_canvas
- Tool definition schema and registry pattern
- Wire handlers to React Flow state (useNodesState, useEdgesState)
- Context-aware system prompt: chatbot knows which page user is on (sandbox vs lesson vs home)
- Backend tool support: search_lessons (searches lesson metadata)
- Mixed tool calls (frontend + backend in one LLM response)
Goal: Conversations persist, context management works.
- Sliding window memory with message summarization
- Session CRUD + localStorage persistence (frontend)
- Session storage on backend (JSON files or SQLite)
- Dynamic context provider (current lesson, canvas state summary)
- Token counting and context budget management
- Session title auto-generation from first message
- Session list/switcher in chat panel header
Goal: Extract universal module from React Flow integration.
- Factor out
@marius/chatbot-widgetnpm package - Define
ChatbotConfiginterface - Create
ChatbotProvidercontext component - Move React Flow-specific code into adapter directory
- Vite library mode build config
- Test with a second adapter (book library) to validate universality
Goal: Voice input/output and production-ready UX.
- Web Speech API integration (STT + TTS)
- VoiceButton component with waveform visual feedback
- Keyboard shortcuts (Cmd+Shift+L toggle, Escape close)
- Responsive layout (overlay mode on narrow screens)
- Error handling, retry logic, reconnection
- Rate limiting on backend
- BYOK mode for API keys
- Third adapter (finance) to validate across three apps
- Documentation + integration guide
- npm package publication
- Backend deployment guide (Docker, Railway)
- Performance audit (bundle size, SSE reconnection, memory leaks)
- Markdown rendering in messages (code blocks, bold, links, syntax highlighting)
- Session persistence across devices (backend-stored sessions)
- OpenAI provider fallback for tool execution
- Local Claude CLI streaming with tool support
- ElevenLabs premium voice integration
- Multi-turn tool execution visualization
- Guided exercises with validation (from tutoring-sandbox spec)
- AI-driven canvas manipulation with animation
- Sandbox page with code editor (Monaco/CodeMirror) + live preview
- Observability integration (Logfire/Phoenix, inspired by ai-agents-workshop)
- Cost tracking and usage dashboard
| Component | Choice | Rationale |
|---|---|---|
| LLM | Claude (Sonnet speed, Opus depth) | Best tool use + streaming |
| Backend | FastAPI (Python) | Async, SSE, first-class Anthropic SDK |
| Communication | SSE + REST | Simpler than WebSocket for pause-resume |
| Frontend | React 19 + TypeScript | Matches all target apps |
| Widget build | Vite 7 library mode | Same tooling as host apps |
| Monorepo | pnpm workspaces + Turborepo | Fast, proven |
| State | useSyncExternalStore | Matches existing crash course pattern |
| Sessions | localStorage + JSON files (backend) | No external DB needed for v1 |
| Widget CSS | CSS modules (vanilla) | No Tailwind version conflicts |
| Voice STT | Web Speech API | Zero dependency |
| Voice TTS | Web Speech API / ElevenLabs | Progressive enhancement |
- Layout.tsx — Mount
ChatToggleandChatPanelas siblings to<main>outlet - SettingsContext.tsx — Follow
useSyncExternalStorepattern for ChatEngine state - LessonPage.tsx — Chat panel coexists with split-view; reuse resize handle pattern
- index.css — Extend CSS var system with
--chatbot-*variables - lessons.ts — Expose lesson metadata to backend for search_lessons tool
- Create
~/Developer/AgenticCoding/chatbot/directory with monorepo structure - Initialize git repo, pnpm workspaces, Turborepo
- Create
development/folder with:development-plan.md— Copy of this plan (persistent reference)ideation.md— Vision, goals, target appsprerequisites.md— Setup requirementslearnings.md— Empty, to be filled during development
- Create
.claude/CLAUDE.mdwith project-specific instructions - Set up FastAPI backend scaffold (main.py, routes, config)
- Set up React widget package scaffold (components, core, types)
- Port provider abstraction from gmail-intelligent-assistant
- v0.1 smoke test: Open crash course, click FAB, type "What is React Flow?", see streamed response
- v0.2 tool test: Type "Add a blue node labeled 'Hello' at position 200,200", see node appear on canvas
- v0.3 memory test: Have a 10+ message conversation, refresh page, conversation loads from session
- v0.4 universality test: Create a minimal second app with different tools, same widget works
- v0.5 voice test: Click mic, say "show me all nodes", LLM calls get_canvas_state and responds with summary