Elwing S3 file storage analysis & garden files plan

files.md

S3 File Storage: Current State & Plan

Where files are stored in S3

All files go to the lefos-conversations-dev bucket (config: CONVERSATION_BUCKET), with one shared S3 client at packages/elwing/src/s3/client.ts.

S3 key hierarchy:

{domainId}/{mailboxId}/{conversationId}/
├── artifacts/{filename}        # Agent-produced files
├── attachments/{attachmentId}  # Cached email/Gmail attachments
exports/{userId}/{exportId}.json  # User data exports
previews/pr-{number}/...          # PR preview builds (separate bucket)

Where files are written to S3

What	Where	Trigger
Artifacts (images, PDFs, code output)	vfs/mounts/artifacts.ts → media.uploadArtifact()	Agent writes to /workspace/artifacts/ via VFS (e.g., GenerateImage, CodeExecution, Bash)
Email attachments (Mailhook)	vfs/mounts/attachments.ts → media.uploadAttachment()	Lazy on first access — downloaded from Mailhook API, then cached to S3
Gmail attachments	vfs/mounts/attachments.ts → media.uploadAttachment()	Lazy on first access — downloaded from Gmail API, then cached to S3
User data exports	export/task.ts → PutObjectCommand	User requests data export from UI

Where files are read from S3

What	Where	Consumer
Artifacts	media.downloadArtifact()	VFS mount (agent reads), Media API (UI serves)
Attachments	media.downloadAttachment()	VFS mount (agent reads), Media API (UI serves)
Media API (HTTP serving)	webapp/routes/media.ts	UI fetches GET /api/media/:spaceId/:mailboxId/:conversationId/{artifacts|attachments}/:name
Email composition	utils/emailAttachments.ts → resolveMediaUrlToData()	Agent attaching files to outgoing emails
Export download	webapp/routes/user.ts → GetObjectCommand	User downloads export JSON

Internal URL scheme

Files are referenced in DB records via media URLs: https://lefosmedia.internal/{domainId}/{mailboxId}/{conversationId}/{type}/{name}. These are NOT real URLs — the UI transforms them to /api/media/... HTTP paths. The mapping is built at runtime in artifactUrls / attachmentUrls maps and baked into conversation records via replaceVfsPaths() in records.ts.

What's NOT tracked in Postgres today

• No files table exists. S3 is the only source of truth for file existence.
• Artifacts: Only discoverable by S3 ListObjectsV2 (see listArtifacts()). No DB metadata — filename, size, content-type are only in S3 object metadata.
• Attachments: Metadata lives in conversation state (in-memory attachmentsById map, populated from email events). The S3 copy is a cache — the canonical source is Mailhook/Gmail.
• Media URLs are embedded inline in conversation record JSON blobs, not in a separate table.

---

Plan: conversation_files table

1. Migration: Create the table

CREATE TABLE conversation_files (
  file_id       text PRIMARY KEY DEFAULT gen_tagged_id('file'),
  space_id      text NOT NULL,
  mailbox_id    text NOT NULL,
  conversation_id text,                    -- NULL for space/mailbox-level files
  file_type     text NOT NULL,             -- 'artifact', 'attachment', 'upload'
  filename      text NOT NULL,             -- display name
  s3_key        text NOT NULL UNIQUE,      -- full S3 key
  content_type  text NOT NULL DEFAULT 'application/octet-stream',
  size_bytes    bigint,
  source        text,                      -- 'agent', 'email', 'gmail', 'user_upload'
  source_ref    text,                      -- email_id, gmail message_id, etc.
  created_at    timestamptz NOT NULL DEFAULT now(),
  created_by    text,                      -- user_id for uploads, NULL for agent

  CONSTRAINT valid_file_type CHECK (file_type IN ('artifact', 'attachment', 'upload'))
);

CREATE INDEX idx_conversation_files_conv
  ON conversation_files (space_id, mailbox_id, conversation_id);
CREATE INDEX idx_conversation_files_mailbox
  ON conversation_files (space_id, mailbox_id);

2. Register files on write (low-risk, additive)

Modify media.uploadArtifact() and media.uploadAttachment() to also INSERT into conversation_files. This is the single choke-point for all S3 writes, so we capture everything:

• uploadArtifact() → insert with file_type='artifact', source='agent'
• uploadAttachment() → insert with file_type='attachment', source='email' or 'gmail'

Use ON CONFLICT (s3_key) DO UPDATE SET size_bytes = ..., updated_at = now() so re-uploads (same artifact overwritten) update the row.

3. Backfill existing S3 files

A one-time script that calls ListObjectsV2 on the bucket and inserts rows for existing files. Parse the S3 key pattern {domainId}/{mailboxId}/{conversationId}/{type}/{name} to populate columns.

4. API: List files for a conversation/mailbox

New route: GET /api/files/:spaceId/:mailboxId[/:conversationId] — queries conversation_files with access check (reuse verifyAccess from media.ts). Returns metadata list. The actual bytes are still served by the existing /api/media/... route.

5. API: Upload files from UI

New route: POST /api/files/:spaceId/:mailboxId[/:conversationId]/upload:

1. Accept multipart form data
2. Upload to S3 with key {spaceId}/{mailboxId}/{conversationId}/uploads/{fileId}-{filename}
3. Insert into conversation_files with file_type='upload', source='user_upload', created_by=userId
4. Return the file metadata + media URL

6. UI: File browser component

A file list panel in the conversation/mailbox view that queries the list API. Shows filename, type badge (artifact/attachment/upload), size, date. Each row links to the existing media download endpoint. An upload button triggers the upload API.

7. Make uploaded files available to the agent

Register user-uploaded files in attachmentsById (or a new VFS mount like /workspace/uploads/) so the agent can reference them. The VFS read path already falls through to S3 via downloadArtifact/downloadAttachment, so this mostly needs a new mount or extending the artifacts mount to also cover uploads.

Execution order

1. Migration (table creation) — zero risk, additive
2. Register on write — modify uploadArtifact/uploadAttachment in media.ts
3. List API + Upload API — new routes
4. Backfill script — one-time
5. UI — file browser component
6. Agent VFS integration — make uploads visible to the agent

---

Eager Attachment Caching

Previously, email attachments were only downloaded from Mailhook and cached to S3 when the agent accessed them via the VFS mount (lazy). This meant attachments the agent never read were absent from S3.

Now (PR #877), all Mailhook attachments are downloaded and uploaded to S3 as a durable Absurd step during email processing, before the agent loop starts. This ensures every attachment is in S3 regardless of whether the agent reads it.

• Gmail attachments are skipped (require per-user OAuth, discovered mid-conversation via CorrespondenceHistory)
• Failures are non-fatal; the lazy VFS path still works as fallback
• Batched concurrency (max 3) with Promise.allSettled