AI Harness Feature Inventory and Failure Modes

Purpose: A practical feature taxonomy for evaluating or designing an AI harness around models such as ChatGPT, Codex, Claude Code, Cursor, OpenCode, Gemini CLI, Cline/Roo Code, or a custom agent runtime.

The key idea is that a harness should not be evaluated only by asking "What can the model do?" It should also be evaluated by asking:

What happens to the user when a harness feature is missing?

A capable model can still feel unusable if the harness has poor state management, no recovery paths, weak context controls, insufficient execution isolation, or no observability.

1. Core Design Principle: Conversation Recoverability

A robust harness should make it difficult for one bad prompt, one huge attachment, one verbose tool call, one mistaken agent action, or one failed experiment to permanently ruin an otherwise useful session.

This requires features such as:

Edit previous message
Delete a turn
Exclude a turn from active context
Branch from any message
Clone conversation up to a message
Retry a response
Retry with another model
Context compaction
Context usage inspection
Checkpoints
Rollback
Conversation handoff summaries
Selective context loading

A useful mental model is:

Visible conversation != Active model context

A message may remain visible in the transcript while being excluded from future model context.

2. Example Failure: Accidental Huge Message

Consider a long-running useful conversation:

M1
M2
M3
M4
M5
M6
M7  <- useful state
M8  <- accidental enormous paste

The accidental message may contain a huge log, source tree, copied document, generated output, or other content.

If the harness has:

no edit,
no delete,
no branch,
no clone-to-here,
no selective context exclusion,
no compaction,

then this happens:

Long-running useful conversation
        |
User accidentally sends enormous message
        |
Message becomes immutable conversation history
        |
Context window reaches maximum
        |
Harness refuses to continue
        |
No recovery operation exists
        |
Entire useful conversation is effectively stranded

The underlying model may still be capable of continuing. The harness has made the state unrecoverable.

A better harness would provide multiple recovery paths.

Recovery path A: Edit the message

Replace the giant message with the intended short prompt and regenerate from that point.

Recovery path B: Delete the message

Return the thread to the state immediately before the accidental paste.

Recovery path C: Branch from the previous message

                  bad giant message
                 /
M1 -> ... -> M7
                 \
                  clean branch -> continue

Recovery path D: Clone conversation up to M7

Create another thread containing M1 through M7, excluding M8.

Recovery path E: Start a new conversation with a compact handoff

New thread context:

- Goal
- Important decisions
- Current architecture
- Constraints
- Completed work
- Pending work

Recovery path F: Exclude M8 from active context

The transcript remains intact, but the model no longer receives M8 on every turn.

3. Reusable Harness Comparison Matrix

Use this structure when evaluating a harness:

Category	Feature	What it enables	What happens if missing	Severity
Conversation	Edit previous message	Correct an earlier prompt and continue from that state	Mistakes become permanently embedded; huge prompts can strand the session	Critical
Conversation	Branch from message	Explore another path without destroying current work	Experiments become destructive; recovery requires rebuilding context	Critical
Conversation	Clone up to message	Continue from an earlier clean state	Cannot remove a bad tail from a useful conversation	Critical
Context	Context usage meter	Shows how much usable context remains	Context exhaustion appears unexpectedly	High
Context	Compaction	Preserve important state while freeing tokens	Long-running sessions eventually become unusable	Critical
Context	Exclude message from context	Keep transcript but remove token cost/influence	Large irrelevant turns permanently consume context	Critical
Execution	Interrupt agent	Stop incorrect work early	Wrong work continues unnecessarily	High
Versioning	Checkpoints	Restore a known-good state	Recovery from mistakes becomes manual	Critical
Coding	Worktree isolation	Run agents safely in parallel	Agents can overwrite or contaminate each other's work	Critical
Tools	Large-output handling	Store verbose output externally and retrieve on demand	Logs/tool results can exhaust context	High
Security	Granular permissions	Balance autonomy with safety	User chooses between constant prompts and excessive access	Critical
Observability	Tool/action history	Understand exactly what the agent did	Failures become hard to diagnose	High
Artifacts	Persistent artifacts	Reuse outputs outside original chat	Useful work becomes trapped inside conversations	Medium
Knowledge	Cross-thread search	Rediscover previous work	Project knowledge becomes fragmented	Medium

4. Conversation State Management

4.1 Edit previous user message

Enables

Correct a typo or incorrect requirement.
Remove accidentally pasted content.
Shorten an oversized message.
Change a premise and regenerate from that point.
Reuse the earlier conversation without adding another corrective message.

If missing

Incorrect prompts remain permanently in the conversation.
The model repeatedly sees obsolete requirements.
A giant accidental prompt can consume the remaining context window.
Correcting the mistake requires another turn, consuming more context.
The user may have to abandon an otherwise useful thread.

Severity: Critical

4.2 Retry / regenerate response

Enables

Re-run the same turn without modifying conversation semantics.
Recover from a transient poor answer or tool failure.
Compare alternative generations.

If missing

The user must send another message such as "try again."
That corrective prompt becomes additional permanent context.
Repeated failures cause conversation bloat.

Severity: High

4.3 Retry with another model or reasoning level

Enables

Escalate a difficult turn to a stronger model.
Preserve the exact same context while comparing outputs.

If missing

Switching models may require recreating the task.
A hard problem can force abandonment of accumulated context.

Severity: Medium to High

4.4 Branch from any earlier message

Enables

Experiment without damaging the main line.
Explore two architectural alternatives.
Recover from a bad later turn.
Test different prompts from the same known state.

If missing

Experiments are destructive.
The user must manually reconstruct previous context.
One poor direction can contaminate all subsequent conversation.

Severity: Critical

4.5 Clone conversation up to message X

Enables

Create a fresh thread from a clean historical point.
Discard a problematic tail.
Continue before an accidental oversized message.

If missing

The user cannot surgically recover a valid earlier conversation state.
Manual copy/paste becomes necessary.

Severity: Critical

4.6 Delete an individual turn

Enables

Remove irrelevant, accidental, incorrect, or sensitive content.
Recover token budget if deletion also affects active context.

If missing

Unwanted content remains permanently attached to the thread.
Incorrect assumptions may keep influencing future answers.
Large turns remain expensive.

Severity: High to Critical

4.7 Exclude a message from model context

Enables

Keep the historical transcript without paying context cost.
Exclude irrelevant logs, attachments, generated output, or abandoned requirements.

If missing

Visible history and active model context are forced to be identical.
Large historical turns repeatedly consume context.
Context becomes increasingly polluted over time.

Severity: Critical

4.8 Conversation checkpoints

Enables

Save a known-good conversational state before experimentation.
Label meaningful milestones.
Restore later.

If missing

There is no explicit safe point.
Complex workflows become harder to recover after a bad turn.

Severity: High

4.9 Restore checkpoint

Enables

Roll context and optionally workspace state back together.

If missing

Recovery becomes a manual reconstruction problem.

Severity: Critical for long-running agents

4.10 Interrupt generation or running agent

Enables

Stop the model when it is clearly moving in the wrong direction.
Prevent unnecessary file edits or tool calls.

If missing

Incorrect work continues until completion.
More cleanup is required later.
Compute, time, and tokens are wasted.

Severity: High

4.11 Steer a running agent

Enables

Correct direction without cancelling the entire task.
Add constraints while execution is underway.

If missing

The user must wait until completion, then start corrective work.

Severity: High

4.12 Queue next instruction

Enables

Add follow-up work while the current task is still executing.
Avoid needing to remember to return later.

If missing

User must manually wait and re-engage after completion.

Severity: Medium

4.13 Resume previous session

Enables

Continue after app close, crash, disconnect, or device switch.

If missing

Work may have to restart.
Long tasks become fragile.

Severity: High

4.14 Cross-device continuation

Enables

Start work on desktop and inspect/control it on mobile or another machine.

If missing

The task is tied to one device or interface.

Severity: Medium

5. Context Management

5.1 Context usage meter

Enables

See how much of the model context window is occupied.
Identify dangerous growth early.

If missing

Context exhaustion arrives unexpectedly.
Users cannot decide when to compact or branch.

Severity: High

5.2 Remaining-context estimate before send

Enables

Warn before a message will consume most of the remaining window.

Example:

This message will use approximately 48% of your remaining context.

If missing

One accidental paste can make the session unusable immediately.

Severity: Critical

5.3 Unusually-large-message confirmation

Enables

Catch accidental source-tree, log, or document pastes before they are committed.

Example:

This message is unusually large. Send anyway?

If missing

Accidental huge messages become easy and expensive.

Severity: High

5.4 Automatic context compaction

Enables

Summarize older turns.
Preserve key decisions while freeing tokens.
Keep long-running tasks alive.

If missing

Every long enough conversation eventually dies from context exhaustion.

Severity: Critical

5.5 Manual compaction

Enables

User explicitly chooses when to compress earlier history.
Useful before major new phases.

If missing

The user cannot control the context lifecycle.

Severity: High

5.6 Context inspector

Enables

See which messages, files, tool results, and summaries consume context.

Possible UI:

Conversation turns     31%
Repository snippets    24%
Tool results           19%
System/instructions    11%
Files                   9%
Remaining               6%

If missing

Users know context is full but not why.

Severity: High

5.7 Selective context loading

Enables

Load only relevant historical turns.
Avoid treating the entire transcript as mandatory input.

If missing

Every old message has permanent token cost regardless of relevance.

Severity: Critical

5.8 Pin important context

Enables

Mark requirements that must survive summarization and compaction.

Examples:

API contract
security rule
target architecture
coding conventions

If missing

Compaction may drop the very constraints that matter most.

Severity: High

5.9 Conversation handoff summary

Enables

Produce a compact state snapshot for another thread or agent.

Useful fields:

Objective
Current status
Decisions
Constraints
Files changed
Known issues
Pending work

If missing

Moving to another thread requires manually rewriting project context.

Severity: High

5.10 New thread from summary

Enables

Treat context exhaustion as a normal lifecycle event instead of failure.

If missing

The handoff still requires manual copy/paste and setup.

Severity: High

5.11 Cross-thread references

Enables

Intentionally import relevant state from another conversation.

If missing

Knowledge gets fragmented across sessions.

Severity: Medium

5.12 Project-level context

Enables

Persistent context independent of individual chat threads.

If missing

Every new thread starts close to zero.

Severity: High

6. Tool Output and Large Data Handling

6.1 Tool-output truncation

Imagine an agent runs:

cat giant-log.txt

and gets 150,000 lines.

A weak harness puts the entire output into active context.

A better harness stores the output externally and returns:

Command produced 150,432 lines.
Full output saved as artifact.
Relevant excerpt: ...

If missing

One verbose command can exhaust the entire context window.

Severity: Critical

6.2 External storage for large tool output

Enables

Keep giant logs, API responses, test traces, and generated data outside conversational context.

If missing

Large transient output is carried forever.

Severity: High

6.3 Lazy retrieval

Enables

Read only relevant pieces when needed.

If missing

Entire files/results must be loaded even when only a few lines matter.

Severity: High

6.4 Context-aware tool-output limits

Enables

Dynamically reduce output size when the remaining context budget is low.

If missing

Tools can unknowingly consume the model's remaining budget.

Severity: High

7. File and Attachment Handling

7.1 File indexing instead of wholesale injection

Weak approach:

700-page PDF -> extract everything -> place everything in context

Better:

PDF
 |
Index externally
 |
Search relevant chunks
 |
Load only required pages/sections

If missing

Large documents crowd out conversation history.

Severity: Critical

7.2 Retrieval-on-demand

Enables

Fetch only the section relevant to the current question.

If missing

File size directly determines context cost.

Severity: High

7.3 Detach file from active context

Enables

Stop an irrelevant attachment from influencing subsequent work.

If missing

Old files continue consuming attention/context.

Severity: High

7.4 File scope controls

Enables

Choose which files are visible to which thread or agent.

If missing

Agents receive irrelevant or excessive data.

Severity: Medium to High

8. Persistent Instructions and Memory

Useful instruction layers:

Platform policy
|
Organization policy
|
User instructions
|
Project instructions
|
Repository instructions
|
Skill instructions
|
Conversation instructions
|
Current prompt

Features

Account-level custom instructions
Personality/style preferences
User memory
Project instructions
Project-only memory
Repository instructions
Agent-specific instructions
Skill-specific instructions

If these are missing

Users repeatedly restate the same constraints.
Instructions drift between sessions.
Tokens are wasted on boilerplate.
Behavior becomes inconsistent.

9. Projects and Workspaces

A useful structure:

Workspace
└── Project
    ├── conversations/
    ├── files/
    ├── memory/
    ├── instructions/
    ├── agents/
    ├── tasks/
    └── artifacts/

Useful features

Multiple conversations in one project
Shared project files
Project instructions
Project memory
Persistent goals
Project search
Collaborators
Permissions
Shared artifacts

If missing

Related information is scattered across unrelated chats.
Context must repeatedly be reconstructed.
Projects cannot outlive individual conversation windows.

10. Artifact Persistence

An artifact should not be trapped inside the message that created it.

Useful features

Persistent generated files
Artifact library
Search artifacts
Reuse in another conversation
Version artifacts
Folder organization
Download
Share
Permission management
Link artifact to project/task

If missing

Important generated code/documents become difficult to rediscover.
Users repeatedly regenerate or manually download/re-upload work.

11. Rich and Editable Outputs

Rich renderers

A harness can render:

Markdown
Tables
Charts
Maps
Code
Diffs
Documents
Images
Diagrams
Interactive widgets
Apps
HTML previews

A useful architecture:

Model output
   |
Structured response
   |
Renderer selection
   ├── Markdown
   ├── Table
   ├── Chart
   ├── Map
   ├── Code
   ├── Document
   ├── Diff
   └── Widget

If missing

Everything is flattened into text.
Complex work becomes harder to inspect and manipulate.

Editable artifacts

Useful capabilities:

Directly edit generated text
Select text and ask AI to modify only that selection
Undo/redo
Fullscreen editor
Continue chatting from the edited artifact state
Edit code in place
Preview code
Run code

If missing

Tiny manual corrections require another full AI turn.
User edits and agent edits become poorly integrated.

12. Versioning

Useful versioning primitives:

artifact v1
   |
AI edit
   |
artifact v2
   |
manual edit
   |
artifact v3
   |
restore v1

Features

Artifact version history
Undo/redo
Conversation branches
Checkpoints
Diff review
Git history
Restore prior version

If missing

Changes become difficult to audit.
Users hesitate to experiment because recovery is unclear.

13. Code Execution

Useful harness features:

Python execution
Other language execution
stdout/stderr
exit status
timeout
generated files
runtime metrics
dependency management
network controls

Possible interface:

execute_code(
    language,
    source,
    timeout,
    filesystem,
    network,
    dependencies
)

If missing

Model can suggest code but cannot verify it.

14. Shell / Terminal Access

Capabilities:

Run commands
Run tests
Run linters
Install dependencies
Run builds
Start dev servers
Inspect processes
Use Git
Use project CLIs

This changes the workflow from:

"Here is a command you should run."

to:

"I ran it, inspected the result, fixed the issue, and verified the fix."

If missing

The user remains the execution bridge for every operation.

15. Sandboxing

Useful controls:

sandbox:
  filesystem:
    read: [...]
    write: [...]
  network:
    allow: [...]
  commands:
    allow: [...]
  processes:
    limits: ...

Possible modes:

Read only
Workspace write
Full access
No external network
Allowlisted network

If missing

Autonomous execution may be unsafe or impossible.

16. Human Approval and Permissions

A good harness should support granular policy.

Example:

Allow automatically:
- read repository
- edit repository
- run tests
- localhost access

Ask:
- git push
- external network
- delete files
- modify secrets

Deny:
- ~/.ssh
- unrelated directories

Possible permission scopes:

Allow once
Allow for current turn
Allow for session
Allow for project
Always ask
Always deny

If missing

The user must choose between constant approval prompts and dangerous broad access.

Severity: Critical

17. Web Search and Research

Basic web retrieval

Features:

Search
Open page
Follow links
Find text
Extract structured information
Cite sources
Source preview

If missing

Model relies only on internal knowledge for current information.

Long-horizon research

Features:

Break question into subproblems
Search many sources
Revise research strategy
Synthesize results
Maintain citations
Accept user steering during research

If missing

Complex research requires manual orchestration.

18. Browser and Computer Use

Features:

Open websites
Click
Type
Scroll
Fill forms
Use authenticated sites
Interact with desktop apps
Read screen state

Difference:

Web search = retrieve information
Browser/computer agent = interact with software

If missing

Model can explain steps but cannot perform UI workflows.

19. External Tools, MCP, Skills, Plugins, Hooks

A useful distinction:

Tools
    Can DO something

Skills
    Know HOW to do something

Plugins
    Package tools + skills + UI

Hooks
    React to lifecycle events

Tool registry

Examples:

GitHub
Gmail
Calendar
Slack
Linear
Database
Internal APIs

MCP

Allows standardized external tool/data providers.

Skills

Reusable workflow instructions, examples, templates, scripts, references, and assets.

Plugins

A packaged integration containing some combination of:

tools
skills
UI
structured results
authentication
hooks

Hooks

Useful lifecycle hooks:

on_session_start
before_tool_call
after_tool_call
before_file_write
after_file_write
before_commit
after_task
on_error

If extension mechanisms are missing

Every new integration requires modifying the harness core.
Reusable workflows cannot be packaged cleanly.

20. Multimodality

Inputs

Text
Images
Screenshots
PDFs
Documents
Spreadsheets
Audio
Voice
Camera capture

Outputs

Text
Images
Audio
Files
Charts
Diagrams
Websites
Code
Documents
Presentations
Interactive widgets

A message should therefore be modeled as something closer to:

Message {
    text[]
    images[]
    files[]
    audio[]
    tool_results[]
    artifacts[]
}

rather than a string.

21. Git and Coding Harness Features

Repository awareness

Inspect repository
Search code
Read status
Understand project structure
Respect repository instructions

Git operations

status
diff
branch
checkout
commit
push
pull
merge
rebase
open PR
inspect PR
inspect CI

Worktree isolation

Without worktrees:

Agent A -> edits app.py
Agent B -> edits app.py

Potential result:

conflicting edits
inconsistent tests
contaminated state
hard-to-understand Git status

With worktrees:

repo
├── agent-A-worktree/
├── agent-B-worktree/
└── agent-C-worktree/

If missing

Parallel coding agents can overwrite or observe partial work from one another.

Severity: Critical

Diff UI

Features:

Changed-file list
Inline additions/deletions
Review comments
Open in editor
Accept/reject change
Summarize diff

If missing

"Done" provides very little assurance.
User must manually inspect entire files/repository.

Multi-repository projects

Useful for:

frontend/
backend/
infra/
shared-library/

If missing

Cross-repository tasks require separate isolated sessions and manual coordination.

22. Environment Management

An execution environment may contain:

repository
branch
worktree
dependencies
setup commands
environment variables
secrets
network policy
filesystem permissions
caches

Useful features:

environment templates
automatic setup script
dependency caching
environment validation
dev-server startup
environment selection per task

If missing

Every task repeatedly rebuilds setup.
Agent results depend on hidden machine state.

23. Secrets

Useful harness features:

Encrypted secret storage
Project-specific secrets
Environment-specific secrets
OAuth
Scoped credentials
Secret redaction
Controlled injection into processes
Prevent secrets entering model output

If missing

Users paste secrets directly into conversation or environment.
Security risk increases dramatically.

24. Local vs Cloud Execution

Useful choices:

Local
├── local files
├── local terminal
└── local machine

Cloud
├── isolated runtime
├── delegated execution
└── persistent remote task

If missing

Users cannot choose between privacy/local integration and scalable remote execution.

25. Background / Delegated Work

Features:

Start long-running task
Leave the interface
Task continues
Return later
Inspect output
Resume conversation
Receive notification

If missing

Long-running work requires the user to remain attached to the session.

26. Task Progress

Useful states:

Queued
Planning
Searching
Reading
Editing
Running tests
Waiting for approval
Blocked
Completed
Failed
Cancelled

If missing

Users do not know whether the agent is progressing, stuck, or idle.

27. Scheduled and Event-Triggered Work

Scheduled tasks

One-off task
Recurring task
Daily
Weekly
Monthly
Reminder
Scheduled research

Event-triggered agents

Example:

TRIGGER
  |
CONDITION
  |
AGENT TASK
  |
ACTION / NOTIFICATION

Possible triggers:

New email
New GitHub event
Slack message
File change
Webhook
Calendar event

Conditional monitoring

Example:

Keep checking X and notify me only when Y becomes true.

Requires:

recurring execution
persistent state
comparison against previous run
notification suppression when unchanged

If missing

Monitoring tasks require manual checking.

28. Notifications

Potential channels:

In-app
Desktop
Mobile push
Email
Task card

If missing

Background tasks cannot effectively re-engage the user.

29. Multi-Agent Execution

Possible topology:

              Root Agent
           /      |       \
     Researcher  Coder   Reviewer
           \      |       /
             Synthesis

Useful features:

Spawn subagent
Give subagent limited context
Assign task
Restrict tools
Run concurrently
Receive result
Terminate
Synthesize

If missing

Root agent serializes everything and wastes context.

30. Context Separation Between Agents

Bad:

Every subagent receives entire master conversation.

Consequences:

token use multiplies
irrelevant data distracts each agent
confidential/unneeded information spreads
context fills faster

Better:

Security agent -> auth files + threat model
Frontend agent -> UI files + UI requirements
DB agent -> schema + migration requirements

Feature name: Context routing / scoped subagent context

Severity if missing: High

31. Agent Handoffs

Example:

Planner
   |
Designer
   |
Developer
   |
Tester
   |
Reviewer

If missing

One agent must retain every role and all context.

32. Parallel Execution

Example:

Task
├── investigate backend bug
├── investigate frontend bug
├── inspect tests
└── inspect documentation

Then synthesize findings.

If missing

Independent work executes serially and takes longer.

33. Agent Isolation

Parallel agents may require isolation of:

context
filesystem
Git tree
tools
permissions
execution state
environment variables

If missing

Parallelism introduces cross-contamination.

34. Planning

Possible workflow:

Goal
 |
Plan
 |
Steps
 |
Execution
 |
Verification

Useful UX:

Show plan
Edit plan
Approve plan
Reorder steps
Mark complete
Replan after failure

If missing

Long tasks can become opaque and hard to steer.

35. Goals and Task Hierarchy

Useful model:

Project
└── Goal
    ├── Task
    ├── Task
    └── Task
         ├── agent run
         └── artifacts

If missing

Long-term work is flattened into unrelated conversation turns.

36. Verification Loop

A coding harness should support:

Implement
 |
Run test
 |
Failure
 |
Inspect
 |
Fix
 |
Run test again
 |
Pass

If missing

Agent may stop after editing code without demonstrating correctness.

37. Code Review Agent

Capabilities:

Inspect diff
Detect bugs
Detect regressions
Detect security issues
Follow project review rules
Produce structured findings

If missing

Implementation and validation rely on the same unchecked reasoning path.

38. Human-in-the-Loop Review

User controls:

Inspect plan
Approve tool call
Inspect command
Inspect diff
Reject change
Modify output manually
Continue agent from modified state

If missing

Agent workflow becomes either overly autonomous or overly manual.

39. Retry and Failure Recovery

Every tool or task should distinguish:

Success
Failure
Timeout
Cancelled
Permission denied
Network unavailable
Partial success

Useful recovery actions:

Retry tool
Retry turn
Retry with stronger model
Retry from checkpoint
Change environment
Change permission
Restore previous state

If missing

Transient errors can derail entire workflows.

40. Resumability

Example:

✓ investigation
✓ architecture
✓ backend
✓ migration
○ frontend
○ tests

If the browser or machine disconnects, a resumable harness can continue from the remaining items.

If missing

Interrupted long-running tasks may need to restart.

Severity: High

41. Observability

Useful information:

Current model
Current agent
Task state
Tool being used
Commands executed
Files changed
Network calls
Elapsed time
Token usage
Cost
Context usage
Subagents
Failures
Approvals
Logs

If missing

Agent behavior becomes difficult to understand, trust, optimize, or debug.

42. Tool-Call Transparency

Useful progress messages:

Searching web...
Reading file...
Running pytest...
Editing app.py...
Calling GitHub...

If missing

The agent may appear frozen or magical.
Users cannot intervene at useful moments.

43. Provenance and Citations

A strong harness should retain:

Claim
 |
Source
 |
Retrieval timestamp
 |
Tool call / query

This helps distinguish:

model memory
current documentation
GitHub issue
web source
retrieved file
tool result

If missing

Verification becomes expensive.
Hallucinations are harder to identify.

44. Audit Trail

Example:

User requested X
Agent ran command Y
Tool returned Z
Agent changed file A
User approved B
Agent pushed commit C

Useful features:

chronological event log
actor identity
timestamps
export
link to artifacts/diffs

If missing

It becomes difficult to reconstruct what happened after a failure.

45. Usage and Resource Tracking

Useful metrics:

Tokens
Inference cost
Tool cost
CPU
RAM
Execution time
Network
Storage
Browser time

If missing

Autonomous agents can consume resources without clear accountability.

46. Sharing and Collaboration

Shareable objects may include:

Chat
Project
File
Folder
Artifact
Code block
Task
Plugin
Generated app/site

Collaboration roles:

Viewer
Editor
Owner

Useful features:

shared files
shared instructions
comments
project-level permissions
shared memory/context where appropriate

If missing

Collaboration falls back to screenshots, copied text, and manual handoffs.

47. Search Across Personal/Project Knowledge

Unified search could cover:

Search
├── conversations
├── documents
├── generated files
├── images
├── projects
├── repositories
├── connected drives
└── connected apps

If missing

Previous work effectively disappears once the user forgets which thread/file contains it.

48. Personalization

Useful controls:

Base style/personality
Tone
Verbosity
Custom instructions
Memory
Preferred language
Preferred output format
Preferred tools
Accessibility preferences
Reasoning preference

If missing

Users repeatedly correct presentation and workflow preferences.

49. Specialized Interaction Modes

A harness may expose different interaction policies:

Chat
Study
Research
Code
Work
Review
Brainstorm
Tutor

The model may be similar, but orchestration, instructions, and UI change.

If missing

Every use case is forced through the same interaction pattern.

50. Remote Control

Useful coding-agent capability:

Desktop machine performs work
Mobile/web acts as control plane
User can inspect progress
Approve operations
Queue prompts
Steer agent
Review diffs

If missing

Agent supervision is tied to the execution machine.

51. Cross-Surface Continuity

Potential surfaces:

Web
Desktop
Mobile
Terminal
IDE
Slack
API
Voice

If missing

State becomes fragmented by interface.

52. IDE Integration

Features:

Current-file awareness
Selected-code awareness
Repository context
Inline editing
Diffs
Terminal
Diagnostics
Test integration

If missing

Coding assistant loses the developer's immediate working context.

53. CLI

Useful CLI features:

Local filesystem access
Shell-native workflow
Pipes
Scripts
Automation
MCP
Project configuration
Permission policies

If missing

Agent cannot integrate naturally with developer automation.

54. SDK / Embeddability

Harness-as-library enables:

CI agent
GitHub bot
Internal developer agent
Ticket-solving agent
Migration agent
Support workflow

If missing

Harness functionality remains locked to one UI.

55. Security Controls

A serious harness may require:

Sandboxing
Filesystem restrictions
Network restrictions
Command restrictions
Domain allowlists
Approval gates
Secret management
OAuth
Role permissions
Plugin permissions
Tool permissions
Project permissions
Workspace policies
Audit logging
Data retention controls
Temporary/no-history mode

If missing

Increased risk of unwanted or unauthorized actions.

56. Privacy Controls

Useful features:

Temporary conversations
History controls
Delete conversation
Delete files
Memory controls
Data export
Training/data-use controls
Retention controls

If missing

Users cannot meaningfully control persistence and reuse of their data.

57. Enterprise Administration

Potential features:

RBAC
Workspace policies
SSO
Allowed/blocked tools
Allowed plugins
Network policies
Admin analytics
Environment controls
Auditability
Retention policy

If missing

Organization-wide deployment becomes difficult to govern.

58. Recommended Top-Level Harness Architecture

A practical taxonomy:

AI HARNESS
│
├── 1. Conversation
│   ├── edit
│   ├── retry
│   ├── branch
│   ├── delete
│   ├── interrupt
│   ├── resume
│   └── history
│
├── 2. Context
│   ├── context meter
│   ├── compaction
│   ├── selective inclusion
│   ├── retrieval
│   ├── memory
│   └── instructions
│
├── 3. Models
│   ├── model selection
│   ├── reasoning effort
│   └── routing
│
├── 4. Tools
│   ├── web
│   ├── browser
│   ├── shell
│   ├── Python
│   ├── computer use
│   └── APIs
│
├── 5. Extensions
│   ├── MCP
│   ├── skills
│   ├── plugins
│   └── hooks
│
├── 6. Agent Runtime
│   ├── planning
│   ├── subagents
│   ├── parallelism
│   ├── delegation
│   └── background work
│
├── 7. Environment
│   ├── filesystem
│   ├── repository
│   ├── terminal
│   ├── sandbox
│   ├── network
│   └── secrets
│
├── 8. Artifacts
│   ├── files
│   ├── documents
│   ├── code
│   ├── images
│   ├── sites
│   └── previews
│
├── 9. Versioning
│   ├── diff
│   ├── undo
│   ├── checkpoint
│   ├── branch
│   └── Git/worktrees
│
├── 10. Human Control
│   ├── approvals
│   ├── steering
│   ├── comments
│   ├── review
│   └── permissions
│
├── 11. Persistence
│   ├── projects
│   ├── library
│   ├── schedules
│   ├── notifications
│   └── cross-device state
│
└── 12. Observability
    ├── progress
    ├── tool calls
    ├── logs
    ├── citations
    ├── cost
    └── audit trail

59. High-Priority Features for a Custom Harness

If prioritizing implementation, these are especially valuable:

Edit any previous prompt and fork execution from there
Explicit conversation branching
Clone conversation up to any turn
Delete/exclude individual turns from active context
Context usage meter
Large-message warning before send
Automatic and manual context compaction
Conversation handoff summary
Retry with different model/reasoning level
Interrupt and steer an active agent
Queue follow-up instructions
Persistent projects
Persistent artifacts independent of chats
Hierarchical instructions
Skills
MCP/tool plugins
Sandboxed shell
Granular approvals
Diff-based file review
Checkpoints and rollback
Git worktrees per agent
Parallel subagents
Background execution
Scheduled/event-triggered execution
Full execution trace/audit log
Cross-device resume
Remote agent control
Rich output components
Editable artifacts
Artifact versioning
Search across all conversations/files/projects
Local + cloud execution
Reusable environments
Verification/test loops
Human approval gates
Large tool-output externalization
Selective retrieval for files
Context routing for subagents
Resumability after crash/disconnect
Provenance for model/tool/retrieval claims

60. Feature-by-Feature Failure Checklist

Use this checklist when testing a harness.

Conversation recovery

Can I edit an earlier prompt?
Can I branch from any turn?
Can I clone the thread up to a selected turn?
Can I delete a turn?
Can I exclude a turn from active model context?
Can I retry a response without adding another message?
Can I retry with another model?
Can I restore a checkpoint?
Can I recover from an accidental giant paste?

Context

Can I see context usage?
Do I get a warning before a huge message consumes the remaining window?
Can the harness compact older context?
Can I request compaction manually?
Can I inspect what consumes context?
Can I pin critical context?
Can irrelevant historical turns be omitted?
Can I transfer a compact summary to a fresh thread?

Files and tools

Are giant tool outputs externalized?
Can I retrieve tool output lazily?
Are large files indexed instead of fully injected?
Can I detach irrelevant files?
Can I limit file scope per agent?

Agent runtime

Can I interrupt execution?
Can I steer while execution is running?
Can I queue a follow-up?
Can I resume after disconnect?
Can agents run in the background?
Can independent work run in parallel?
Can subagents receive scoped context only?

Coding

Does each parallel agent get an isolated worktree?
Can I inspect diffs?
Can I restore prior code state?
Does the agent run tests after edits?
Can a reviewer agent inspect the implementation?
Are Git operations visible and auditable?

Security

Are filesystem permissions scoped?
Is network access scoped?
Can I allow an operation once/session/project?
Are secrets handled separately from conversation text?
Are destructive operations gated?

Observability

Can I see what tool is currently running?
Can I see commands executed?
Can I see files changed?
Can I inspect failures?
Can I see token/context/cost usage?
Is there an audit trail?
Are factual claims linked to provenance?

Persistence

Are artifacts independent of chat lifetime?
Can I search old conversations?
Can I search across projects?
Can I resume work from another device?
Can scheduled/background tasks notify me?

61. Failure Severity Scale

A simple severity scale can make harness comparisons more useful.

Severity	Meaning
Critical	Missing feature can cause irreversible loss of useful state, unsafe behavior, or make long-running tasks unusable
High	Missing feature regularly causes substantial wasted work, manual reconstruction, or poor reliability
Medium	Missing feature causes friction but has a reasonable workaround
Low	Primarily convenience or polish

Examples:

No branch-from-message: Critical
No context meter: High
No queued prompts: Medium
No custom appearance setting: Low

62. Key Evaluation Principle

Do not evaluate only:

Edit messages: Yes / No
Branch: Yes / No
MCP: Yes / No

Evaluate:

Feature
   |
What user workflow does it enable?
   |
What failure mode appears without it?
   |
How recoverable is that failure?
   |
How severe is the missing capability?

This turns a feature checklist into a harness reliability specification.

63. Final Principle

A model can be extremely capable and still feel weak inside a poor harness.

A large portion of practical AI capability comes from the surrounding system:

state management
context management
retrieval
persistence
versioning
execution
sandboxing
permissions
orchestration
observability
recovery
rich UI
artifacts

The most important design question is therefore not only:

"Can the AI do this?"

but also:

"If something goes wrong halfway through, can the user recover without losing the work that came before it?"

udaylunawat/ai_harness_feature_inventory_and_failure_modes.md

AI Harness Feature Inventory and Failure Modes

1. Core Design Principle: Conversation Recoverability

2. Example Failure: Accidental Huge Message

Recovery path A: Edit the message

Recovery path B: Delete the message

Recovery path C: Branch from the previous message

Recovery path D: Clone conversation up to M7

Recovery path E: Start a new conversation with a compact handoff

Recovery path F: Exclude M8 from active context

3. Reusable Harness Comparison Matrix

4. Conversation State Management

4.1 Edit previous user message

4.2 Retry / regenerate response

4.3 Retry with another model or reasoning level

4.4 Branch from any earlier message

4.5 Clone conversation up to message X

4.6 Delete an individual turn

4.7 Exclude a message from model context

4.8 Conversation checkpoints

4.9 Restore checkpoint

4.10 Interrupt generation or running agent

4.11 Steer a running agent

4.12 Queue next instruction

4.13 Resume previous session

4.14 Cross-device continuation

5. Context Management

5.1 Context usage meter

5.2 Remaining-context estimate before send

5.3 Unusually-large-message confirmation

5.4 Automatic context compaction

5.5 Manual compaction

5.6 Context inspector

5.7 Selective context loading

5.8 Pin important context

5.9 Conversation handoff summary

5.10 New thread from summary

5.11 Cross-thread references

5.12 Project-level context

6. Tool Output and Large Data Handling

6.1 Tool-output truncation

6.2 External storage for large tool output

6.3 Lazy retrieval

6.4 Context-aware tool-output limits

7. File and Attachment Handling

7.1 File indexing instead of wholesale injection

7.2 Retrieval-on-demand

7.3 Detach file from active context

7.4 File scope controls

8. Persistent Instructions and Memory

Features

9. Projects and Workspaces

Useful features

10. Artifact Persistence

Useful features

11. Rich and Editable Outputs

Rich renderers

Editable artifacts

12. Versioning

Features

13. Code Execution

14. Shell / Terminal Access

15. Sandboxing

16. Human Approval and Permissions

17. Web Search and Research

Basic web retrieval

Long-horizon research

18. Browser and Computer Use

19. External Tools, MCP, Skills, Plugins, Hooks

Tool registry

MCP

Skills

Plugins

Hooks

20. Multimodality

Inputs

Outputs

21. Git and Coding Harness Features

Repository awareness

Git operations