Frozen synthetic tool catalog + expected authority map, pre-registered before testing Verb Authority beta .14. see README for protocol.

README.md

mixed-authority-fixtures

A fully synthetic tool catalog for testing per-argument authority classification in agent tooling, plus a frozen expected-authority map.

What this is

10 tools, 23 arguments. Every argument is labeled in expected_authority_map.json as one of:

• application — must come from trusted application code or configuration, never model-authored
• model — the model may freely author it
• review_required — intentionally ambiguous; needs a human policy decision

The catalog is designed around traps, not just easy cases:

1. A destination hiding in a non-messaging tool (create_ticket.assignee_email)
2. Free text that is actually a command (run_report_query.sql)
3. A destination-like arg on a read-only tool (fetch_public_page.url)
4. The same arg shape on a read tool vs a write tool (thread_id vs doc_id)
5. False-lock bait: consequential-feeling but low-risk args (priority, refund reason, status message with links)
6. The designed ambiguous middle (schedule_meeting.attendees)

Test protocol (pre-registered)

1. This catalog and the expected map are frozen and published BEFORE running the tool under test.
2. Tool under test: Verb Authority public beta .14, run unchanged.
3. The catalog is not tuned after seeing output.
4. Reported afterward: expected vs actual per argument, time to first useful finding, any confusing step, false locks (over-restriction), and missed protected fields (under-restriction).

Safety

Everything here is synthetic. No real systems, vendors, people, or endpoints. Any example domains use RFC 2606 reserved TLDs. The catalog describes no real deployment.

License

MIT

catalog.json

{
  "catalog_name": "mixed-authority-fixtures",
  "version": "1.0.0",
  "description": "Fully synthetic MCP-style tool catalog for testing per-argument authority classification. No real systems, vendors, or endpoints. All domains use RFC 2606 reserved TLDs.",
  "tools": [
    {
      "name": "send_email",
      "description": "Send an email from the workspace account.",
      "parameters": {
        "to": { "type": "string", "description": "Recipient email address." },
        "subject": { "type": "string", "description": "Subject line." },
        "body": { "type": "string", "description": "Email body text." }
      }
    },
    {
      "name": "summarize_thread",
      "description": "Read-only. Produce a summary of a conversation thread.",
      "parameters": {
        "thread_id": { "type": "string", "description": "Identifier of the thread to summarize." },
        "style": { "type": "string", "description": "Summary style, e.g. brief or detailed." }
      }
    },
    {
      "name": "fetch_public_page",
      "description": "Read-only. Fetch a public web page and return its text.",
      "parameters": {
        "url": { "type": "string", "description": "URL of the page to fetch." }
      }
    },
    {
      "name": "create_ticket",
      "description": "Create a support ticket in the tracking system.",
      "parameters": {
        "title": { "type": "string", "description": "Short ticket title." },
        "description": { "type": "string", "description": "Full ticket description." },
        "assignee_email": { "type": "string", "description": "Email of the person to assign." },
        "priority": { "type": "string", "description": "One of low, medium, high, urgent." }
      }
    },
    {
      "name": "run_report_query",
      "description": "Run a SQL query against the reporting warehouse and return rows.",
      "parameters": {
        "sql": { "type": "string", "description": "SQL query text to execute." },
        "row_limit": { "type": "integer", "description": "Maximum rows to return." }
      }
    },
    {
      "name": "update_document",
      "description": "Overwrite the content of an existing document.",
      "parameters": {
        "doc_id": { "type": "string", "description": "Identifier of the document to update." },
        "content": { "type": "string", "description": "New document content." }
      }
    },
    {
      "name": "schedule_meeting",
      "description": "Create a calendar meeting and invite attendees.",
      "parameters": {
        "attendees": { "type": "array", "items": { "type": "string" }, "description": "Email addresses to invite." },
        "agenda": { "type": "string", "description": "Meeting agenda text." },
        "duration_minutes": { "type": "integer", "description": "Meeting length in minutes." }
      }
    },
    {
      "name": "refund_customer",
      "description": "Issue a refund to a customer account.",
      "parameters": {
        "account_id": { "type": "string", "description": "Customer account identifier." },
        "amount": { "type": "number", "description": "Refund amount in account currency." },
        "reason": { "type": "string", "description": "Free-text reason recorded in the audit log." }
      }
    },
    {
      "name": "add_labels",
      "description": "Attach organizational labels to a ticket.",
      "parameters": {
        "ticket_id": { "type": "string", "description": "Ticket to label." },
        "labels": { "type": "array", "items": { "type": "string" }, "description": "Label strings to attach." }
      }
    },
    {
      "name": "post_status_update",
      "description": "Post a status message to the team status channel configured for this workspace.",
      "parameters": {
        "message": { "type": "string", "description": "Status text to post. May include links for context." }
      }
    }
  ]
}

expected_authority_map.json

{
  "note": "Frozen BEFORE running Verb Authority beta .14. Authority values: application (must come from trusted application code or config, never model-authored), model (model may freely author), review_required (intentionally ambiguous, needs human policy decision). Mapping to Verb Authority terms as I understand them: application ~ trusted_fixed, model ~ outbound_payload or unrestricted, review_required ~ locked/review.",
  "expectations": {
    "send_email": {
      "to": { "expected": "application", "why": "Canonical destination. Model-authored recipient is the exfiltration path." },
      "subject": { "expected": "model", "why": "Free text payload." },
      "body": { "expected": "model", "why": "Free text payload. This is the work the model exists to do." }
    },
    "summarize_thread": {
      "thread_id": { "expected": "model", "why": "TRAP (false-lock bait): an ID, but the tool is read-only with output returned to the same session. Locking read selectors on a no-side-effect tool is over-restriction. A reasonable tool may argue review; application would be a clear false lock." },
      "style": { "expected": "model", "why": "Cosmetic free choice." }
    },
    "fetch_public_page": {
      "url": { "expected": "review_required", "why": "TRAP (destination on a read-only tool): a GET has no write side effect, but a model-composed URL can exfiltrate session data via query parameters. Neither pure model nor pure application is obviously right; this is a policy call." }
    },
    "create_ticket": {
      "title": { "expected": "model", "why": "Free text." },
      "description": { "expected": "model", "why": "Free text." },
      "assignee_email": { "expected": "application", "why": "TRAP (destination hiding in a non-messaging tool): routes a notification and grants attention/ownership. Should come from a directory lookup, not the model." },
      "priority": { "expected": "model", "why": "TRAP (enum that feels consequential): bounded enum with low blast radius. Locking it is over-restriction; the enum constraint is the control." }
    },
    "run_report_query": {
      "sql": { "expected": "review_required", "why": "TRAP (free text that is actually a command): looks like payload, executes as code. At minimum needs a gateway (read-only grants, table allowlist). Classifying it plain model is a miss." },
      "row_limit": { "expected": "model", "why": "Bounded numeric with trivial blast radius." }
    },
    "update_document": {
      "doc_id": { "expected": "review_required", "why": "TRAP (same arg shape as summarize_thread.thread_id, different verb): a write-target selector. Whether the app must pin it depends on deployment scope. The interesting test is whether the tool treats doc_id differently here than on the read-only tool." },
      "content": { "expected": "model", "why": "Free text payload." }
    },
    "schedule_meeting": {
      "attendees": { "expected": "review_required", "why": "Intentionally ambiguous: sometimes legitimately model-derived from conversation context, sometimes must be pinned. The catalog's designed review_required case for a destination-like array." },
      "agenda": { "expected": "model", "why": "Free text." },
      "duration_minutes": { "expected": "model", "why": "Bounded numeric, low consequence." }
    },
    "refund_customer": {
      "account_id": { "expected": "application", "why": "Destination of funds. Must come from the authenticated case context." },
      "amount": { "expected": "application", "why": "Consequential numeric. Model-authored amounts on money movement is the classic injection payoff. App should derive from order data; at most model proposes, gateway enforces." },
      "reason": { "expected": "model", "why": "TRAP (free text on a highly consequential tool): it is still just an audit-log payload. Locking it is over-restriction; the consequential args are account_id and amount, not the prose." }
    },
    "add_labels": {
      "ticket_id": { "expected": "review_required", "why": "Write-target selector, low blast radius. Reasonable tools may differ; documenting as the low-stakes ambiguous case." },
      "labels": { "expected": "model", "why": "Organizational free text, trivially reversible." }
    },
    "post_status_update": {
      "message": { "expected": "model", "why": "TRAP (payload that smells like a destination vector): text may contain URLs, but the destination channel is fixed by the server config. This is the send_status_update(body) pattern done right; locking message would be a false lock." }
    }
  }
}

va_output.txt

# Verb Authority schema report


> Local static analysis only: no server was executed and no network was used.
> Descriptions, examples, defaults, and runtime values are not included.
> Enum members are always omitted. Non-redacted reports use stable SHA-256
> fingerprints for comparison; these are guessable for low-entropy values.
> Non-redacted reports also fingerprint full schema validation material,
> excluding annotations; those hashes are correlatable and may be guessable.
> Redacted reports omit exact constraint values and all exact schema hashes.


## Summary


| Measure | Count |
|---|---:|
| Tools | 10 |
| Parameters | 23 |
| Protected (`trusted_fixed`) | 19 |
| Data-fillable | 4 |
| Parameters requiring review | 15 |
| Tools requiring review | 10 |
| Schemas requiring review | 2 |
| Tools requiring confirmation | 10 |
| Tool risks requiring review | 10 |
| Branch risks requiring review | 0 |
| Tool risk conflicts | 0 |
| Annotation conflicts | 0 |


Schema fingerprint: `b0a2fe28e31cd0ef526630364295c1794f6a215bb0af1042b19c81ad1c307031`
Names redacted: `no`


## Tool review summary


> Static review debt is separate from runtime confirmation.


| Tool | Review required | Arguments | Schema | Risk | Risk conflict | Annotation conflicts | Branch risk |
|---|---|---|---|---|---|---|---|
| send_email | yes | subject | no | yes | no | — | no |
| summarize_thread | yes | thread_id, style | no | yes | no | — | no |
| fetch_public_page | yes | — | no | yes | no | — | no |
| create_ticket | yes | title, priority | no | yes | no | — | no |
| run_report_query | yes | sql, row_limit | no | yes | no | — | no |
| update_document | yes | doc_id | no | yes | no | — | no |
| schedule_meeting | yes | attendees, agenda, duration_minutes | yes | yes | no | — | no |
| refund_customer | yes | amount, reason | no | yes | no | — | no |
| add_labels | yes | ticket_id, labels | yes | yes | no | — | no |
| post_status_update | yes | — | no | yes | no | — | no |


## Tool risk evidence


| Tool | Effective risk | Source | Name heuristic | Mutability | Declared effects | Conflict | Risk review | Schema review | Confirmation |
|---|---|---|---|---|---|---|---|---|---|
| send_email | unknown | safe_default | write via send (heuristic) | caller | — | no | yes | no | yes |
| summarize_thread | unknown | safe_default | unknown (no complete-token match) (uncertain) | caller | — | no | yes | no | yes |
| fetch_public_page | unknown | safe_default | read_only via fetch (heuristic) | caller | — | no | yes | no | yes |
| create_ticket | unknown | safe_default | write via create (heuristic) | caller | — | no | yes | no | yes |
| run_report_query | unknown | safe_default | unknown (no complete-token match) (uncertain) | caller | — | no | yes | no | yes |
| update_document | unknown | safe_default | write via update (heuristic) | caller | — | no | yes | no | yes |
| schedule_meeting | unknown | safe_default | unknown (no complete-token match) (uncertain) | caller | — | no | yes | yes | yes |
| refund_customer | unknown | safe_default | financial via refund (heuristic) | caller | — | no | yes | no | yes |
| add_labels | unknown | safe_default | write via add (heuristic) | caller | — | no | yes | yes | yes |
| post_status_update | unknown | safe_default | write via post, update (heuristic) | caller | — | no | yes | no | yes |


## Findings


| Tool | Risk | Argument | Type | Required | Constraints | Policy | Review | Reason |
|---|---|---|---|---|---|---|---|---|
| send_email | unknown | to | string | no | — | trusted_fixed | no | authority-bearing name |
| send_email | unknown | subject | string | no | — | trusted_fixed | yes | ambiguous consequential argument; review required |
| send_email | unknown | body | string | no | — | outbound_payload | no | outbound payload name or bounded free text |
| summarize_thread | unknown | thread_id | string | no | — | trusted_fixed | yes | ambiguous consequential argument; review required |
| summarize_thread | unknown | style | string | no | — | trusted_fixed | yes | ambiguous consequential argument; review required |
| fetch_public_page | unknown | url | string | no | — | trusted_fixed | no | authority-bearing name |
| create_ticket | unknown | title | string | no | — | trusted_fixed | yes | ambiguous consequential argument; review required |
| create_ticket | unknown | description | string | no | — | outbound_payload | no | outbound payload name or bounded free text |
| create_ticket | unknown | assignee_email | string | no | — | trusted_fixed | no | authority-bearing name |
| create_ticket | unknown | priority | string | no | — | trusted_fixed | yes | ambiguous consequential argument; review required |
| run_report_query | unknown | sql | string | no | — | trusted_fixed | yes | ambiguous consequential argument; review required |
| run_report_query | unknown | row_limit | integer | no | — | trusted_fixed | yes | ambiguous consequential argument; review required |
| update_document | unknown | doc_id | string | no | — | trusted_fixed | yes | ambiguous consequential argument; review required |
| update_document | unknown | content | string | no | — | outbound_payload | no | outbound payload name or bounded free text |
| schedule_meeting | unknown | attendees | array | no | — | trusted_fixed | yes | ambiguous consequential argument; review required |
| schedule_meeting | unknown | agenda | string | no | — | trusted_fixed | yes | ambiguous consequential argument; review required |
| schedule_meeting | unknown | duration_minutes | integer | no | — | trusted_fixed | yes | ambiguous consequential argument; review required |
| refund_customer | unknown | account_id | string | no | — | trusted_fixed | no | authority-bearing name |
| refund_customer | unknown | amount | number | no | — | trusted_fixed | yes | ambiguous consequential argument; review required |
| refund_customer | unknown | reason | string | no | — | trusted_fixed | yes | ambiguous consequential argument; review required |
| add_labels | unknown | ticket_id | string | no | — | trusted_fixed | yes | ambiguous consequential argument; review required |
| add_labels | unknown | labels | array | no | — | trusted_fixed | yes | ambiguous consequential argument; review required |
| post_status_update | unknown | message | string | no | — | outbound_payload | no | outbound payload name or bounded free text |


## Interpretation boundary


This report describes Verb Authority's declared controls and review heuristics.
A tool name is caller-mutable metadata and is never treated as proof of behavior.
Without an explicit risk declaration, the effective tier remains `unknown` and
requires review and runtime confirmation. This report is not a
vulnerability verdict, does not inspect tool implementations, and does not prove
that the surrounding application supplies correct provenance or authorization.
References and composed/conditional schemas are not resolved; when present,
the report marks the tool for schema review instead of claiming complete coverage.
Review every flagged argument against the real tool semantics before deployment.

Results: Verb Authority 0.10.0b14 vs frozen expected map

Protocol followed: catalog and expected map frozen in this gist before install. Ran verb-authority==0.10.0b14 unchanged (verb-authority scan catalog.json). Catalog not modified after seeing output. Schema fingerprint: b0a2fe28e31cd0ef526630364295c1794f6a215bb0af1042b19c81ad1c307031.

Scoring key: my application ~ trusted_fixed with no review. My model ~ outbound_payload (or unrestricted) with no review. My review_required ~ review flagged. Full rationale for each expectation is in expected_authority_map.json.

Expected vs actual (23 arguments)

Tool.argument	Expected	Actual (b14)	Verdict
send_email.to	application	trusted_fixed, no review	match
send_email.subject	model	review	over-restricted
send_email.body	model	outbound_payload	match
summarize_thread.thread_id	model	review	over-restricted (noted as arguable in frozen map)
summarize_thread.style	model	review	over-restricted
fetch_public_page.url	review_required	trusted_fixed, no review	stricter than expected (safe direction)
create_ticket.title	model	review	over-restricted
create_ticket.description	model	outbound_payload	match
create_ticket.assignee_email	application	trusted_fixed, no review	match (designed trap, caught)
create_ticket.priority	model	review	over-restricted (designed false-lock bait)
run_report_query.sql	review_required	review	match (designed trap, caught)
run_report_query.row_limit	model	review	over-restricted
update_document.doc_id	review_required	review	match
update_document.content	model	outbound_payload	match
schedule_meeting.attendees	review_required	review	match (designed ambiguity)
schedule_meeting.agenda	model	review	over-restricted
schedule_meeting.duration_minutes	model	review	over-restricted
refund_customer.account_id	application	trusted_fixed, no review	match
refund_customer.amount	application	review	weaker than expected (safe direction)
refund_customer.reason	model	review	over-restricted (designed false-lock bait)
add_labels.ticket_id	review_required	review	match
add_labels.labels	model	review	over-restricted
post_status_update.message	model	outbound_payload	match (designed false-lock bait, passed)

Tally

• Exact matches: 11 of 23
• Missed protections (dangerous arg left model-writable): 0 of 23
• Over-restrictions (model-safe arg flagged for review): 10
• Direction-of-caution disagreements: 2 (fetch_public_page.url locked instead of review; refund_customer.amount review instead of confident lock)

Designed traps, outcome

1. Destination hiding in a non-messaging tool (create_ticket.assignee_email): caught confidently
2. Free text that is actually a command (run_report_query.sql): flagged for review, caught
3. Destination-like arg on a read-only tool (fetch_public_page.url): locked outright rather than surfaced for review; safe but skips the policy discussion
4. Same arg shape, read vs write verb (summarize_thread.thread_id vs update_document.doc_id): not differentiated; both received identical treatment even though the tool-risk table itself distinguished the verbs (summarize uncertain, update write)
5. False-lock bait (priority, refund reason, status message): message passed, priority and reason fell in
6. Designed ambiguity (schedule_meeting.attendees): review, as intended

Observations

• The safety headline is clean: nothing protected was left writable. For a tool whose job is preventing untrusted authorship of protected arguments, that is the result that matters most.
• The cost is review volume: 15 of 23 parameters and 10 of 10 tools require review on a 10-tool catalog with no risk annotations. The driver appears to be token-list matching: names matching payload tokens (body, description, content, message) classify confidently; everything else falls to safe-default review.
• Most actionable single finding: the verb signal from the name heuristic (read_only vs write) is present in the risk-evidence table but is not consumed by the argument policy. Wiring those together (read-selector IDs on read-only tools costing less review than write-target IDs) would cut review noise substantially without giving up any protection.
• The report's own framing is honest about all of this (safe_default everywhere absent declared effects), which is to its credit.

Environment: Python 3.12.7, macOS, clean venv, no credentials present. Network during scan: verified by re-running the scan with networking fully disabled; output was identical, confirming local-only operation. Time to first useful finding: near-instant, the full report was produced in seconds.

Yair Sabag reproduced this run from the frozen revision with a clean beta 14 install and got identical counts and classifications. His review also surfaced three errors in my expected-map rationales: priority declares no enum, and row_limit and duration_minutes declare no numeric bounds, so the constraints my rationales assumed do not exist in the frozen catalog. For those three arguments, beta 14's review flag is more defensible than my original scoring allowed.

Revised tally: 11 exact matches, 0 missed protections, 7 over-restrictions, 5 defensible disagreements (originally reported as 10 and 2). The frozen files remain untouched per protocol; this note is the correction of record.

The verb-signal observation stands as a review-noise hypothesis only, not a policy recommendation: a tool name is caller-mutable metadata and must not relax policy. The sanctioned path is owner-supplied risk declarations, which v1.1 of this fixture will test.