Status: Draft — log structure defined; initial entries seeded from closed/in-progress EG-### gaps Frameworks: ISO 42001 (Clause 10, A.10.4), ISO 27001:2022 (Clause 10), NIST AI RMF (GOVERN-1.5) Review cycle: Quarterly Owner: Compliance lead (log maintenance + quarterly review) · Platform engineering (gap closure evidence) Vanta controls:
iso42001CPL-68, CPL-60 ·nistAiRmfGOV-3100, GOV-3107 Vanta test:approved-aims-improvement-policy-exists(NEEDS_ATTENTION)
This log closes the ISO 42001 PDCA loop (Plan → Do → Check → Act) by recording every improvement to Scrydon's AI systems — whether driven by incident review, gap closure, customer feedback, monitoring trends, or regulatory change. It is the primary evidence artifact for:
- Vanta test
approved-aims-improvement-policy-exists(currently NEEDS_ATTENTION) - ISO 42001 Clause 10 (Improvement) — requires documented continual improvement actions
- NIST AI RMF GOVERN-1.5 — AI risk governance includes mechanisms for organizational learning
Improvements recorded here are the output of the PDCA cycle. Inputs come from the source channels below.
Applies to all AI systems in the AI System Inventory and the AI governance controls documented in the compliance framework.
| Source | Cadence | Input to Log |
|---|---|---|
| Incident review | Per incident | Post-incident corrective actions |
| Customer feedback | Ongoing; reviewed quarterly | Feature and safety improvement requests |
| Internal engineering review | Per PR / per sprint | Engineering gap closures (EG-### IDs) |
| AIUC-1 quarterly delta | Quarterly | New AIUC-1 requirements from changelog |
| Post-Deployment Monitoring trends | Weekly (automated) + quarterly (manual review) | Drift events, threshold breaches |
| External security / pen-test findings | Annually + per engagement | Security remediation actions |
| Compliance framework updates | Per framework publication | Framework alignment gaps |
Each entry in the log follows this structure:
| Field | Description |
|---|---|
| ID | Unique improvement ID (format: AI-IMP-YYYY-NNN) |
| Source | Source channel (e.g., EG-019 gap closure, monitoring trend, customer feedback) |
| AI System(s) | Affected system(s) from inventory |
| Description | What was improved and why |
| Hypothesis | Expected outcome after improvement |
| Metric | How improvement will be measured |
| Owner | Engineering or compliance owner |
| Target Date | Planned completion date |
| Status | Open / In Progress / Closed |
| Evidence | PR number, ADR, test results, Vanta evidence link |
| Closed Date | Date closed (when Status = Closed) |
AI-IMP-2026-001 — DB TLS Enforcement
| Field | Value |
|---|---|
| Source | EG-002 (Sensitive columns in plain text — DB TLS gap) |
| AI System(s) | All (auth DB, agentic DB, analytics DB, ontology DB) |
| Description | Enforced TLS on all Postgres connections in production via shared resolveSslMode() helper; in-cluster Postgres now terminates TLS via self-signed cert |
| Hypothesis | Eliminates risk of credential/token interception over plaintext DB connections |
| Metric | ssl=on confirmed in Postgres server logs; NODE_ENV=production → ssl: 'require' behavior verified |
| Owner | Platform engineering |
| Target Date | 2026-04-16 |
| Status | Closed |
| Evidence | packages/common/src/node/db-tls.ts; helm/scrydon/templates/infra/db.yaml; helm/scrydon/templates/infra/db-tls-secret.yaml |
| Closed Date | 2026-04-16 |
AI-IMP-2026-002 — Default-On Inline DLP (EG-019)
| Field | Value |
|---|---|
| Source | EG-019 (No mandatory inline LLM content DLP — critical) |
| AI System(s) | Workflow Engine, Copilot, Agents, Embedding Pipeline |
| Description | Implemented withDlp capability-layer decorator — default-on PII detection / content scanning / classifier signals on every LLM/STT/TTS/embedding call. Fail-closed on missing scanContext. Streaming scanned per flush window. 13 dlp.* audit events with strict Zod schemas. |
| Hypothesis | Every LLM call is DLP-scanned; PII leakage via unguarded egress is eliminated as a default failure mode |
| Metric | 100% of LLM calls wrapped by withDlp; 0 calls that bypass decorator (verified in PR review); audit events emitted per scan |
| Owner | Platform engineering |
| Target Date | 2026-04-21 |
| Status | Substantially closed (residuals: Guardrails v2 migration + EG-012 prompt-injection validator) |
| Evidence | packages/sdk-authoring/src/integrations/dlp/with-dlp.ts; ADR 2026-04-17; PR #767 |
| Closed Date | 2026-04-21 |
AI-IMP-2026-003 — SIEM Streaming Infrastructure (EG-005)
| Field | Value |
|---|---|
| Source | EG-005 (No SIEM streaming for on-premises monitoring) |
| AI System(s) | All (audit log system) |
| Description | Implemented HMAC-signed webhook subscriptions per org (at-least-once delivery + DLQ), NDJSON streaming export (date-range, paginated), per-org retention with nightly chunked DELETE. Dapr cron components in production Helm. |
| Hypothesis | Customers can forward audit events to their SIEM in near-real-time with cryptographic authenticity |
| Metric | Webhook subscription delivers events within 10 min of emission; DLQ depth = 0 under normal load |
| Owner | Platform engineering |
| Target Date | 2026-05-08 |
| Status | Substantially closed (customer runbook polish pending) |
| Evidence | packages/better-auth-audit-log/src/routes/audit-log-subscriptions.ts; helm/scrydon/templates/auth/dapr-cron-audit-log.yaml; security/siem-forwarding |
| Closed Date | 2026-05-08 |
AI-IMP-2026-004 — Classification Enforcement Foundation (EG-020)
| Field | Value |
|---|---|
| Source | EG-020 (Bell–LaPadula classification enforcement) |
| AI System(s) | RAG / Knowledge Base, Agents, Embedding Pipeline |
| Description | Shipped classification schema (migrations 0024–0026 + 0047–0048), BLP aggregation helper, clearance_egress rule, user_clearance table with four-eyes overrides, classification library, DLP extensions, retrieval clearance filter, and ClassificationPanel admin UI. |
| Hypothesis | Documents above user clearance are invisible at retrieval; tool dispatch to over-clearance endpoints is blocked; spillage events are detected and attributed |
| Metric | Integration test: cross-clearance query returns 0 rows; security.clearance_egress_blocked emitted on violation |
| Owner | Platform engineering |
| Target Date | 2026-05-06 |
| Status | Foundation closed (integration follow-up: wire loadEffectiveUserClearance into all search() callers; per-document classification UI; chunk-level propagation) |
| Evidence | packages/db-agentic/schema.ts (migrations); packages/better-auth-integrations/src/classification/; ADR 2026-04-24 |
| Closed Date | 2026-05-06 |
AI-IMP-2026-005 — AI Quality Metric Persistence (EG-008)
| Field | Value |
|---|---|
| Source | EG-008 (AI performance monitoring not persisted) |
| AI System(s) | Workflow Engine, Guardrails, Copilot |
| Description | Add guardrails_results table to agentic DB; add OTel span attributes for hallucination score and PII entity counts; create AI governance Grafana dashboard |
| Hypothesis | Hallucination rates, PII trigger rates, and guardrails effectiveness become measurable as time-series trends |
| Metric | Hallucination score captured in DB for > 95% of Guardrails-enabled workflow runs; Grafana dashboard shows 7-day rolling hallucination rate |
| Owner | Platform engineering |
| Target Date | Q3 2026 |
| Status | Open |
| Evidence | EG-008; apps/agentic/lib/core/telemetry.ts (existing OTel instrumentation) |
| Closed Date | — |
AI-IMP-2026-006 — Prompt Injection Detection (EG-012)
| Field | Value |
|---|---|
| Source | EG-012 (No prompt injection detection — Vanta risk R-19) |
| AI System(s) | Workflow Engine, Copilot, Agents |
| Description | Implement pattern-based prompt-injection validator in Guardrails engine; add to adversarial CI test suite; sanitize user-provided systemPrompt in workflow blocks |
| Hypothesis | Common injection patterns (role-play override, instruction injection, indirect RAG injection) are detected and blocked at the withDlp boundary |
| Metric | > 95% detection rate on curated injection corpus; 0 jailbreaks on test set |
| Owner | Platform engineering |
| Target Date | Q3 2026 |
| Status | Open |
| Evidence | EG-012; packages/sdk-authoring/src/integrations/templates/scrydon/guardrails/validators/ |
| Closed Date | — |
AI-IMP-2026-007 — PII Test Coverage Expansion (EG-015)
| Field | Value |
|---|---|
| Source | EG-015 (PII detection test coverage gaps) |
| AI System(s) | withDlp, Guardrails |
| Description | Add test cases for all 27 entity types (currently ~6 tested); create benchmark dataset per region (US, UK, EU, APAC); measure and document precision/recall per entity type |
| Hypothesis | PII detection accuracy is measurable and consistently ≥ 90% recall across all entity types |
| Metric | 27/27 entity types have ≥ 3 test cases; precision ≥ 95%, recall ≥ 90% per entity (where detection method allows) |
| Owner | Platform engineering |
| Target Date | Q3 2026 |
| Status | Open |
| Evidence | EG-015; packages/sdk-authoring/src/integrations/templates/scrydon/guardrails/__tests__/validators.test.ts:84-266 |
| Closed Date | — |
AI-IMP-2026-008 — Agent Autonomy Framework (EG-016)
| Field | Value |
|---|---|
| Source | EG-016 (Incomplete agent autonomy framework) |
| AI System(s) | Agents, Copilot |
| Description | Add workspace-level autonomy policy; classify tools by risk level; persist approval decisions to audit log; add batch cancellation endpoint |
| Hypothesis | Admins can enforce tool approval requirements org-wide; approval decisions are auditable; emergency stop available for batch operations |
| Metric | Workspace-level policy enforced for all tool calls in affected orgs; approval decisions in auth.audit_log; batch cancellation API functional |
| Owner | Platform engineering |
| Target Date | TBD |
| Status | Open |
| Evidence | EG-016; apps/agentic/app/api/copilot/confirm/route.ts |
| Closed Date | — |
| Role | Responsibility |
|---|---|
| Compliance lead | Chairs review; updates Vanta improvement evidence; signs off on closure criteria |
| Engineering lead | Reports on open improvements; provides evidence for closed items |
| Security lead | Reviews security-relevant improvements; confirms adversarial eval results |
- Export of all open improvement entries from this log.
- Post-Deployment Monitoring trend report for the quarter.
- AIUC-1 changelog delta (new requirements from quarterly AIUC-1 refresh).
- Vanta test status for
approved-aims-improvement-policy-existsand related tests.
- Updated status for each improvement entry.
- New entries for improvements identified during the review.
- Evidence package uploaded to Vanta for closed items.
- Meeting minutes stored in Vanta under the AIMS improvement evidence folder.
| Activity | Timing |
|---|---|
| Quarterly review meeting | First week of each quarter |
| Vanta evidence upload | Within 5 business days of review meeting |
| AIUC-1 changelog review | Within 2 weeks of each quarterly AIUC-1 publication |
An improvement entry is closed when all of the following are true:
- The described change is merged to
main(or the relevant production branch). - Automated tests cover the changed behavior and pass in CI.
- The corresponding EG-### entry in gaps is updated to reflect the new status.
- Evidence is uploaded to Vanta (PR link, test results, or ADR).
- The compliance lead has reviewed and confirmed the closure.
Partial closures (e.g., foundation shipped, integration follow-up pending) are recorded as Substantially closed with an explicit list of residual items.
| Issue | Description | Impact |
|---|---|---|
| No formal AIUC-1 quarterly delta review process | AIUC-1 refreshes quarterly; no process exists to triage new requirements | New AIUC-1 requirements may not be captured in this log |
| Improvement meeting minutes not yet stored in Vanta | Quarterly reviews not yet started | Vanta test approved-aims-improvement-policy-exists NEEDS_ATTENTION |
| EG-021 | No quarterly external eval to feed improvement inputs | Improvement loop lacks external adversarial test data |
- Schedule first quarterly improvement review (target: Q3 2026, first week of July)
- Upload this document and initial log entries to Vanta as evidence for
approved-aims-improvement-policy-exists - Establish AIUC-1 quarterly delta review process (assign owner, add to engineering calendar)
- Close AI-IMP-2026-001 through AI-IMP-2026-004 evidence in Vanta with PR/ADR references
- Add improvement review meeting minutes template to Vanta evidence folder
- Link quarterly review outputs to Post-Deployment Monitoring and Value Maintenance frameworks
- Define target date for AI-IMP-2026-008 (EG-016) based on engineering roadmap