portable-storge-specs.md

Generating Portable Storage Specs

How we get from the S3 API specifications to a vendor-neutral, machine-readable subset.

The gist

• Filter the AWS smithy spec down to the operations defined in tier-1.yml
• Rewrite the API docs to be vendor neutral and coherent for that subset
• Merge the new docs with the bare bones smithy spec to create the new one
• Translate the smithy spec into an openAPI spec.

Once the baseline spec is agreed:

• We test tools and services against it, generating a smithy spec for each.
• We diff the spec against our baseline to assert compliance or identify divergence.

The layout

storage repo

• tier-1.yaml: Define the allowed operations, parameters, and headers.
• operations/*.md: Documentation for each operation.
• Spec Outputs: The generated rfc-storage-tier-1.smithy.json and rfc-storage-tier-1.openapi.yaml.

storage-spec-cli repo

• bootstrap: Extracts a subset of shapes and operations from a source AWS Smithy model.
• compile: Combines smithy AST and markdown docs to create new smithy and openapi spec.
• test: Record a compatibilty test run for a service.
• diff: Compare smithy ASTs for compatibility or schema drift.

Getting started

Ensure both the storage and storage-spec-cli repositories are cloned as siblings. Link the CLI globally to run it locally:

cd ../storage-spec-cli
npm install
npm run build
npm link
cd ../storage

The storage-spec binary is now available globally.

Workflow

1. Bootstrap the smithy AST - Download and filter based on tier-1.yaml. Extracts initial Markdown doc templates under operations/:

   storage-spec bootstrap --source ./tier-1.yaml --output ./rfc-storage-tier-1.smithy.bare.json --extract-docs ./operations

2. Refine the docs - Edit the extracted markdown files in operations/ to improve the descriptions.
3. Compile the Specs - Merge the bare AST and the md docs into the final Smithy AST and OpenAPI artifacts:

   storage-spec compile --input ./rfc-storage-tier-1.smithy.bare.json --docs ./operations --output-smithy ./rfc-storage-tier-1.smithy.json --output-openapi ./rfc-storage-tier-1.openapi.yaml

4. Generate HTML Documentation Builds the OpenAPI YAML into a client-facing HTML document:

   npx @redocly/cli build-docs ./rfc-storage-tier-1.openapi.yaml -o ./docs/index.html -t ./docs/template.hbs

Specification Formats

Smithy AST JSON (rfc-storage-tier-1.smithy.json)

• Purpose: Serves as the primary source of truth.
• Rationale: S3 routes multiple distinct operations (e.g., CopyObject and PutObject) to identical HTTP paths and methods using headers and query parameters, and relies on AWS SigV4 traits. Smithy models these protocol-specific traits and signatures natively.
• Usage: Client routing engines, service proxies, and compliance validators.

OpenAPI 3.1 YAML (rfc-storage-tier-1.openapi.yaml)

• Purpose: Developer-facing REST representation.
• Rationale: Exposes S3 features to standard API tooling ecosystems (Redocly, Postman, SDK generators).
• Limitations: The S3 API does things that cannot be described in openAPI. See: openapi-issues.md

The Pipeline

1. Bootstrap Flow

Extracts a subset of shapes and operations from a source AWS Smithy model using the tier manifest as a positive list, producing a bare AST and markdown document templates.

graph LR
    aws_ast["AWS S3 Smithy AST"] -->|Input Model| bootstrap["storage-spec bootstrap"]
    manifest["tier-X.yaml"] -->|Positive List| bootstrap
    bootstrap -->|Generate| bare_ast["rfc-storage-tier-X.smithy.bare.json"]
    bootstrap -->|Extract Templates| op_docs["operations/*.md"]

Loading

2. Compilation Flow

Combines the bare AST and manually curated markdown documentation to produce the final documented Smithy AST and OpenAPI specifications.

graph LR
    bare_ast["rfc-storage-tier-X.smithy.bare.json"] -->|Input| compile["storage-spec compile"]
    op_docs["operations/*.md (Curated Prose)"] -->|Input| compile
    compile -->|Generate| doc_ast["rfc-storage-tier-X.smithy.json"]
    compile -->|Generate| openapi["rfc-storage-tier-X.openapi.yaml"]

Loading

3. Compliance Test Flow

Runs a local proxy to intercept client S3 request traffic and build a compatible operations log.

graph LR
    client["S3 Client SDK"] -->|Request Traffic| test_proxy["storage-spec test"]
    test_proxy -->|Forward Request| target["Target S3 Provider (MinIO/Ceph)"]
    test_proxy -->|Record Traversed APIs| comp_ast["compatible-s3.json"]

Loading

4. Spec Diff Flow

Performs static schema auditing against the baseline specification to identify drift or check provider compatibility.

graph TD
    subgraph Drift ["Schema Drift Auditing"]
        doc_ast2["rfc-storage-tier-X.smithy.json"] -->|Baseline Spec| diff_drift["storage-spec diff"]
        aws_ast2["AWS S3 Smithy AST"] -->|Incoming AWS Updates| diff_drift
        diff_drift -->|Report| drift_report["test-report.md"]
    end

    subgraph Compliance ["Provider Compliance Auditing"]
        doc_ast3["rfc-storage-tier-X.smithy.json"] -->|Baseline| diff_comp["storage-spec diff"]
        provider_ast["provider-s3-ast.json"] -->|Compatible| diff_comp
        diff_comp -->|Report| comp_report["compliance-report.md"]
    end

    Drift ~~~ Compliance

Loading

Compliance Testing

We can audit specifications and test compatibility of S3 storage providers (like MinIO, Ceph, or Garage) using the diff and test commands.

Identifying Differences (Drift) between Specs

To see how our subset compares to the complete AWS S3 model (for example, to identify which S3 operations are not currently supported by our tier), we compare our baseline spec with the incoming S3 updates model:

storage-spec diff rfc-storage-tier-1.smithy.json aws-s3-smithy-ast.json

Checking Provider Compatibility

To check if a specific storage provider conforms to our portable standard:

1. Run the provider locally or in staging.
2. Run your application client test suite through the storage-spec test proxy to capture all S3 calls and output a compatible-s3.json profile representing the provider's API surface.
3. Compare the generated profile against our baseline spec to highlight any differences, unsupported operations, or parameter mismatches:

storage-spec diff rfc-storage-tier-1.smithy.json compatible-s3.json