Markdown importer design spec (superpowers, excluded from repo)

spec.md

Markdown Importer — Design Spec

Overview

Add a Markdown-to-ContentState importer to draftjs_exporter, complementing the existing ContentState-to-HTML/Markdown exporter. The design follows a two-phase architecture: parsing (Markdown text → valid ContentState) then filtering (ContentState → ContentState with content policy applied). A public MarkdownImporter class wires them together with a config shape that mirrors the exporter's pattern.

Architecture

Markdown text
     │
     ▼
┌─────────────────────┐
│   MarkdownParser    │  ← Phase 1: "engine"
│   (parser config)   │     Guarantees structural integrity.
└────────┬────────────┘
         │  ContentState (always valid)
         ▼
┌─────────────────────┐
│  ContentStateFilter │  ← Phase 2: filtering
│   (filter rules)    │     Enforces content policy.
└────────┬────────────┘
         │  ContentState (valid, filtered)
         ▼
┌─────────────────────┐
│  MarkdownImporter   │  ← Public API class
│  wires parser +     │     Config(config).import_markdown(md)
│  filter together    │
└─────────────────────┘

New modules

• draftjs_exporter/markdown_importer/ — MarkdownImporter class and config builder
• draftjs_exporter/markdown_parser/ — parsing engine, no dependencies; includes resolvers.py with the scheme_resolver helper
• draftjs_exporter/contentstate_filter/ — filtering engine, reusable independently

Cross-cutting decisions

• Single engine with a clear seam: the parser is referenced by dotted path in config, matching the exporter's engine pattern. A third-party parser (e.g., mistune-backed) can be swapped in later.
• No external dependencies — the built-in parser is hand-written.
• CommonMark core coverage: paragraphs, ATX headings, blockquotes, fenced code, thematic breaks, unordered/ordered lists with depth tracking, bold, italic, inline code, links, images, hard line breaks. No reference-style links, HTML blocks, Setext headings, or indented code.
• Configurable entity resolution: link and image URLs pass through a resolver chain before entities are created. This supports both "public" Markdown (![alt](/media/...jpg) → IMAGE with src/alt) and "internal" representations (![alt](wagtail://image?id=10&alt=alt&format=left) → IMAGE with id/src/alt/format) with the same parser.
• Limited inline HTML: a configurable whitelist of inline HTML tags (e.g., <sup>, <sub>) maps to inline styles, for formatting that has no Markdown equivalent. Unrecognized HTML is treated as literal text — never as markup — so there is no XSS surface at the ContentState level.

MarkdownParser — Phase 1

Interface

class ParserConfig(TypedDict, total=False):
    headings: bool                  # default True
    blockquote: bool                # default True
    code_fenced: bool               # default True
    thematic_break: bool            # default True
    unordered_list: bool            # default True
    ordered_list: bool              # default True
    emphasis: bool                  # default True  — bold + italic
    code_inline: bool               # default True
    links: bool                     # default True
    images: bool                    # default True
    line_breaks: bool               # default True  — hard breaks only
    link_resolvers: list[EntityResolver]    # default []
    image_resolvers: list[EntityResolver]   # default []
    inline_html_styles: dict[str, str]      # default {} — e.g. {"sup": "SUPERSCRIPT"}


class MarkdownParser:
    __slots__ = ("config",)

    def __init__(self, config: ParserConfig | None = None) -> None: ...
    def parse(self, markdown: str) -> ContentState: ...

Parsing approach

A hand-written recursive descent parser operating on a line iterator. No AST — the parser emits ContentState blocks directly. Block-level parsing handles constructs that span lines (lists, blockquotes, fenced code). Inline parsing uses character-by-character scanning with delimiter-run tracking for emphasis resolution (matching opener/closer pairs at block end).

Construct mapping

Construct	ContentState output
Paragraph (blank-line separated)	unstyled block
ATX headings (# through ######)	header-one through header-six
> blockquote	blockquote block, > prefix stripped per line
Fenced code (``` or ~~~)	code-block block
--- / *** / ___ thematic break	atomic block with HORIZONTAL_RULE entity
* /- /+ unordered lists	unordered-list-item blocks with depth
1. ordered lists	ordered-list-item blocks with depth
**bold** / __bold__	BOLD inline style range
*italic* / _italic_	ITALIC inline style range
`code`	CODE inline style range
[text](url)	LINK entity — data from link resolver chain (default: {"url": url})
	IMAGE entity — data from image resolver chain (default: {"src": url, "alt": alt})
Hard line break (two spaces + newline)	\n in block text
<sup>text</sup> (if configured)	SUPERSCRIPT inline style range
<sub>text</sub> (if configured)	SUBSCRIPT inline style range

Safety guarantee — structural integrity

Every code path produces either a valid ContentState or raises MarkdownParseError. Invariants: entity keys are monotonically increasing, block keys are unique, depth values match list nesting, entity ranges always have corresponding entityMap entries, inline style ranges are non-overlapping and in-bounds.

Error handling

Strict parsing — unexpected input raises MarkdownParseError with line number and message. No silent degradation.

Entity resolution

When the parser encounters a link or image, it does not create the entity directly. Instead it passes the URL (and the label or alt text) through a resolver chain. Each resolver either returns a resolution or None to defer to the next resolver. If no resolver matches, the default applies.

class EntityResolution(TypedDict, total=False):
    type: str                # entity type, e.g. "LINK", "IMAGE", "DOCUMENT", "EMBED"
    data: dict[str, Any]     # entity data
    mutability: Mutability   # default "MUTABLE"


# Receives (url, label_or_alt). Returns a resolution, or None to defer.
EntityResolver = Callable[[str, str], EntityResolution | None]

• Link resolvers receive (url, link_text). The link text always becomes the block's children; resolvers only control the entity type and data.
• Image resolvers receive (url, alt_text). Resolvers control the entity type and data, including how the alt text is incorporated.
• Resolvers can return any entity type — e.g., a wagtail://document?id=1 link URL resolves to a DOCUMENT entity, not LINK.

Defaults (empty resolver chains):

• Links: {"type": "LINK", "data": {"url": url}}
• Images: {"type": "IMAGE", "data": {"src": url, "alt": alt}}

Shipped helper — scheme_resolver: the library provides a generic resolver factory for scheme-based internal URLs, in draftjs_exporter.markdown_parser.resolvers:

def scheme_resolver(
    scheme: str,                       # e.g. "wagtail"
    type_map: dict[str, str],          # URL host → entity type
                                       # e.g. {"image": "IMAGE", "page": "LINK"}
    coerce: dict[str, Callable[[str], Any]] | None = None,
                                       # per-key data coercion, e.g. {"id": int}
    label_key: str | None = None,      # data key filled from the Markdown label
                                       # e.g. "alt" for images; None for links
) -> EntityResolver: ...

It parses URLs of the form scheme://kind?key=value&...: the host selects the entity type via type_map, query parameters become entity data (with optional coercion), and — when label_key is set — a non-empty Markdown label fills that data key if the query string didn't already provide it. Returns None for URLs that don't match the scheme or have an unmapped host.

Example — Wagtail-style internal syntax:

config: ImporterConfig = {
    "parser_config": {
        "link_resolvers": [
            scheme_resolver(
                "wagtail",
                {"page": "LINK", "document": "DOCUMENT", "user": "LINK"},
                coerce={"id": int},
            ),
        ],
        "image_resolvers": [
            scheme_resolver(
                "wagtail",
                {"image": "IMAGE", "media": "EMBED"},
                coerce={"id": int},
                label_key="alt",
            ),
        ],
        "inline_html_styles": {
            "sup": INLINE_STYLES.SUPERSCRIPT,
            "sub": INLINE_STYLES.SUBSCRIPT,
        },
    },
}

# [label](wagtail://page?id=3)
#   → LINK entity, data {"id": 3}
# ![alt](wagtail://image?id=10&alt=alt&format=left)
#   → IMAGE entity, data {"id": 10, "alt": "alt", "format": "left"}
# ![alt](/media/...jpg)  (no resolver match → default)
#   → IMAGE entity, data {"src": "/media/...jpg", "alt": "alt"}

Limited inline HTML

The inline parser recognizes paired tags <tag>content</tag> where tag is a key of inline_html_styles, and wraps the content in the corresponding inline style range. Content is parsed recursively, so <sup>**bold**</sup> produces overlapping SUPERSCRIPT and BOLD ranges.

• Tags with attributes (<sup class="x">) are not recognized and pass through as literal text.
• Tags not in the whitelist pass through as literal text (< and > characters are preserved in the block text).
• No block-level HTML, no void elements, no arbitrary HTML — the whitelist only maps paired inline tags to inline styles.
• Default is an empty whitelist: no HTML is interpreted.

Escaping (backslash escapes)

The inline parser inverts CommonMark backslash escapes: any ASCII punctuation character preceded by \ is imported as that literal character. The accepted set is the full CommonMark escapable punctuation set (!"#$%&'()*+,-./:;<=>?@[\]^_{|}~`), so the importer accepts user-authored escapes beyond the subset the exporter emits.

Link and image destinations are unescaped: \(, \), and \\ inside ](…) revert to literal characters, inverting the exporter's escape_link_destination. Percent-encoded bytes the exporter emits for whitespace and control characters are left intact — percent-encoding is not a Markdown escape.

Code span delimiters are matched by equal-length backtick runs, mirroring the exporter's code_span_delimiters: a span whose content contains a backtick is opened and closed by a longer run, and CommonMark padding spaces are stripped. Backslash escapes are not processed inside code spans.

Known round-trip gaps (deferred):

• Underscore flanking is not implemented. The exporter's default italic marker is _, and it leaves intraword _ runs unescaped (treating them as inert per CommonMark flanking) while also emitting intraword _ markers for legitimate mid-word italic. Those two cases are structurally identical to plain identifiers like foo_bar_baz, so the importer cannot distinguish them; applying CommonMark flaking here would break mid-word italic round-trips (e.g. fan_tastic_ for italic on "tastic"). Resolving this fully requires an exporter-side change (use * for italic when the marker would be intraword, or default italic to *).

ContentStateFilter — Phase 2

Interface

# Callable receives the matched object depending on rule type:
# - block rules: the block dict (Block)
# - entity rules: the entity dict (Entity)
# - inline_style rules: the style name string
# Returns a replacement object of the same shape, or None to remove.
FilterCallback = Callable[[Block | Entity | str], Block | Entity | str | None]

FilterAction = Literal["remove", "keep", "demote"] | FilterCallback


class FilterRule(TypedDict, total=False):
    type: Literal["block", "inline_style", "entity"]
    match: str          # type to match (e.g., "header-one")
    action: FilterAction


class ContentStateFilter:
    __slots__ = ("rules",)

    def __init__(self, rules: list[FilterRule] | None = None) -> None: ...
    def apply(self, content_state: ContentState) -> ContentState: ...

Built-in actions

Action	Effect
"remove"	Delete matching block/entity/style (block content is lost)
"keep"	No-op — useful as explicit default
"demote"	Headings only: H1→H2, …, H6→removed
Callable	Receives the matched object, returns replacement or None to remove

Processing model

Walk blocks in order. For each block: check block-type rules, strip disallowed inline styles from inlineStyleRanges, resolve entity rules against entityRanges (sync with entityMap, prune orphaned keys). When a block is removed, its inline styles and entities are discarded with it — no cascading cleanup is needed on sibling blocks.

After list-item removals, recalculate depths: the depth of each remaining list item is determined by counting how many consecutive list-wrapper openings are pending at that position, maintaining valid Draft.js nesting.

Rules run in definition order. Default is keep (not deny).

Safety guarantee — content policy

Rules are declarative. Callable returns are validated before insertion. The filter never produces invalid ContentState.

MarkdownImporter — Public API

Interface

class ImporterConfig(TypedDict, total=False):
    parser: str                     # dotted path, default: built-in
    parser_config: ParserConfig     # options passed to parser
    filter_rules: list[FilterRule]  # rules applied after parsing


class MarkdownImporter:
    __slots__ = ("parser", "filter")

    def __init__(self, config: ImporterConfig | None = None) -> None: ...
    def import_markdown(self, markdown: str) -> ContentState: ...

Usage examples

from draftjs_exporter import BLOCK_TYPES, MarkdownImporter

# Simple import
importer = MarkdownImporter()
cs = importer.import_markdown("# Hello\n\nWorld")

# Demote level-1 headings
importer = MarkdownImporter({
    "filter_rules": [
        {"type": "block", "match": BLOCK_TYPES.HEADER_ONE, "action": "demote"},
    ],
})

# Disable blockquotes
importer = MarkdownImporter({
    "parser_config": {"blockquote": False},
})

Error surface

class MarkdownParseError(Exception):
    """Raised when Markdown input cannot be parsed."""
    line: int | None
    message: str

Public API additions to draftjs_exporter.__init__

New exports: MarkdownImporter, MarkdownParser, ContentStateFilter, ImporterConfig, ParserConfig, FilterRule, MarkdownParseError, EntityResolution, EntityResolver, scheme_resolver.

Testing strategy

Unit tests

• tests/markdown_parser/test_parser.py — each construct in isolation, edge cases
• tests/markdown_parser/test_inline_parser.py — emphasis, links, images, escapes
• tests/markdown_parser/test_parser_config.py — config toggles disable constructs
• tests/markdown_parser/test_parser_errors.py — malformed input → MarkdownParseError
• tests/markdown_parser/test_resolvers.py — resolver chain ordering, defaults, custom resolvers returning non-default entity types (e.g., DOCUMENT, EMBED)
• tests/markdown_parser/test_scheme_resolver.py — the shipped helper: scheme matching, host→type mapping, query param extraction, coercion, alt fallback
• tests/markdown_parser/test_inline_html.py — whitelisted tags → styles, recursive content, tags with attributes ignored, non-whitelisted tags literal
• tests/contentstate_filter/test_filter.py — each action type per target kind
• tests/contentstate_filter/test_filter_rules.py — ordering, validation, defaults
• tests/markdown_importer/test_importer.py — wiring, config defaults, error propagation
• Escaping round-trip cases in tests/test_imports.py (direct class) and escaping direct-import fixtures in tests/test_imports.json, mirroring draftjs_exporter.markdown.escape output to lock the round-trip contract.

Snapshot tests

New tests/test_imports.py, following the test_exports.py pattern:

• Round-trip over existing fixtures (tests/test_exports.json): import each fixture's recorded output.markdown and compare structurally with the fixture's content_state. Most fixtures round-trip identically. The fixture format gains an optional "import" field: when present, it records the expected post-import ContentState for fixtures with known information loss (e.g., "Style map defaults" — single-tilde strikethrough is GFM, out of importer scope) instead of maintaining a skip-list in test code.
• Direct import fixtures (new tests/test_imports.json): Markdown → ContentState cases for what existing fixtures cannot cover — syntax variants the exporter never emits (*italic*, + bullets, ~~~ fences, *** thematic breaks, 1) ordered lists), and importer-only features (inline_html_styles whitelists, internal-URL resolvers, parser toggles).
• The "Style map defaults" fixture doubles as an integration test for inline_html_styles: with a full whitelist configured, only the strikethrough range is lost on round-trip.
• Extend tests/test_exports.json with Markdown → ContentState entries.

Property-based tests

• Extend tests/test_properties.py:
  • Round-trip: import(export(cs)) preserves block types and entity data
  • Filter idempotency: filter(filter(cs)) == filter(cs)
  • Filter validity: for any valid ContentState, filter(cs) is also valid

Coverage target

100% per project convention.

Example and documentation

Example script

Update example.py to demonstrate importing Markdown and re-exporting it. The updated script shows both directions: Markdown → ContentState → HTML, and ContentState → Markdown → ContentState.

Documentation

New docs page at docs/markdown-importer.md covering:

• Getting started with MarkdownImporter
• Parser config reference (all toggles)
• Entity resolution: resolver chains, defaults, scheme_resolver helper, writing custom resolvers, internal-URL syntax example (e.g., wagtail://)
• Limited inline HTML: inline_html_styles whitelist, safety model
• Filter rule reference (all actions, rule types)
• Safety guarantees
• Custom parser engines (swapping the parser)

Update docs/index.md to link to the new page. Update .agents/skills/draftjs-exporter/SKILL.md to include the new public API.

Out of scope

• Reference-style links ([text][ref]) and link reference definitions
• Block-level HTML and arbitrary inline HTML — unrecognized HTML always passes through as literal text; only whitelisted paired inline tags (via inline_html_styles) are interpreted
• HTML tags with attributes (pass through as literal text)
• Setext headings (===, ---)
• Indented code blocks (4-space indent)
• Tables (GFM extension)
• Autolinks (<url>)
• Nested emphasis edge cases (middle-of-word matching)
• Underscore flanking rules (intraword snake_case emphasis resolution)
• Empty list items