auth-token.md

Design: proxying token-based auth (GitHub) alongside HTTP Basic (GitLab)

Status: proposal / not yet implemented. This document describes the problem, the current code's assumptions, and a design + implementation plan for adding GitHub token-auth support without breaking the existing GitLab (HTTP Basic) flow. All file references are relative to git_cdn/ (e.g. util.py means git_cdn/git_cdn/util.py), consistent with doc/overview.md.

Table of contents

1. Problem statement
2. Current architecture (relevant parts)
3. Why "GitHub doesn't support Basic auth" needs unpacking
4. Goals / non-goals
5. Design
6. Backward compatibility
7. Security considerations
8. Testing plan
9. Implementation plan
10. Open questions / risks

---

1. Problem statement

git_cdn currently has exactly one hard-coded assumption about credentials, baked in at three separate points in the code: the client's Authorization header is always Basic base64(user:pass), and git_cdn is allowed to decode it and re-embed the decoded user:pass pair into a git+https://user:pass@host/... URL for local git clone/git fetch subprocesses.

That assumption breaks for GitHub-style token auth, where the credential a client presents isn't naturally a user:pass pair — it's a single opaque token (a Personal Access Token, a GitHub App installation token, or an OAuth token), and it may arrive at git_cdn in a different HTTP scheme (Authorization: Bearer <token> or Authorization: token <token>) rather than pre-packaged as Basic. We need git_cdn to accept those forms from the client and reliably translate them into whatever GitHub's git-http backend actually expects, without regressing the existing GitLab path.

2. Current architecture (relevant parts)

(Full detail in doc/overview.md §2–§4, §10; this section pulls out only what's relevant to auth.)

Credentials only ever come from the inbound Authorization header — git_cdn never parses .netrc (app.py: helpers.netrc_from_env = lambda: None) and never inspects the URL for user:pass@.

Three hard-coded "it's Basic" assumptions:

• check_auth() (git_cdn.py:82-86) — if ENFORCE_AUTH is set and no Authorization header is present at all, replies 401 with WWW-Authenticate: Basic realm="Git Proxy". Doesn't inspect the scheme of a header that is present, so this alone doesn't block other schemes — but it always advertises Basic to clients that show up with nothing.
• get_url_creds_from_auth() (util.py:90-97) — unconditionally base64-decodes whatever follows the first space in the header value and splits on :, with no check for a literal "Basic " prefix:

  def get_url_creds_from_auth(auth):
      creds = base64.b64decode(auth.split(" ", 1)[-1]).decode()
      return ":".join([urllib.parse.quote_plus(p) for p in creds.split(":", 1)])

  Called from handle_upload_pack (git_cdn.py:448-450) right before constructing a RepoCache. A Bearer <token> header fed through this either raises (invalid base64 in most real tokens) or silently produces garbage.
• generate_url() (util.py:100-105) — embeds the already-decoded user:pass string into the upstream URL by string-replacing http(s):// with http(s)://user:pass@, which is then passed straight to git clone --bare <url> / git fetch ... <url> (repo_cache.py:148,181). This is the only place credentials reach the actual git subprocess — HTTP headers aren't available to it, only a URL string.

There are also two places where the raw header is forwarded to upstream verbatim, with no decoding at all — these already work with any scheme, as long as GitHub is happy to receive that exact header back unchanged:

• The dumb-proxy path (proxify_with_data, git_cdn.py:358-393) — used for info/refs, git-receive-pack (push), and info/lfs/objects/batch.
• The upload-pack auth/existence pre-check (handle_upload_pack, git_cdn.py:427-446) — a GET .../info/refs?service=git-upload-pack sent with {"Authorization": auth} copied from the client, before any local work happens.

So the actual gap is narrower than "git_cdn only speaks Basic": only the git-subprocess path (RepoCache/generate_url) requires the header to be Basic, because that's the one place credentials must be turned into a URL string rather than just relayed as a header. The two HTTP-proxy paths are already scheme-agnostic pass-throughs.

remove_git_credentials() (util.py:166-176) and RepoCache.run_git's inline redaction (repo_cache.py:114-121) both assume the secret-to-redact is exactly the self.auth string embedded in a URL — this still works for whatever string ends up in the URL, regardless of scheme, as long as we keep constructing the URL the same way.

repo_cache.py:122-123 also converts a literal "HTTP Basic: Access denied" upstream stderr string into HTTPUnauthorized — GitHub's git-http backend uses different wording for auth failures (see §10), so this check will need a second pattern or a more general one.

3. Why "GitHub doesn't support Basic auth" needs unpacking

This matters because it changes what "support" means here. GitHub retired account-password authentication over Basic auth in 2021, but git's smart-HTTP transport itself is still Basic-auth-shaped on the wire for token credentials — GitHub's own docs tell you to do:

git clone https://<PAT>@github.com/owner/repo.git                       # classic/fine-grained PAT
git clone https://x-access-token:<INSTALLATION_TOKEN>@github.com/owner/repo.git   # GitHub App token

Both of those are still Authorization: Basic base64(user:pass) under the hood — just with a token standing in for the password (and sometimes for the username too). So GitHub isn't asking git_cdn to speak a new wire auth scheme to it; the real requirement is almost certainly on the client → git_cdn side: whatever internal tooling/CI is calling git_cdn holds a GitHub token (PAT, GitHub App installation token, or an OAuth token from a device-flow login) and is not guaranteed to hand it to git_cdn pre-packaged as Basic base64(user:pass) the way a GitLab CI job token integration does today (gitlab-ci-token:<CI_JOB_TOKEN>, see tests/conftest.py). Depending on the calling tool, it may show up as:

• Authorization: Bearer <token> (increasingly common convention for OAuth-style tokens), or
• Authorization: token <token> (GitHub's own API convention, sometimes reused for git-tooling integrations), or
• a raw token with no scheme at all if a caller builds the header by hand.

This needs confirming against the actual calling tooling before finalizing the design — see §10 — but the design below is built to accept any of these without assuming a specific one, since the cost of generality here is small (one more elif branch) and the cost of guessing wrong is a second design doc.

4. Goals / non-goals

Goals

• Accept Authorization headers in Basic, Bearer, and token schemes from clients, in addition to today's Basic-only assumption.
• Reliably produce a GitHub-compatible credential for the git-subprocess path (RepoCache), regardless of which scheme the client used to present it.
• Keep GitLab's existing Basic-auth flow byte-for-byte unchanged — this is an additive change, not a rewrite.
• Make the upstream "flavor" (GitLab-style vs. GitHub-style credential conventions) explicit configuration, not inferred from the URL, since git_cdn is meant to work with either.

Non-goals

• SSH auth (out of scope — README already excludes it; unrelated to this change).
• GitHub App authentication flow itself (JWT → installation token exchange). git_cdn is a proxy, not an identity provider — it assumes the client already holds a valid, ready-to-use token (PAT or installation token) and only needs help getting that token in front of GitHub in the right shape. Minting/refreshing tokens is the caller's responsibility.
• Per-repo or per-request auth-mode switching. One git_cdn deployment fronts one upstream (GITSERVER_UPSTREAM), so one auth mode per deployment is sufficient, matching how GITSERVER_UPSTREAM itself already works.
• Changing the pass-through behavior of the LFS/push/info-refs dumb-proxy paths beyond what's needed for header normalization — they already work as generic relays.

5. Design

5.1 Configuration

Add to settings.py, alongside the existing gitserver_upstream/enforce_auth fields:

class UpstreamAuthMode(str, Enum):
    basic = "basic"     # current behavior: client sends Basic user:pass, relayed as-is
    github = "github"   # client may send Basic/Bearer/token; normalized to GitHub conventions

class Settings(BaseSettings):
    ...
    upstream_auth_mode: UpstreamAuthMode = UpstreamAuthMode.basic
    github_app_username: str = "x-access-token"

• UPSTREAM_AUTH_MODE (env var, via pydantic-settings' existing name-mapping convention) defaults to basic — zero behavior change for existing GitLab deployments that don't set it.
• GITHUB_APP_USERNAME lets an operator override the username GitHub App installation tokens are embedded under (x-access-token by default, matching GitHub's documented convention) without a code change, in case this is ever used against a GitHub Enterprise instance with different requirements.
• Follows the existing test_settings.py convention (monkeypatch.setenv + rebuild Settings()) for its own tests.

5.2 New module: git_cdn/auth.py

Rather than teaching util.py's free functions more special cases inline, pull the scheme-parsing and credential-encoding logic into one small module, since it's about to grow a second axis of variation (scheme × upstream convention) instead of just one (scheme):

@dataclass(frozen=True)
class ParsedCredentials:
    scheme: str            # "basic", "bearer", "token"
    username: str | None   # None if the header carried a bare token, no username
    secret: str            # password, or the bearer/token value itself


def parse_auth_header(auth: str) -> ParsedCredentials | None:
    """Parse a client-supplied Authorization header into scheme + credentials.
    Returns None if the header is missing/unparseable (caller decides what that means)."""
    scheme, _, rest = auth.partition(" ")
    scheme = scheme.lower()
    if scheme == "basic":
        try:
            user, _, password = base64.b64decode(rest).decode().partition(":")
        except (binascii.Error, UnicodeDecodeError):
            return None
        return ParsedCredentials("basic", user, password)
    if scheme in ("bearer", "token"):
        return ParsedCredentials(scheme, None, rest.strip())
    return None


def url_creds(parsed: ParsedCredentials, mode: UpstreamAuthMode, github_app_username: str) -> str:
    """Produce the user:pass string to embed in a git remote URL for subprocess use."""
    if parsed.scheme == "basic":
        user, secret = parsed.username, parsed.secret
    else:
        # bearer/token: no username was supplied by the client, so pick GitHub's
        # documented convention for a token-as-password credential.
        user, secret = github_app_username, parsed.secret
    return ":".join(urllib.parse.quote_plus(p) for p in (user, secret))

get_url_creds_from_auth() and generate_url() in util.py stay as-is for the basic mode (so nothing about the GitLab path changes at the byte level); handle_upload_pack picks between the old function and the new auth.py path based on settings.upstream_auth_mode. This keeps the existing, already-tested test_auth_quote.py behavior completely untouched.

5.3 Integration points

File	Change
settings.py	Add upstream_auth_mode, github_app_username fields (§5.1).
auth.py (new)	parse_auth_header(), url_creds() as above; unit-tested in isolation.
git_cdn.py — handle_upload_pack (git_cdn.py:448-450)	When settings.upstream_auth_mode == "github", replace the get_url_creds_from_auth(auth) call with auth.parse_auth_header(auth) → auth.url_creds(...); on parse failure, return 401 instead of crashing on a bad base64 decode (today's code doesn't guard this at all for malformed Basic headers either — worth fixing regardless, see §10).
git_cdn.py — check_auth() (git_cdn.py:82-86)	WWW-Authenticate value becomes mode-dependent: Basic realm="Git Proxy" unchanged for basic mode; for github mode, omit WWW-Authenticate entirely (or send Bearer realm="Git Proxy") since there's no useful interactive credential prompt to trigger for a token flow the way there is for a human typing a GitLab password.
repo_cache.py — run_git() (repo_cache.py:122-123)	The "HTTP Basic: Access denied" stderr match only covers GitLab's wording. Add GitHub's actual auth-failure stderr text once confirmed against a real 401 (§10) — likely a second literal string check, not a regex, consistent with the existing style.
util.py	Unchanged — kept exactly as today for the basic mode path, per §5.2.

The two HTTP-proxy pass-through paths (proxify_with_data, and the pre-check GET in handle_upload_pack) need no code change in basic mode. In github mode, they should re-encode the outgoing Authorization header to Basic before forwarding — rather than relaying a Bearer/token header byte-for-byte — so GitHub's git-http backend always receives the one auth shape it's documented to accept for git operations, independent of how the client phrased it. Concretely: headers["Authorization"] = "Basic " + base64(url_creds(...)) right after fix_headers() in proxify_with_data, and the equivalent right before building the pre-check headers dict in handle_upload_pack, both gated on settings.upstream_auth_mode == "github".

5.4 Credential-redaction

remove_git_credentials() and RepoCache.run_git's inline self.auth redaction both operate on whatever ends up embedded in the URL string, not on the original header — since §5.2 still produces a user:pass-shaped string for the URL regardless of the original scheme, no changes needed here. Logging of the raw incoming Authorization header is already handled generically by hide_auth_on_headers() (log.py) and is scheme-agnostic already (it masks the header value, not its decoded contents).

6. Backward compatibility

UPSTREAM_AUTH_MODE defaults to basic. With no config change, every code path described above takes the exact branch it takes today — same functions, same call sites, same test coverage. A GitLab deployment upgrading git_cdn sees zero behavior difference. GitHub support is opt-in per deployment via UPSTREAM_AUTH_MODE=github, matching the existing one-upstream-per-deployment model (GITSERVER_UPSTREAM already implies "this instance talks to exactly one git host").

7. Security considerations

• Basic-auth-over-TLS-only applies equally to token credentials — nothing here changes the existing requirement (README: "BasicAuth is only safe over TLS", enforced at the nginx/HAProxy layer per doc/deployment.md); a Bearer/token header re-encoded to Basic is exactly as sensitive as today's GitLab password and must only ever travel over TLS, same as now.
• GitHub tokens (especially fine-grained PATs and installation tokens) are typically shorter-lived than a GitLab CI job token. A long-running git fetch inside RepoCache.update() (§4 of doc/overview.md) that outlives the token's validity window will simply fail with an auth error partway through, same failure mode as an already-revoked GitLab token today — no new handling needed, but worth calling out in the GitHub-specific deployment doc once written, since the failure will show up more often given shorter token lifetimes.
• check_auth()'s relaxed WWW-Authenticate in github mode (§5.3) is a UX choice, not a security boundary — auth is still re-validated against upstream on every request regardless of what challenge header was sent (per doc/overview.md §2's description of the pre-check flow).

8. Testing plan

• tests/test_auth.py (new) — unit tests for parse_auth_header()/url_creds(): Basic (existing behavior parity check against get_url_creds_from_auth), Bearer <token>, token <token>, malformed/empty headers, and the GitHub-App-username default/override. Mirrors the existing test_auth_quote.py conventions (same repo, same style, can live alongside it or replace it if util.py's functions are eventually folded into auth.py — not proposed here to keep this change additive).
• tests/test_settings.py — extend with UPSTREAM_AUTH_MODE/GITHUB_APP_USERNAME env-var round-trip cases, following the file's existing monkeypatch.setenv pattern.
• tests/test_integ.py — this file already runs real end-to-end clones against a live upstream, gated by requires_upstream_auth/CREDS/CI_JOB_TOKEN env vars (per doc/overview.md §13). Add an equivalent GitHub-targeted parametrization (a GITHUB_TOKEN env var, skipped unless set, same pattern as today's GitLab-only default) so the token-auth path is exercised against real GitHub, not just mocked — this is the only way to actually confirm the auth-failure stderr wording and Bearer-header behavior referenced as open questions in §10.
• Regression check: run the full existing suite unmodified with UPSTREAM_AUTH_MODE unset, to confirm the basic default path is provably untouched.

9. Implementation plan

1. Add UpstreamAuthMode/upstream_auth_mode/github_app_username to settings.py + test_settings.py cases. No behavior change yet (nothing reads the new setting).
2. Add git_cdn/auth.py (parse_auth_header, url_creds) + tests/test_auth.py. Still not wired into any request path yet — pure addition, fully unit-testable in isolation.
3. Wire handle_upload_pack's credential extraction (git_cdn.py:448-450) to branch on settings.upstream_auth_mode, calling into auth.py for the github case.
4. Wire the two dumb-proxy Authorization-forwarding sites (proxify_with_data, handle_upload_pack's pre-check) to re-encode to Basic in github mode (§5.3).
5. Update check_auth()'s WWW-Authenticate behavior per mode.
6. Confirm and encode GitHub's actual auth-failure stderr string in repo_cache.py:122-123 (needs a real 401 captured from GitHub first — see §10).
7. Add the GitHub-targeted integration test parametrization (§8) — this is what actually validates steps 3-6 against reality rather than assumptions.
8. Update README.md's "Technical features" section (currently: "Tested with Gitlab, but should work with any BasicAuth git+http(s) server") and add a UPSTREAM_AUTH_MODE/ GITHUB_APP_USERNAME row to its configuration table, and to doc/overview.md §10's superset table.
9. Optionally extend doc/deployment.md's local stack with a second, token-authenticated example (it currently only clones a public unauthenticated repo — see doc/deployment.md step 5) once a disposable test PAT/token convention for local dev is decided.

Steps 1-2 have no blast radius (pure additions) and can land independently of confirming the open questions below; steps 3+ depend on resolving §10.

10. Open questions / risks

These need answers from whoever owns the calling tooling / has a real GitHub token to test with, before finalizing the exact wire-level behavior — the design above is intentionally built so that answering them means adjusting auth.py's parsing branches and repo_cache.py's string match rather than re-architecting anything:

1. What scheme does the actual calling client send? This doc assumes Bearer/token are the candidates worth supporting (§3), but this should be confirmed against the specific tool/CI-system that will call git_cdn with a GitHub credential, rather than guessed. If it turns out the caller can just be configured to send Basic base64(x-access-token:<token>) directly, most of §5.2/§5.3 collapses to "no code change needed, just document the convention" — worth checking this first since it's by far the cheapest outcome.
2. GitHub App installation tokens vs. classic/fine-grained PATs vs. OAuth tokens — do all three need support, or just one? They differ only in the conventional username (§5.1's GITHUB_APP_USERNAME), so supporting all three is cheap, but worth confirming scope before building test coverage for all of them.
3. Exact GitHub auth-failure stderr wording, needed for repo_cache.py:122-123's HTTPUnauthorized conversion — capture a real failed git clone against GitHub with a bad/ expired token to get the literal string (git's own generic fatal: Authentication failed wrapping GitHub's specific message, most likely).
4. Does GitHub's git-http backend accept a raw Bearer/token header directly (making §5.3's re-encoding unnecessary for the pass-through paths), or does it require Basic specifically? This is directly testable with a single authenticated curl against https://github.com/<owner>/<repo>.git/info/refs?service=git-upload-pack using each header form, and should be done before implementing step 4 of §9.
5. Token expiry mid-request (§7) — low risk given git_cdn's existing retry/backoff already tolerates transient upstream failures, but not explicitly handled as a distinct case; flag if installation tokens' typical ~1 hour lifetime turns out to be short enough to matter in practice (e.g. for very large initial clones).