TCP stream corruption in uploads across Comcast's network

README.md

TCP stream corruption in uploads across Comcast's network

We've noticed that some % of upstream packets going to from our office wifi to Cloudflare are partially replaced by all 0-bytes. This happens above the network layer because the TCP checksum is correct.

We typically observe this as https uploads failing with "bad record mac", but we can reproduce the exact corruption pattern using a public echo server.

We are not the only people with this problem: Xfinity forum, March 2026 · Xfinity forum, July 2026 · Efpophis gist

Interestingly, the other Comcast customer in the same office as us does not have the problem (despite their router being in the same room as ours); and other upstream destinations (i.e. not Cloudflare) are not affected from our network.

The damage

What we observe through the echo-server is that the tail end of some IP packets are being replaced by all zeros:

• the first 230 bytes of the IP packet survive intact -- constant across IPv4 and IPv6 once the 20-byte header size difference is accounted for
• every byte after that arrives as 0x00
• the zeroed run ends precisely on a TCP segment boundary
• total stream length is preserved and adjacent segments are untouched
• the damaged packet arrives with a valid TCP checksum, so the receiving stack accepts it and the byte stream stays aligned

Measured payload sizes: 1198 bytes zeroed over IPv4, 1178 over IPv6. Adding headers back gives the same figure both ways: 20+20+1198 = 40+20+1178 = 1238.

Verified on both address families, each measured against the MSS of its own connection:

Family	MSS	Zeroed	IP+TCP headers	Packet bytes kept
IPv4	1388	1198	40	230
IPv6	1348	1178	60	230

Observed rate: roughly 1 damaged segment per 20-50 MB uploaded. Failure probability scales with bytes sent after about the first 50 KB of a connection -- 64 KiB uploads fail ~0.1% of the time, 1 MiB uploads 7-12%.

Latest measurement run

Affected circuit, public IP 73.243.42.115, Cloudflare colo DEN, over Wi-Fi:

./upload-test.sh https://cloudflare.com/             150   ->  5/150 failed
./upload-test.sh https://s3.us-west-2.amazonaws.com/ 150   ->  0/150 failed
./corruption-test.py cf 1500   -> 6 events / 140.6 MiB verified

All seven damaged segments across those six transfers were identical: 1178 bytes, entirely zero, contiguous, ending exactly on a segment boundary, 230 bytes of the IP packet kept. One transfer contained two independently damaged segments. Four of the six events landed within six consecutive iterations (1469, 1471, 1473, 1474), which illustrates the clustering described below.

Ruled out

Suspect	How it was eliminated
Client hardware	Same laptop clean on a second Comcast circuit in the same building: 0/400 vs ~10%
NIC offloads	TSO and checksum offload disabled, no change
MTU / fragmentation	Full 1500-byte MTU confirmed; clamping MSS to 1300 did not help
Gateway NAT	IPv6 is not translated and fails at the same rate
Gateway state	Power cycle, no change (rate measured before and after)
SecurityEdge	Disabled: rate roughly halved but corruption continued, byte-for-byte identical
Destination	Uploads to AWS, Google and Facebook clean across 350 MB interleaved with failing Cloudflare uploads
Line quality	Downloads clean over 320 MB; the working circuit has more than twice the first-hop jitter

Path

Failing and working traffic share every Comcast hop up to be-3211-pe11.910fifteenth.co.ibone.comcast.net, diverging only afterwards. The March report implicates the equivalent router in Chicago, be-2211-pe11.350ecermak.il.ibone.comcast.net -- same pe11 role on the same ibone backbone.

Usage

python3 make-payload.py plain 1048576 /tmp/payload1m.bin

# Fails on the affected circuit (expect ~7-12%)
./upload-test.sh https://cloudflare.com/ 150
./upload-test.sh https://cloud.zed.dev/  150
./upload-test.sh https://discord.com/    150

# Controls -- expect zero. Run alternately with the above.
./upload-test.sh https://s3.us-west-2.amazonaws.com/ 150
./upload-test.sh https://www.google.com/             150

# Both address families fail, ruling out NAT
./upload-test.sh https://cloudflare.com/ 100 -4
./upload-test.sh https://cloudflare.com/ 100 -6

# Show the actual corrupted bytes (plain HTTP, echoed back and compared)
./corruption-test.py mss
./corruption-test.py cf  1500    # ~1 event per 50 MB on the affected circuit
./corruption-test.py aws 1500    # control: expect zero

Two things to watch out for

Failures come in bursts. The longest clean run recorded on the faulty circuit was 67 consecutive 1 MiB uploads. A 20-request test passes routinely on a broken line. Run the full 150 and treat a single clean run as inconclusive.

Size matters. Use 1 MiB payloads for pass/fail work. corruption-test.py is limited to 96 KiB because that is the largest body postman-echo.com accepts, which is why it needs far more iterations to catch an event.

corruption-test.py

#!/usr/bin/env python3
"""Detect SILENT upload corruption by echoing a known payload back.

Uploads over plain HTTP (no TLS) to an endpoint that echoes the request body,
then compares byte for byte. TLS only ever reports "bad record mac"; this shows
the damaged bytes themselves.

  ./corruption-test.py cf   [iters]   # via Cloudflare  (postman-echo.com)
  ./corruption-test.py cf4  [iters]   # ... forced IPv4
  ./corruption-test.py cf6  [iters]   # ... forced IPv6
  ./corruption-test.py aws  [iters]   # control, no CDN (httpbin.org)
  ./corruption-test.py mss            # negotiated MSS per destination

The MSS is read from the *actual* upload connection, so the segment-alignment
figures are always computed against the right segment size.
"""
import sys, json, base64, socket, http.client

SIZE = 98304                       # postman-echo rejects bodies >~128 KiB
PAYLOAD = bytes(((i * 7 + (i >> 8)) % 251) + 1 for i in range(SIZE))
assert 0 not in PAYLOAD, "payload must contain no zero bytes"

ENDPOINTS = {                       # mode -> (host, path, decoder, family)
    "cf":   ("postman-echo.com", "/post", "postman", None),
    "cf4":  ("postman-echo.com", "/post", "postman", socket.AF_INET),
    "cf6":  ("postman-echo.com", "/post", "postman", socket.AF_INET6),
    "aws":  ("httpbin.org",      "/post", "httpbin", None),
    "aws4": ("httpbin.org",      "/post", "httpbin", socket.AF_INET),
}

def post(host, path, body, family=None, timeout=40):
    """POST body, returning (status, response, mss, address_family)."""
    c = http.client.HTTPConnection(host, 80, timeout=timeout)
    if family is not None:                       # pin the address family
        ai = socket.getaddrinfo(host, 80, family, socket.SOCK_STREAM)[0]
        s = socket.socket(family, socket.SOCK_STREAM)
        s.settimeout(timeout); s.connect(ai[4]); c.sock = s
    else:
        c.connect()
    try:
        mss = c.sock.getsockopt(socket.IPPROTO_TCP, socket.TCP_MAXSEG)
        fam = c.sock.family
        c.request("POST", path, body=body,
                  headers={"Host": host, "Content-Type": "application/octet-stream",
                           "Content-Length": str(len(body)),
                           "Accept-Encoding": "identity"})
        r = c.getresponse()
        return r.status, r.read(), mss, fam
    finally:
        c.close()

def decode(raw, kind):
    d = json.loads(raw, strict=False)
    if kind == "postman":
        n = d.get("data")
        return bytes(n["data"]) if isinstance(n, dict) and n.get("type") == "Buffer" else None
    s = d.get("data", "")
    pre = "data:application/octet-stream;base64,"
    return base64.b64decode(s[len(pre):]) if s.startswith(pre) else None

def mss_probe():
    tgts = [("cloudflare.com",443,socket.AF_INET), ("cloudflare.com",443,socket.AF_INET6),
            ("postman-echo.com",80,socket.AF_INET), ("postman-echo.com",80,socket.AF_INET6),
            ("s3.us-west-2.amazonaws.com",443,socket.AF_INET),
            ("httpbin.org",443,socket.AF_INET)]
    for host, port, fam in tgts:
        try:
            ai = socket.getaddrinfo(host, port, fam, socket.SOCK_STREAM)[0]
            s = socket.socket(fam, socket.SOCK_STREAM); s.settimeout(10); s.connect(ai[4])
            print(f"  {host:30} {'v4' if fam==socket.AF_INET else 'v6'}  MSS={s.getsockopt(socket.IPPROTO_TCP, socket.TCP_MAXSEG)}")
            s.close()
        except Exception as e:
            print(f"  {host:30} error: {e}")

def report(got, n, mss, fam):
    iphdr = 20 if fam == socket.AF_INET else 40
    diff = [i for i in range(SIZE) if PAYLOAD[i] != got[i]]
    runs, s, p = [], diff[0], diff[0]
    for x in diff[1:]:
        if x != p + 1: runs.append((s, p)); s = x
        p = x
    runs.append((s, p))
    print(f"  !! CORRUPTION on iteration {n}  "
          f"({'IPv4' if fam==socket.AF_INET else 'IPv6'}, MSS {mss}, "
          f"{len(runs)} damaged run(s), body length kept={len(got)==SIZE})")
    for a, b in runs:
        L = b - a + 1
        prefix = a - (a // mss) * mss
        print(f"     start {a:6d}  len {L:5d}  all_zero={set(got[a:b+1])=={0}}  "
              f"ends_on_segment_boundary={(a+L) % mss == 0}  "
              f"packet_bytes_kept={prefix + iphdr + 20}")
    open(f"corrupt_{n}.bin", "wb").write(got)

def main():
    mode = sys.argv[1] if len(sys.argv) > 1 else "cf"
    if mode == "mss":
        mss_probe(); return
    iters = int(sys.argv[2]) if len(sys.argv) > 2 else 1500
    host, path, kind, family = ENDPOINTS[mode]
    print(f"endpoint {host} ({'via Cloudflare' if mode.startswith('cf') else 'control, no CDN'}), "
          f"{iters} x {SIZE//1024} KiB")
    events = ok = 0
    for i in range(iters):
        try:
            st, raw, mss, fam = post(host, path, PAYLOAD, family)
            if st != 200: continue
            got = decode(raw, kind)
            if got is None: continue
            ok += len(got)
            if got != PAYLOAD:
                events += 1
                report(got, i, mss, fam)
        except Exception:
            pass
        if (i+1) % 300 == 0:
            print(f"  ...{i+1}/{iters}  events={events}  ({ok/1048576:.0f} MiB verified)", flush=True)
    print(f"\nRESULT: {events} corruption events over {ok/1048576:.1f} MiB verified")

main()

make-payload.py

#!/usr/bin/env python3
"""Generate test payloads.

  plain <size> <file>   position-encoded counter (fine for pass/fail testing)
  nozero <size> <file>  every byte in 1..251, never 0x00 -- required for the
                        corruption test, so that any zero byte received is
                        unambiguously damage rather than original data.
"""
import sys
mode, size, path = sys.argv[1], int(sys.argv[2]), sys.argv[3]
with open(path, "wb") as f:
    if mode == "plain":
        for i in range(0, size, 8):
            f.write(i.to_bytes(8, "big"))
    elif mode == "nozero":
        f.write(bytes(((i * 7 + (i >> 8)) % 251) + 1 for i in range(size)))
    else:
        sys.exit("mode must be 'plain' or 'nozero'")
print(f"wrote {path} ({size} bytes, {mode})")

upload-test.sh

#!/bin/bash
# Pass/fail upload test.  usage: ./upload-test.sh <url> [count] [curl-args...]
URL="${1:?usage: ./upload-test.sh <url> [count] [curl args...]}"
N="${2:-150}"; shift 2 2>/dev/null || shift 1
PAY=/tmp/payload1m.bin
[ -f "$PAY" ] || python3 "$(dirname "$0")/make-payload.py" plain 1048576 "$PAY" >/dev/null

echo "target    : $URL"
echo "public IP : $(curl -4 -sS --max-time 10 https://cloudflare.com/cdn-cgi/trace 2>/dev/null | awk -F= '/^ip=/{print $2}')"
echo "uploading $N x 1 MiB ..."
fails=0
for i in $(seq 1 "$N"); do
  err=$(curl -sS --max-time 40 -o /dev/null --data-binary @"$PAY" "$@" "$URL" 2>&1)
  if [ -n "$err" ]; then
    fails=$((fails+1))
    [ "$fails" -le 3 ] && echo "  request $i: $(echo "$err" | cut -c1-96)"
  fi
done
echo "RESULT: $fails/$N failed"