APFS copy-on-write sharing for Git worktrees: implementation guide for coding agents, reference helper, and tests

README.md

Save disk space across Git worktrees with APFS copy-on-write clones

Git worktrees share the Git object database, but large checked-out files can still occupy separate disk blocks. On macOS/APFS, identical files can share their data blocks while remaining independently editable files.

In one dependency-heavy iOS repository, a post-checkout sharing pass shared about 761–766 MB of logical file contents per worktree, across roughly 217–249 files. An initial real-worktree experiment measured a 760.14 MB decrease in APFS private data. These are workload-specific results, not a promise for every repository. All MB figures here are decimal.

The core idea

1. Let Git create and check out the worktree normally. Finish dependency and project generation.
2. Enumerate tracked regular files in the destination. This helper considers files at least 64 KiB and conservatively excludes Swift files, a policy inherited from its original repository.
3. Compare each file with the same relative path in the existing checkout using SHA-256. Different branches are fine: different contents are left alone.
4. Use native clonefile(2) to clone matching source data into a temporary directory beside the destination.
5. Preserve and verify the destination's metadata, and verify the staged file's contents both before and after copying metadata.
6. Check that the destination has not changed, then atomically replace it with the clone.

APFS allocates private blocks when either file is edited. Deleting the source does not invalidate the clone. Each file has its own inode. Hard links do not provide this independent-edit behavior.

Implementation brief for your coding agent

Use a single agent to adapt this technique to the repository you are working in. Read its worktree creation scripts and repository instructions first. Keep the normal Git registration, checkout, dependency setup, and cleanup flow. Add one synchronous sharing pass after setup and before handing the worktree to a user, editor, or build. An ordinary implementation task is sufficient; no persistent goal loop or multi-agent orchestration is needed.

The attached Python helper is reference code, not a requirement to replace your tooling. Preserve its guarded per-file operation and adapt only the integration and repository policy. In particular:

• Start with tracked regular files at least 64 KiB. The .swift exclusion belongs to the original repository, not APFS; choose exclusions appropriate to your repository. Never traverse .git, ignored caches, or symlinked paths.
• Compare actual bytes, including materialized Git LFS contents. Do not assume matching branch names, object IDs, sizes, or clean Git status prove identical on-disk contents.
• Keep sharing optional with an explicit opt-out. Unsupported platforms, filesystems, and metadata cases should keep ordinary copies and report why sharing was skipped.
• Put reports somewhere ignored or outside the checkout. Report cloned logical bytes separately from measured allocation savings. Preserve the complete before/after Git status, including pre-existing changes.
• Keep the destination exclusively owned throughout the pass. Do not run this automatically against worktrees already in use. A last-moment stat check cannot make replacement safe against concurrent writers.
• Implement the verification checklist below before enabling sharing by default. Follow the repository's normal approval process for that behavior change.

Three technical parts to preserve

1. Native cloning with an explicit failure path. clonefile(source, staged, 0) creates an independent file backed by shared data blocks. The staged path must not exist and must be on the same filesystem as the source. In Python, the helper calls libc with ctypes and captures errno. A hard link is not a substitute. /bin/cp -c can fall back to a full copy, so command success alone does not prove sharing.

2. Verified publication. The important ordering is:

reject unsupported paths and metadata
record destination identity and hash
require source hash == destination hash
clone source into destination-local temporary staging
require staged hash == original destination hash
copy destination xattrs, mode, timestamps, and flags to staged clone
verify preserved destination metadata
require staged hash == original destination hash AGAIN
require destination identity unchanged
atomically replace destination with staged clone

If any pre-publication step fails, keep the destination and remove staging. Preserve cancellation cleanup. Do not delete the original first, and do not copy file data back over the staged clone when restoring metadata. The helper uses copyfile(..., COPYFILE_XATTR) for metadata-only xattr copying.

3. Sharing-aware measurement. The helper's sharing() function uses getattrlist(2) with extended file attributes: ATTR_CMNEXT_PRIVATESIZE, ATTR_CMNEXT_CLONEID, and ATTR_CMNEXT_CLONE_REFCNT. Verify attribute masks and packed layouts against the installed SDK before adapting this low-level code. Matching nonzero clone IDs establish fully shared data streams; ordinary copies provide a useful negative control. logical_bytes.cloned alone does not establish the number of bytes reclaimed.

Acceptance checklist

• Identical files acquire matching nonzero clone IDs but retain different inodes; different contents remain untouched.
• Editing either clone leaves the other unchanged, and deleting the source leaves the destination readable.
• Destination metadata survives. A staged-content change during metadata copying is rejected by the final hash.
• Symlinks, ACLs, hardlinked destinations, unsupported filesystems, and cross-volume pairs are skipped safely.
• Inject clone failure and ENOSPC; interrupt during staging with SIGTERM. Originals must survive and staging must be cleaned.
• A repeat pass recognizes already-shared files. Git status and LFS hashes remain correct; branch switching and normal worktree cleanup still work.
• Run the normal build or another meaningful repository integration check after a real worktree creation. Distinguish fresh results from the historical measurements below.

Usage

Requires macOS, an APFS volume shared by both checkouts, Git, and Python 3.11+ (hashlib.file_digest). The original integration was tested with Python 3.13.11; support across macOS versions has not been established.

Save worktree_file_sharing.py outside the destination, then run after your normal worktree creation/setup command:

python3 worktree_file_sharing.py \
  --source /path/to/existing-checkout \
  --repo /path/to/new-worktree

This command applies changes. Use it only while your setup process exclusively owns the new worktree: no editors, builds, Git operations, or other writers there until it finishes. Keep the source quiescent too. The stat checks are safeguards, not concurrency locks or protection against hostile filesystem races.

The helper prints a JSON summary and writes build/worktree-sharing.json inside the destination. Arrange for that report path to be ignored in your repository; otherwise the report itself will appear as an untracked file. The reported logical_bytes.cloned is the payload cloned, not a measurement of newly reclaimed physical storage. Repeating the pass recognizes already-shared files.

The script is a standalone extraction of an existing helper, retaining its conservative behavior. It skips symlinks, hardlinked destinations, different owners, ACLs, cross-volume pairs, and differing contents. Unsupported clone/attribute operations retain the ordinary files and report errors. Review the JSON: exit success can include skipped files. It does not copy Git metadata, ignored build caches, or an entire checkout.

SIGINT/SIGTERM unwind staging. SIGKILL can leave .worktree-sharing-* staging directories. This pass reduces retained space; checkout still initially needs the ordinary disk allocation. Later writes need free space for private blocks.

Why the metadata checks matter

Identical bytes can have different extended attributes. In the original experiment, requiring identical xattrs shared only 203.8 MB. Copying the destination's xattrs onto the staged clone unlocked another 555.9 MB while preserving destination metadata.

The helper uses metadata-only copyfile(3) for xattrs and verifies mode, owner/group, modification time, flags, and xattrs before replacement. It does not promise identical inode numbers, creation times, or ctimes. ACL cases are deliberately skipped. A final content hash matters because some metadata can affect file-data representation.

Measuring the benefit

Do not add up du or st_blocks and call that physical usage: shared blocks can be counted more than once. Use APFS clone IDs to verify sharing and private-size attributes or a controlled volume free-space experiment to quantify allocation. Snapshots and delayed reclamation affect when space becomes free.

An isolated APFS-volume experiment used the same 815.51 MB corpus for each strategy:

Strategy	Additional volume allocation
Explicit byte-by-byte copy	846.54 MB
Per-file clones via /bin/cp -cR	3.73 MB
Bulk native clone	3.74 MB

These are historical corpus measurements, not complete-worktree benchmarks. The integrated verification pass added about 12–20 seconds of setup time. Bulk cloning did not materially improve disk savings, so the integration kept normal Git checkout and used per-file replacement. Current local clonefile(2) documentation also discourages directory cloning.

The original integration verified independent edits, source deletion, branch switching, Git LFS hashes, unsupported-filesystem/cross-volume fallback, cancellation, and an app build. Real disk exhaustion was not tested (ENOSPC was injected), and a second Mac/oldest supported OS remains unverified. The accompanying tests cover the core file operation, not that full integration.

Run the extracted tests

python3 -m unittest -v test_worktree_file_sharing

Run on an APFS Mac. Tests create temporary files, use actual native clones, and inject failures to check that originals survive. They do not alter an existing checkout.

Native API references

Consult the manuals shipped with your Mac: man 2 clonefile, man 3 copyfile, man 2 getattrlist, and man 1 cp. The helper's APFS attribute constants/layout come from the macOS SDK's sys/attr.h; xattr copying uses COPYFILE_XATTR from copyfile.h.

The reusable technique is: normal checkout, content comparison, staged native clone, destination metadata preservation, verification, atomic replacement.

test_worktree_file_sharing.py

"""Intent: Exercise the lifecycle's file-sharing core against real APFS.

Boundary: macOS-only temporary files; never edits an existing user checkout.
These tests prove the prototype's narrow contract, not concurrent-writer safety.
"""

from pathlib import Path
import errno
import os
import subprocess
import sys
import tempfile
import time
import unittest
from unittest.mock import patch

import worktree_file_sharing as spike


@unittest.skipUnless(sys.platform == "darwin", "APFS spike requires macOS")
class APFSSpikeTests(unittest.TestCase):
    def setUp(self):
        self.temporary = tempfile.TemporaryDirectory()
        self.addCleanup(self.temporary.cleanup)
        self.root = Path(self.temporary.name).resolve()
        self.source = self.root / "source"
        self.target = self.root / "target"
        self.contents = os.urandom(128 * 1024)
        self.source.write_bytes(self.contents)
        self.target.write_bytes(self.contents)

    def apply(self):
        return spike.deduplicate_file(self.source, self.target, apply=True)

    def test_clone_shares_data_preserves_mode_and_timestamp_and_is_independent(self):
        self.target.chmod(0o755)
        os.utime(self.target, ns=(1234567890000000000, 1234567890000000000))
        previous = self.target.stat()
        self.assertEqual(self.apply(), "cloned")
        self.assertEqual(spike.sharing(self.source)["clone_id"], spike.sharing(self.target)["clone_id"])
        self.assertNotEqual(self.source.stat().st_ino, self.target.stat().st_ino)
        self.assertEqual(previous.st_mode, self.target.stat().st_mode)
        self.assertEqual(previous.st_mtime_ns, self.target.stat().st_mtime_ns)
        self.assertEqual(self.apply(), "already_shared")
        with self.target.open("r+b") as stream:
            stream.write(b"changed")
        self.assertEqual(self.source.read_bytes(), self.contents)
        self.assertNotEqual(self.target.read_bytes(), self.contents)

    def test_different_content_is_left_intact(self):
        changed = b"x" * len(self.contents)
        self.target.write_bytes(changed)
        self.assertEqual(self.apply(), "different_content")
        self.assertEqual(self.target.read_bytes(), changed)

    def test_dry_run_leaves_inode_intact(self):
        previous = spike.identity(self.target)
        self.assertEqual(spike.deduplicate_file(self.source, self.target, apply=False), "eligible")
        self.assertEqual(previous, spike.identity(self.target))

    def test_symlink_and_hardlink_are_skipped(self):
        self.target.unlink()
        self.target.symlink_to(self.source)
        self.assertEqual(self.apply(), "symlink")
        self.target.unlink()
        self.target.write_bytes(self.contents)
        os.link(self.target, self.root / "alias")
        self.assertEqual(self.apply(), "hardlinked_destination")

    def test_xattrs_are_preserved_or_differences_skipped(self):
        spike.run("/usr/bin/xattr", "-w", "com.keepsafe.spike", "original", str(self.target))
        self.assertEqual(self.apply(), "different_xattrs")
        spike.run("/usr/bin/xattr", "-w", "com.keepsafe.spike", "original", str(self.source))
        self.assertEqual(self.apply(), "cloned")
        self.assertEqual(spike.run("/usr/bin/xattr", "-p", "com.keepsafe.spike", str(self.target)).strip(), b"original")

    def test_acl_is_skipped(self):
        spike.run("/bin/chmod", "+a", "everyone allow read", str(self.target))
        self.assertEqual(self.apply(), "acl")

    def test_opt_in_xattr_copy_preserves_destination_and_removes_source_only_attrs(self):
        spike.run("/usr/bin/xattr", "-w", "com.keepsafe.source-only", "source", str(self.source))
        spike.run("/usr/bin/xattr", "-w", "com.keepsafe.target-only", "target", str(self.target))
        expected = spike.run("/usr/bin/xattr", "-lx", str(self.target))
        self.assertEqual(spike.deduplicate_file(self.source, self.target, apply=True, copy_xattrs=True), "cloned")
        self.assertEqual(spike.run("/usr/bin/xattr", "-lx", str(self.target)), expected)

    def test_failed_clone_keeps_original_and_cleans_staging(self):
        with patch.object(spike, "clone", side_effect=OSError("injected failure")):
            with self.assertRaises(OSError):
                self.apply()
        self.assertEqual(self.target.read_bytes(), self.contents)
        self.assertFalse(list(self.root.glob(".worktree-sharing-*")))

    def test_metadata_copy_cannot_publish_changed_file_data(self):
        real_copystat = spike.shutil.copystat

        def corrupt(source, target):
            target.write_bytes(b"metadata operation changed file data")
            real_copystat(source, target)

        with patch.object(spike.shutil, "copystat", side_effect=corrupt):
            with self.assertRaisesRegex(RuntimeError, "Metadata copying changed file data"):
                self.apply()
        self.assertEqual(self.target.read_bytes(), self.contents)

    def test_destination_change_during_clone_is_not_overwritten(self):
        real_clone = spike.clone

        def mutate(source, target):
            real_clone(source, target)
            self.target.write_bytes(b"new user edit")

        with patch.object(spike, "clone", side_effect=mutate):
            with self.assertRaisesRegex(RuntimeError, "Destination changed"):
                self.apply()
        self.assertEqual(self.target.read_bytes(), b"new user edit")

    def test_source_change_during_clone_is_not_published(self):
        real_clone = spike.clone

        def mutate(source, target):
            source.write_bytes(b"source changed")
            real_clone(source, target)

        with patch.object(spike, "clone", side_effect=mutate):
            with self.assertRaisesRegex(RuntimeError, "Source changed"):
                self.apply()
        self.assertEqual(self.target.read_bytes(), self.contents)

    def test_source_deletion_does_not_break_clone(self):
        self.assertEqual(self.apply(), "cloned")
        self.source.unlink()
        self.assertEqual(self.target.read_bytes(), self.contents)

    def test_sigterm_cleans_staging_and_preserves_original(self):
        marker = self.root / "ready"
        repo = self.root / "repo"
        repo.mkdir()
        target = repo / "source"
        target.write_bytes(self.contents)
        spike.run("git", "init", "-q", str(repo))
        spike.run("git", "add", "source", cwd=repo)
        script = """
import sys, time
from pathlib import Path
import worktree_file_sharing as sharing
real_clone = sharing.clone
marker = Path(sys.argv[3])
def slow_clone(source, target):
    real_clone(source, target)
    marker.touch()
    time.sleep(30)
sharing.clone = slow_clone
sys.argv = ["sharing", "--repo", sys.argv[1], "--source", sys.argv[2]]
raise SystemExit(sharing.main())
"""
        process = subprocess.Popen([sys.executable, "-c", script, str(repo), str(self.root), str(marker)])
        try:
            deadline = time.monotonic() + 10
            while not marker.exists() and process.poll() is None and time.monotonic() < deadline:
                time.sleep(0.02)
            self.assertTrue(marker.exists())
            process.terminate()
            self.assertEqual(process.wait(timeout=10), 130)
        finally:
            if process.poll() is None:
                process.kill()
                process.wait()
        self.assertEqual(target.read_bytes(), self.contents)
        self.assertFalse(list(repo.glob(".worktree-sharing-*")))

    def test_sharing_pass_keeps_files_on_unsupported_clone_and_low_disk(self):
        destination = self.root / "repo"
        destination.mkdir()
        (destination / "asset").write_bytes(self.contents)
        (self.root / "asset").write_bytes(self.contents)
        spike.run("git", "init", "-q", str(destination))
        spike.run("git", "add", "asset", cwd=destination)
        for code in (errno.ENOTSUP, errno.ENOSPC):
            with self.subTest(errno=code), patch.object(spike, "clone", side_effect=OSError(code, os.strerror(code))):
                result = spike.share_files(destination, self.root)
                self.assertEqual(result["counts"], {"kept_after_error": 1})
                self.assertTrue(result["git_status_unchanged"])
                self.assertEqual((destination / "asset").read_bytes(), self.contents)
                self.assertFalse(list(destination.glob(".worktree-sharing-*")))

    def test_unsupported_platform_skips_without_touching_files(self):
        with patch.object(spike.sys, "platform", "linux"):
            result = spike.share_files(self.root, self.root)
        self.assertIn("requires macOS", result["skipped"])
        self.assertEqual(self.target.read_bytes(), self.contents)


if __name__ == "__main__":
    unittest.main()

worktree_file_sharing.py

#!/usr/bin/env python3
"""Share identical large tracked files after a new worktree finishes generation.

Intent: Reduce retained disk usage through independent APFS file clones.
Boundary: The lifecycle caller exclusively owns the new destination until this
pass completes. No Git metadata, ignored files, Swift files, or differing content
is copied. Staging is destination-local and removed on failure or cancellation.
"""

import argparse
import ctypes
import hashlib
import json
import os
from pathlib import Path
import re
import shutil
import signal
import struct
import subprocess
import sys
import tempfile
import time


LIBC = ctypes.CDLL(None, use_errno=True)
ATTR_PRIVATE = 0x8
ATTR_CLONE_ID = 0x100
ATTR_REFCOUNT = 0x1000


def run(*args, cwd=None):
    return subprocess.check_output(args, cwd=cwd)


def digest(path):
    with path.open("rb") as stream:
        return hashlib.file_digest(stream, "sha256").hexdigest()


def clone(source, target):
    if LIBC.clonefile(os.fsencode(source), os.fsencode(target), 0) != 0:
        error = ctypes.get_errno()
        raise OSError(error, os.strerror(error), str(target))


def sharing(path):
    # Values/layout are from the installed macOS SDK sys/attr.h and getattrlist(2).
    mask = ATTR_PRIVATE | ATTR_CLONE_ID | ATTR_REFCOUNT
    attrs = ctypes.create_string_buffer(struct.pack("=HHIIIII", 5, 0, 0x80000000, 0, 0, 0, mask))
    buf = ctypes.create_string_buffer(128)
    if LIBC.getattrlist(os.fsencode(path), attrs, buf, len(buf), 0x20) != 0:
        error = ctypes.get_errno()
        raise OSError(error, os.strerror(error), str(path))
    values = struct.unpack_from("=I5IQQI", buf.raw)
    if values[0] != 44 or values[5] & mask != mask:
        raise RuntimeError(f"APFS sharing attributes unavailable: {path}")
    return dict(private_bytes=values[6], clone_id=values[7], references=values[8])


def tracked(repo):
    return [Path(os.fsdecode(p)) for p in run("git", "ls-files", "-z", cwd=repo).split(b"\0") if p]


def identity(path):
    info = path.lstat()
    return info.st_dev, info.st_ino, info.st_size, info.st_mtime_ns, info.st_ctime_ns


def has_acl(path):
    # Do not attempt to reproduce APFS ACL inheritance in this bounded spike.
    return re.search(rb"\n\s*\d+:", run("/bin/ls", "-lde", str(path))) is not None


def deduplicate_file(source, target, *, apply, copy_xattrs=False):
    # Reject symlinks in all components, not only the leaf.
    if any(p.is_symlink() for p in [source, *source.parents, target, *target.parents]):
        return "symlink"
    if not source.is_file() or not target.is_file():
        return "missing"
    if source.stat().st_dev != target.stat().st_dev:
        return "different_volume"
    if source.samefile(target):
        return "same_file"
    if target.stat().st_nlink != 1:
        return "hardlinked_destination"
    if source.stat().st_uid != target.stat().st_uid or source.stat().st_gid != target.stat().st_gid:
        return "different_owner"
    initial = identity(target)
    if source.stat().st_size != target.stat().st_size:
        return "different_content"
    expected = digest(target)
    if digest(source) != expected:
        return "different_content"
    source_clone_id = sharing(source)["clone_id"]
    if source_clone_id and source_clone_id == sharing(target)["clone_id"]:
        return "already_shared"
    if any(has_acl(path) for path in (source, target, target.parent)):
        return "acl"
    target_xattrs = run("/usr/bin/xattr", "-lx", str(target))
    if not copy_xattrs and run("/usr/bin/xattr", "-lx", str(source)) != target_xattrs:
        return "different_xattrs"
    if not apply:
        return "eligible"
    # Destination-local staging supports atomic replacement and leaves the original
    # intact on a failed clone/hash/metadata check. Never use a hard link.
    with tempfile.TemporaryDirectory(prefix=".worktree-sharing-", dir=target.parent) as temporary:
        staged = Path(temporary) / "clone"
        clone(source, staged)
        if digest(staged) != expected:
            raise RuntimeError("Source changed during cloning")
        if copy_xattrs:
            # COPYFILE_XATTR (copyfile.h): replace only extended attributes,
            # from the original destination; never copy its file data.
            if LIBC.copyfile(os.fsencode(target), os.fsencode(staged), None, 1 << 2) != 0:
                error = ctypes.get_errno()
                raise OSError(error, os.strerror(error), str(staged))
        shutil.copystat(target, staged)
        # Both modes require exact destination attributes before publication.
        if run("/usr/bin/xattr", "-lx", str(staged)) != target_xattrs:
            raise RuntimeError("Clone changed extended attributes")
        old_stat, new_stat = target.stat(), staged.stat()
        if (old_stat.st_uid, old_stat.st_gid, old_stat.st_mode, old_stat.st_mtime_ns, old_stat.st_flags) != (
                new_stat.st_uid, new_stat.st_gid, new_stat.st_mode, new_stat.st_mtime_ns, new_stat.st_flags):
            raise RuntimeError("Clone changed destination metadata")
        # Some extended attributes affect file-data representation. Verify after
        # all metadata operations as well as immediately after cloning.
        if digest(staged) != expected:
            raise RuntimeError("Metadata copying changed file data")
        if identity(target) != initial:
            raise RuntimeError("Destination changed during cloning")
        os.replace(staged, target)
    # Verification precedes publication. The source can subsequently change or
    # disappear without affecting this independent clone; do not inspect it again.
    return "cloned"


def share_files(repo, source):
    result = {"counts": {}, "logical_bytes": {}, "errors": [], "source": str(source)}
    if sys.platform != "darwin":
        result["skipped"] = "APFS sharing requires macOS"
        return result
    if repo.stat().st_dev != source.stat().st_dev:
        result["skipped"] = "Source and destination are on different volumes"
        return result
    before_status = run("git", "status", "--porcelain=v1", "-z", cwd=repo)
    started = time.monotonic()
    for relative in tracked(repo):
        target = repo / relative
        if target.suffix == ".swift" or target.is_symlink() or not target.is_file():
            continue
        size = target.stat().st_size
        if size < 65536:
            continue
        try:
            outcome = deduplicate_file(source / relative, target, apply=True, copy_xattrs=True)
        except (OSError, RuntimeError, subprocess.CalledProcessError) as error:
            # A failed clone or metadata check leaves the original in place.
            outcome = "kept_after_error"
            if len(result["errors"]) < 10:
                result["errors"].append({"path": str(relative), "error": str(error)})
        result["counts"][outcome] = result["counts"].get(outcome, 0) + 1
        result["logical_bytes"][outcome] = result["logical_bytes"].get(outcome, 0) + size
    after_status = run("git", "status", "--porcelain=v1", "-z", cwd=repo)
    if before_status != after_status:
        raise RuntimeError("Worktree changed during file sharing; inspect it before use")
    result.update(seconds=time.monotonic() - started, git_status_unchanged=True)
    return result


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--repo", type=Path, required=True)
    parser.add_argument("--source", type=Path, required=True)
    args = parser.parse_args()

    def interrupted(_signum, _frame):
        raise KeyboardInterrupt

    signal.signal(signal.SIGTERM, interrupted)
    try:
        result = share_files(args.repo.resolve(), args.source.resolve())
        report = args.repo / "build" / "worktree-sharing.json"
        report.parent.mkdir(parents=True, exist_ok=True)
        report.write_text(json.dumps(result, indent=2) + "\n")
        print("Worktree file sharing: " + json.dumps(result, sort_keys=True))
        return 0
    except KeyboardInterrupt:
        print("File sharing interrupted; original files and completed clones remain usable.", file=sys.stderr)
        return 130


if __name__ == "__main__":
    raise SystemExit(main())