You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Timestamps are approximate offsets into the recording, not conference clock
times. The linked start times below open the accompanying video. Release
status, proposals, and product capabilities are reported as presented at the
conference. Benchmarks are the presenters' measurements, not independently
reproduced results; unresolved questions remain explicitly unresolved.
Overview
Review is increasingly a coordination problem. Generating more code
does not automatically produce coherent changes or shared understanding.
Scaling requires doing less unnecessary work. Shorter graph walks,
fewer packfiles, batched fetches, and appropriately scoped history can
matter more than simply adding resources.
Git is becoming more extensible from within. Incremental Rust adoption
and storage interfaces aim to preserve a shared implementation rather than
force every large user to maintain a separate fork.
The data model determines which workflows are possible. Portable
metadata, reversible repository projections, distributed rewrite history,
and replaceable forge components offer different ways to improve
collaboration.
SHA-256 adoption is an ecosystem migration. Rehashing objects is only
part of the work; tools, dependencies, signatures, and historical references
also need a transition path.
Speaker: Tomas Reimers (Graphite/Cursor).
Recording: 00:33:30-01:12:00, including Q&A.
Easier changes, harder collaboration
Reimers argued that AI was making individual code changes easier without
making collaborative software development comparably easy. His focus was
chiefly closed-source teams where an author gets a change reviewed and merged,
rather than the full range of open-source contribution practices.
(00:34:27-00:35:57)
His historical account ran from Michael Fagan's 1976 work on inspections at
IBM, through emailed patches with inline comments and approvals, to Guido van
Rossum's Mondrian web interface at Google and subsequent forge interfaces.
The continuity mattered: successive tools improved the medium while
preserving many underlying review habits, rather than completely reinventing
the activity. (00:35:57-00:40:35)
Three different jobs for review
Reimers separated review into three purposes:
Validation: does the change do what its author claims?
Change management: is the work organized into coherent, individually
revertible units?
Alignment and shared understanding: can someone other than the author
understand and maintain the result?
AI could help with validation without automatically providing the other two.
Change management did not mean making every diff as small as possible, and
shared understanding was not equivalent to an automated approval.
(00:41:36-00:44:41)
A slide showed roughly 25% of pull requests receiving bot comments in sampled
repositories; it did not measure the proportion of comments written by bots.
Reimers also reported seeing simultaneously open pull requests rise from
roughly four to 20 or more, alongside much faster rewriting and feedback
cycles. These were observations from the workloads he discussed, not universal
industry statistics. (00:41:55; 00:45:45-00:48:21)
Agents, human attention, and evidence of behavior
Reimers proposed connecting agents directly to the forge rather than making
people copy review comments into a coding tool. This required interfaces,
context, and access appropriate to the work. It also left unresolved questions
about identity and authorization: whether an agent acts as the author, on the
author's behalf, or as a coauthor is not merely a presentation detail.
(00:48:51-00:50:23)
His second proposal was to direct scarce human attention both across the
pull-request inbox and within an individual change. Reviewers should be shown
architectural decisions, relevant context, and actions requiring judgment,
while routine follow-ups could be automated. (00:50:23-00:51:55)
Third, review should take outputs as seriously as source inputs. Screenshots,
videos, terminal output, runnable previews, and acceptance testing can expose
behavior that a diff alone does not communicate. The appropriate evidence
still depends on the software: standards suitable for a disposable marketing
site should not be applied indiscriminately to performance- or
security-sensitive core Git. He briefly identified Origin as Graphite's new
forge work at Cursor. (00:51:55-00:53:28)
Q&A: semantic changes and the limits of validation
Asked about change management, Reimers clarified that he meant the review
workflow, not necessarily its interface. A million-line mechanical rename
could constitute one semantic change; a large bundle of unrelated edits would
not become coherent merely by sharing a branch. (00:54:58-00:58:35)
Upfront agreement on a design or prompt did not replace inspecting the
realized result. Mocks and prompts could omit error states, transitions, or
assumptions about queries and data design. Reviewing what was actually built
also helped establish the shared understanding required for later
maintenance. (00:54:58-00:58:35)
An audience member challenged his optimism about validation: agents might
find remote edge cases while missing a better data structure or good API
design. Reimers explicitly conceded that validation was not solved. He
saw a path forward through AI and continuous integration, and distinguished
whether an implementation works as claimed from whether it is the right
thing to build. Architectural judgment remained a point of disagreement, not
an established automated capability. (01:04:44-01:06:47)
Q&A: shared infrastructure and unresolved maintenance
Reimers wanted review information and metadata to move alongside code through
a shared protocol, rather than remain in forge-specific silos. His provisional
answer was to use the forge's existing storage; he did not claim that the
desired standard already existed. (00:59:06-01:00:38)
He favored stacking as a way to keep coherent changes separately reviewable.
Long-lived feature branches that bundle work can reflect tooling compromises
rather than an ideal review unit. Different team risk profiles also produce
different review cultures; stacking and monorepo support remained part of his
tooling ambitions, not a prescription for one universal workflow.
(01:01:08-01:04:44)
Executable environments mattered for both agents developing and testing
changes and humans reviewing them. One example automatically ran an
application before and after a user-facing change, capturing videos or
screenshots. Equivalent support would be needed beyond websites, including
terminal, mobile, desktop, and remote-desktop workflows.
(01:06:47-01:09:52)
Reviewing the accumulated architecture of millions of lines, rather than just
the next incremental diff, remained an open cultural and maintenance problem.
Reimers treated it as distinct from improving an ordinary change-review
interface. Security scanning was mentioned as a possible starting point, not
a solved general approach. (01:09:52-01:11:55)
Rust in Git: Status, Internals, and Features
Speaker: brian m. carlson (GitHub).
Recording: 01:14:30-01:42:49, including Q&A.
Incremental adoption and explicit dependencies
The direction was incremental Rust adoption, not a wholesale rewrite.
carlson emphasized memory- and thread-safety checks, easier construction of
correct code, C interoperability, contributor participation, and consistent
iterators, maps, and sets. Initialized state and fewer mutable globals also
supported the longer-term goals of making Git usable as a library and
handling multiple repositories. (01:14:30-01:17:53)
A concrete C example concerned object-identifier comparison: an algorithm
field set to zero could make the operation depend implicitly on
repository-global hash state. The Rust approach made initialization and
dependencies explicit instead. The motivation was to make such requirements
easier to reason about, not to promise that Rust code could contain no bugs.
(01:14:30-01:17:53)
In Q&A, carlson described an aspiration for most main code to become Rust over
a long period. Some C platform and compatibility glue, and likely SHA1DC,
would remain. (01:36:30-01:38:37)
Portability and a sustainable compiler policy
Portability was the principal tradeoff. carlson identified concerns involving
Linux ia64 and alpha, HPE NonStop, and some BSD targets, reflecting LLVM's
narrower coverage compared with the C/GCC ecosystem. Rust's six-week release
cadence should not become a requirement for Git users to upgrade their
compiler every six weeks. (01:17:53-01:21:34)
Two alternative compiler efforts were under investigation. gccrs was
targeting Rust 1.49, which Git's Rust code was also targeting to accommodate
it. This was not proposed as a permanent minimum. mrustc, a C++ Rust
compiler, still needed static-library support, Cargo adaptation, and platform
patches for libcore and libstd. Distributions might need additional
patches of their own. These were possible routes, not completed portability
solutions. (01:18:58-01:21:03)
The proposed compiler policy followed the Rust version in Debian stable,
with another year of support for oldstable, giving roughly a three-year
useful window. That policy also required caution when choosing crates that
demanded the newest Rust release. (01:20:31-01:21:34)
Rollout and code already upstream
The adoption roadmap presented at the talk distinguished existing defaults
from the planned Git 3.0 requirement. (01:21:55)
Version or milestone
Rust policy described
Git 2.52
Meson autodetects Rust; Rust is disabled by default in Makefile builds.
Git 2.55
Enabled by default, with an opt-out. Without a compiler, the build fails unless Rust is disabled.
Git 3.0
Rust planned to become mandatory.
Last pre-3.0 version
Planned long-term support for systems without Rust.
Code already upstream included Patrick Steinhardt's varint parser, a
hash-algorithm abstraction, a C-compatible object-identifier structure, and
support for checksum-file writing, hashing, and fsync.
(01:22:07-01:23:08)
SHA interoperability and the C/Rust boundary
carlson separately described code already written for SHA interoperability;
this work was not necessarily all merged. Its security motivation was
concrete: pointer-heavy conversion code would become exposed to untrusted
fetch and pull protocol data. That made the conversion boundary a particularly
valuable place to use Rust. (01:23:08-01:27:41)
Several mechanisms supported this work:
A shared error representation bridged C and Rust's Result type. Its C
form occupied 128 bits, arranged as two 64-bit words, with an error code,
flags, and a pointer or integer payload. An error code of zero meant
success; flags tracked matters such as allocation ownership. Localized
formatting remained in C to preserve the gettext path. (01:23:55)
Buffer and output support covered strbuf, retaining C-style NUL
termination, alongside Vec<u8>, the Write trait, and native Rust
formatting. (01:25:00)
Object maps related identifiers across hash algorithms for loose objects,
submodules, and shallow points. A trait abstraction allowed in-memory unit
tests. (01:25:40-01:26:41)
Parsing and conversion covered trees, header-based commit and tag parsing,
signatures, hexadecimal object identifiers, and Display serialization.
Processing was zero-copy where possible, with multiline signatures an
exception. (01:26:11-01:27:41)
The boundary was still bidirectional: existing C and Rust static libraries
called one another. Consequently, this was not yet a collection of cleanly
separated, standalone reusable Rust crates. (01:36:30-01:38:37)
Future work and contributor questions
carlson had started base85 binary-diff work shortly before the talk. Planned
work included index and pack parsing and serialization, a hash-algorithm
migration tool, and more efficient or incremental parsing of large trees.
(01:28:12-01:29:48)
The talk also announced GitHub SHA-256 repositories in private preview,
not general availability. (01:29:48-01:30:19)
In Q&A, portability remained an intentional goal, but one requiring people
to perform and maintain the ports. Rust reviewer capacity was another
constraint: learning the language could initially slow upstreaming. New
isolated features and self-contained subsystems offered better entry points
than especially complex performance or format migrations. Pluggable
reference handling and future object work were candidate boundaries, and
parser contributions were welcome. (01:30:19-01:35:58)
The long-term language direction was not a proposal for an agent-driven
rewrite. carlson said Git's contribution policy, provenance requirements, and
Developer Certificate of Origin requirements precluded most AI-authored
contributions, distinguishing those submissions from AI-assisted review or
correctness checking. (01:36:30-01:38:37)
carlson reported broad support at the 2024 contributor summit, with
portability the main concern rather than ideological opposition, and credited
an xdiff Rust proposal with helping stimulate the process. Reducing
memory-safety vulnerability classes was a goal, without assuming that every
incoming security report was valid. Users of less common platforms still
needed up-to-date Git, but continued support depended on actual porting
resources. (01:39:07-01:42:49)
Git Meta: the Git Notes we deserve
Speaker: Scott Chacon (GitButler).
Recording: 02:03:39-02:46:55, including Q&A.
Why metadata needs its own representation
Chacon argued that repository metadata increasingly needs to describe more
than commits. Author and committer fields, trailers, signatures, and tracked
files cover existing uses, but human and AI provenance introduces models,
prompts, and transcripts. Reviews, evolution history, license and CI
attestations, and binary locks or hints add further requirements. His goal was
a shared representation supporting fast local queries and updates, ordinary
Git transport, and manageable concurrent edits. (02:04:09-02:09:15)
He contrasted this with enlarging commit messages, which can overwhelm logs
and cannot be changed without changing commit identity and descendants. In
his account, custom headers are poorly exposed in normal interfaces, and
ordinary git rebase drops unknown headers. Files mix administrative data
with source history; git notes becomes awkward for large collections and
concurrent structured mutation. Git Meta was intended to address these
limits, rather than merely provide longer commit annotations.
(02:09:15-02:12:19)
Targets, value types, and the specification
Git Meta was presented as a specification, with a Rust
command-line tool and library serving as a reference implementation. Chacon
was not asking other implementations to standardize on that code. Targets
include commits, branches, change IDs, paths, and whole projects. A path
target identifies a path independently of any particular commit.
(02:12:19-02:15:25)
Namespaced keys hold strings, lists, or sets. Comments and CI events fit
lists; owners and reviewers fit sets. Two demonstrated commands attached an
agent-model value to HEAD and requested JSON metadata for paths under
src. The second was a prefix query, not merely a lookup for one exact path.
(02:13:50; 02:14:30)
git meta set commit:HEAD agent:model astra
git meta get --json path:src
Git-backed storage and ordered publication
The reference implementation uses SQLite as its mutable local key-value
store; the specification does not mandate it. Shared state is serialized as
ordinary Git commits, trees, and blobs under refs/meta/local/main. Targets
use a fan-out layout, namespaced keys map into tree paths, and a reserved
__value blob holds a leaf value. Lists use a __list tree whose entry names
combine a timestamp with a content-hash fragment, allowing independent
appends to interleave. Selected key namespaces can be serialized to separate
refs for different sharing destinations. (02:15:25-02:18:05)
Publication is fast-forward-only. After a rejected push, a writer fetches the
updated tip, reconciles its changes, rebases, and retries; another intervening
push can require another attempt. The concurrent-add examples unioned lists
and sets. For conflicting strings, the reference implementation retains the
retrying writer's local value, while the earlier value remains in committed
history. (02:18:05-02:20:07)
In Q&A, Chacon explicitly characterized this as CRDT-shaped, not a CRDT
library implementation. Ordinary Git trees and
ordered publication provide the demonstrated convergence. Ref updates still
coordinate a linear history, prompting a question about centralized ordering.
This is not an order-independent winner for strings, nor a claim that every
conflict is automatically resolved in a semantically useful way.
(02:33:02-02:34:32; 02:45:47-02:46:51)
Keeping the working set small without erasing history
Serialization commits cache which keys they changed, helping locate a key's
last historical write. The tip tree can be pruned to a configurable recent
working set while retaining past values in history. Blobless cloning
initially obtains commits and trees, then hydrates active values through a
promisor remote. Fetches can be batched, for example retrieving the required
metadata keys for one commit in a single round trip. (02:20:07-02:24:41)
Optional shallow history further reduces client state, but an unknown older
key may require explicit deepening rather than being permanently unavailable.
Pruning the tip is not physical deletion or garbage collection of historical
blobs: stronger erasure requires a separate history strategy. Pinning selected
keys against pruning was not implemented in the reference implementation.
(02:23:10-02:24:41; 02:43:12-02:45:16)
Chacon's benchmark chart showed the following totals:
Dataset
Illustrated total work
100,000 keys
12.8 seconds
One million keys
79.5 seconds
Five million keys
553 seconds
Five million keys, shallow
93 seconds
These totals included background indexing after setup returned; they were
not all blocking setup latency. The shallow run had no repack phase. These
were his own synthetic or representative tests, not general performance
guarantees. (02:24:55)
Applications and unresolved sharing conventions
Chacon demonstrated importing GitHub pull-request metadata for local,
pull-request-oriented blame: lines were grouped by pull request rather than
individual commit, with a URL. The resulting information remained useful
offline and transferable, illustrating an application beyond annotations
displayed alongside commit messages. (02:25:43-02:26:35)
He also described a local issue-tracker experiment. Keys beginning with
local: are not serialized by the reference implementation. Large images,
screenshots, and video were not intended as ordinary payloads. Options
included keeping data local, using separate refs, or placing binaries
elsewhere or in LFS and attaching references. The design was not a replacement
for arbitrary binary storage. (02:28:25-02:30:30; 02:39:38-02:40:38)
The Q&A left several interoperability questions open. Filtering public and
private namespaces was possible in the design but not extensively validated
in real-world scenarios. Composite commit, path, and line-range review targets
were not standardized; structured values could encode them, but shared
schemas were still needed. Standard target fan-out constrained arbitrary new
target kinds, with the project target available for experiments. A forge's
default presentation of unknown keys remained undecided.
(02:33:02-02:36:34; 02:39:08-02:43:12)
Chacon chose existing Git transport for immediate forge compatibility rather
than requiring a new API. Very large datasets and enterprise topologies still
needed evaluation; scaling to billions of keys was not established.
(02:36:34-02:38:35)
Scaling Git for the AI Era
Speaker: Elijah Newren (GitHub).
Recording: 02:47:35-03:23:25, including Q&A.
Ending merge-base walks when one side is exhausted
Newren's premise was that growing load makes both large and small
inefficiencies compound. He focused on improving server and client behavior
without abandoning Git, rather than treating increased demand as a reason to
replace it. (02:47:53-02:48:54)
He described merge-base computation as a two-color backward traversal that
finds best common ancestors. The old algorithm continued until every queued
commit was stale. Even a recent pull request could trigger a huge traversal
if the main branch had recently merged a branch rooted far back. The
optimization stops when either branch-exclusive color is exhausted: no
further first meeting between the two sides is possible.
(02:48:54-02:53:40)
Newren reported 70-fold speedups in many cases in one monorepo and a 20-fold
average in another. He said the optimization was deployed at GitHub and
scheduled for the next upstream release. Valid commit-graph data was a
prerequisite, however: rollout exposed a pre-existing bug involving corrupt
commit-graph data that required repair and hardening. The deployment
therefore involved correctness work as well as reducing traversal time.
(02:54:20-02:55:41)
Aggregating packs without doing a full repack
Many packfiles multiply object-lookup overhead. Under continuous pushes,
Newren said even a full or geometric repack taking 20-30 minutes could finish
with thousands of packs still present. His proposed pack-aggregate
operation combines recent small packs and loose objects while leaving large
optimized packs alone. Expensive delta searches, bitmaps, multi-pack indexes,
and other maintenance remain the job of a full repack. Aggregation can run
before, and concurrently with, longer maintenance. (02:55:41-03:03:50)
Starting packs
Starting loose objects
Resulting packs
Resulting loose objects
Time
1,427
13,578
5
0
6.17 seconds
7,260
54,493
14
0
45.33 seconds
These were experimental measurements. Fewer packs helped object lookup,
ancestor queries, and ahead/behind calculations; diff-heavy workloads
benefited less dramatically. (02:58:14-02:59:14; 03:00:55)
Stress testing uncovered races, many of them pre-existing. After fixes,
Newren reported 24 hours with thousands of aggregations and repacks alongside
concurrent pushes, merges, and readers, without failures or missing objects.
An approximately 20-patch series was being prepared. These results described
experimental work, not a generally shipped command.
(03:02:19-03:03:50)
Batching partial-clone fetches across known ranges
Newren distinguished two separate controls: partial cloning with
--filter=blob:none retains commits and trees while deferring blobs; sparse
checkout limits which files are checked out. Neither makes
one-object-at-a-time lazy fetching efficient. Network round trips can
dominate, so required blobs need to be batched per operation.
(03:04:51-03:07:27)
His timeline traced this work from checkout in 2017, through diff and delta
bases in 2019 and rename, pack, and merge work in 2020-2021, to cherry,
grep, and revision-range backfilling in 2026. He reported some
cherry-related cases becoming hundreds of times faster.
(03:07:27-03:07:57)
Per-operation batching can still leave git log -p fetching once per commit.
When the desired revision range is known, proactive backfilling can fetch its
blobs together. In the demonstrated sequence, using the backfill improvements
being discussed, the log command required no additional fetches:
git backfill A..D
git log -p A..D
Although A is excluded from the commit set A..D, showing B's patch
requires A's file version. The change therefore includes boundary-parent
blobs by default, rather than limiting collection strictly to the commits
selected by the range. (03:08:27-03:09:50)
The Q&A highlighted why manual prefetch cannot transparently repair every CI
pipeline. Backfill follows sparse-checkout configuration, but git log
historically ignores those boundaries and can fetch files outside them.
Broader optional sparsity modes remained work to do, and repeated batch-fetch
bugs showed that other commands were not all fixed. The demonstration was not
a universal no-fetch guarantee. (03:09:58-03:10:29; 03:15:08-03:17:13)
Sending less from shallow clones
Shallow cloning with --depth truncates known ancestry. In Newren's example,
a local boundary commit, C2, no longer appeared among the server's
advertised ref tips. If the client cannot find common history, it can send a
large reachable snapshot instead of only the new objects. Combined with
partial cloning, this can first download gigabytes of missing data from the
server and then push the same data back. (03:10:29-03:13:36)
The proposed fix assumes that the destination has the shallow boundary and
sends just the new data; the server rejects the push if that prerequisite is
absent. Newren described this as shifting an existing shallow-history
assumption by one commit, rather than eliminating assumptions about
unavailable ancestry. Patches were on the mailing list at the time of the
talk, not presented as already landed. (03:13:36-03:14:37)
Client rollout and remaining hardening work
Audience questions raised further shallow-history problems: amending,
rebasing, or cherry-picking can create surprising root history and misleading
corruption errors. Newren said hardening was being worked on. He preferred
delivering client fixes through upstream releases rather than a
GitHub-distributed fork. Server rollout was controllable; the timing of users'
client upgrades was not. (03:17:13-03:20:44)
Newren did not consider push.negotiate=true a complete solution. It depends
on server-side fetch support using protocol version 2, and quick object
lookups during concurrent server repacking can still miss common commits. He
also argued that a client should not upload gigabytes merely to discover that
the wrong destination lacks the required history. Negotiation helps, but a
backstop against the shallow-push failure remained necessary.
(03:21:14-03:23:17)
Josh: Beyond subtrees
Speaker: Christian Schilling.
Recording: 04:31:15-04:57:49, including Q&A.
Monorepo development with independently shared subsets
Josh originated with an embedded and automotive supplier that wanted
monorepo development internally while sharing different subsets with
different customers. Schilling described git subtree as too slow for that
workflow. The requirement was not merely to extract files once, but to keep
working across repository boundaries. (04:31:32-04:33:36)
Rust provided another demanding use case: coordinated changes across the
compiler and tools such as Miri and rust-analyzer, while preserving autonomy
for individual teams. Schilling described a gradual move from submodules and
subtrees toward Josh. Rust's broad, complex history exposed correctness and
performance edge cases that simpler repositories had not.
(04:33:36-04:35:08)
Self-contained projections of Git history
Josh constructs materialized views of Git's content-addressed history graph,
rather than filtering only the current working tree. A projection produces
self-contained Git history that can be distributed without continually
consulting the original server for missing objects. Its relationship to the
original repository supports mapping subsequent edits back.
(04:35:08-04:37:12)
The filter language had more than 30 composable operations. Examples shown
on the slide included:
Filter
Effect
:/apps/web
Select the contents of a folder.
::**/Cargo.toml
Select matching Cargo.toml files.
:exclude[::docs/]
Exclude the documentation directory.
:[::apps/web/,::libs/lib1/]
Combine selections while retaining their layout.
Composition is not generally commutative: operation order can matter.
Schilling also described experimental content-aware filter computation using
checked-in Starlark and WebAssembly, with wasmi, potentially deriving
selections from build dependencies. The final design was not settled.
(04:37:35-04:39:15)
Josh offers both a local command-line interface and a server-side Git proxy.
As clarified in Q&A, local filtering first downloads the complete repository.
A partial-clone-first optimization was not implemented. Avoiding transfer of
excluded content and history therefore required the proxy; a locally produced
small projection did not imply a small initial download.
(04:50:04-04:51:37)
Mapping edits back without resurrecting deleted files
A projection discards information, so reversing its updated tree cannot
reconstruct the entire monorepo. Simply overlaying the result on the previous
full tree is also incorrect: files deleted in the projection would remain in
the old tree and reappear.
The reverse path instead:
Starts with the previous full tree.
Removes all paths that the projection is allowed to modify.
Inserts the reverse-mapped contents of the updated projection.
This preserves both untouched monorepo content and deletions made through
the filtered view. The excluded data comes from the existing full tree, not
from inverting an information-losing transformation alone. Schilling
distinguished this tree-level operation from commit-topology mapping, which
requires additional logic. (04:39:46-04:41:53)
Incremental reuse, rather than universally faster filtering
Schilling attributed performance to direct work on Git's graph and data
structures, cached incremental transformations, and avoiding unnecessary
traversal or quadratic algorithms. Other techniques included composing
known-fast operations, deferring I/O, working in memory, and using
header-only access where full decompression was unnecessary.
(04:43:57-04:46:29)
His medium-sized benchmark used git/git, with 81,879 commits. An update of
100 first-parent commits introduced 375 reachable commits because of merges.
The subtree comparison shown at 04:43:45 was:
Work
Josh
git-filter-repo
Cold filtering
5.58 seconds
4.77 seconds
Processing the update
196 milliseconds, incrementally
4.59 seconds, full rerun
The cold result favored git-filter-repo. The dramatic difference came from
reusing previous work, not from Josh being uniformly faster on an initial
transformation. Schilling explicitly noted that the update comparison was
not like-for-like: git-filter-repo does not provide Josh's incremental mode.
(04:42:56-04:43:57)
Workflow experiments and Q&A
Schilling compared inexpensive filtering with inexpensive branching: making
an operation cheap can change how people collaborate. He suggested
applications beyond source code, including other data maintained in Git.
(04:47:00-04:50:04)
An experimental CI approach described each job's working set through a filter
and a dependency graph, resembling a coarse-grained build system. AI could
generate filter descriptions or work within narrowed repository contexts,
but was not needed inside the deterministic filtering engine.
(04:52:08-04:55:44)
Schilling also distinguished the tools' purposes. git-filter-repo supports
general one-off history rewrites and arbitrary scripting. Josh targets
simultaneous original and filtered references with ongoing round-trip
development. Those are different capabilities, not grounds for treating
either tool as a universal replacement for the other.
(04:56:14-04:57:46)
Checking out another branch of Version Control
Speakers: Raphaël Gomès and Pierre-Yves David (Mercurial).
Recording: 04:59:25-05:35:20, including Q&A.
Convergence does not mean automatic agreement
Gomès and David argued that small data-model differences can enable
substantially different distributed workflows. Their reference point was a
conflict-free replicated data type: replicas accept concurrent updates and,
after exchanging information, converge on the same representable state. That
state may still contain conflicting work. Convergence does not automatically
determine user intent or choose the correct code.
(05:01:53-05:05:26)
They contrasted centralized version control, where reconciliation precedes
committing to the central repository, with distributed systems that allow
local commits and reconciliation later. Git's immutable object store exchanges
well. The contrast concerned mutable, locally named branch pointers and their
tracking configuration, which do not themselves provide globally convergent
branch identity. (05:01:53-05:06:57)
In the Mercurial model they described, branch identity belongs to commits.
Coloring the graph by branch makes membership explicit and allows heads to be
derived. Exchanging only part of a branch or a stack retains that membership;
moving within a branch does not require a detached-head analogue merely
because the checkout is away from a tip. (05:06:57-05:09:00)
Multiple heads explicitly represent concurrent work. Peers can exchange that
state before choosing whether to merge, rebase, fold, or drop changes. The
presenters argued that one representation of divergence avoids multiplying
special cases for every combination of how divergence arose and how it is
resolved. (05:09:30-05:13:04)
Exchanging rewrite intent as well as replacement commits
Rewriting history creates new content-addressed commits; it does not modify
old objects. Moving a pointer alone does not communicate that an earlier
change was intentionally dropped or superseded. Other replicas can
reintroduce old copies, while dependent changes remain based on obsolete
work. (05:13:04-05:15:40)
Mercurial's obsolescence markers are stored separately from the commit graph.
They record rewrite relationships, successors, who performed the operation
and when, including splits and folds. This carries more information than a
single change identifier. David dated the broader changeset-evolution effort
to 2011. (05:15:40-05:17:12)
At 05:18:55, the demonstration used hg obslog --rev s0 to trace an amendment
followed by another user's rebase. In the broader example, rewriting a base
change and moving it onto main left dependent changes orphaned. Following
the successor chain identifies the replacement base and permits automatic
relocation in a simple case. (05:17:12-05:19:18)
Obsolescence-aware patch comparison can also distinguish actual edits or
conflict-resolution changes from unrelated rebase noise. The capability
presented was distributed mutable history: immutable commit objects
accompanied by exchanged information about how their meaning and successors
evolve. (05:19:18-05:20:48)
Flat manifests and history-aware compression
The storage discussion contrasted Git's trees with Mercurial's flat manifest
mapping paths to file revisions. A flat representation initially appears
disadvantaged, especially when a small change affects a large
repository-wide listing. (05:21:19-05:23:20)
The presenters described more than a decade of optimization exploiting the
similarity of successive manifests and known history when choosing delta
bases. Compressed storage is built online and can be reused during exchange,
without periodic Git-style repacking for the storage path described. Chain
heuristics balance length and size, maintaining practical read costs while
allowing the delta structure to accelerate comparisons.
(05:22:50-05:27:28; 05:32:35-05:33:39)
Their selected measurements distinguished metadata from file contents:
Timestamp
Workload and portion measured
Reported comparison
05:23:40
Homebrew-cask tree/manifest portion
350 MB to 50 MB
05:24:35
Large private repository, 26 million revisions
2 TB in a tree-like representation versus 26 GB of manifests, or 1.3%
05:25:50
Nixpkgs file contents
1 GB to 0.5 GB
These were not whole-repository size comparisons. At 05:24:53, the presenters
also reported 1.6 PB of uncompressed manifests and approximately 60,000-fold
compression for the private-repository case. That ratio was a manifest-history
result, not a guarantee for ordinary file data.
The fast-diff slide at 05:27:05 associated a roughly 100 MB manifest with a
one-microsecond comparison; the spoken account described real cases ranging
from microseconds to milliseconds. The mechanism reused storage and delta
relationships, rather than scanning an arbitrary 100 MB in that time.
Further format and algorithm work remained ongoing.
Avoiding mandatory central round trips
For sufficiently large repositories, the presenters argued, the important
threshold is whether the data fits on a developer machine or forge, rather
than a small difference in command latency.
A uniform distributed model allows developers and CI systems to exchange
directly. Intermediate commits need not all pass through a central service,
avoiding a mandatory central bottleneck and global network round trip for
every step.
A Git-serving layer backed by Mercurial storage, and shared Rust components,
were possibilities under exploration. They were not announced as a shipped
Git backend. (05:28:01-05:31:35)
Q&A: competing successors preserve a conflict
Competing rewrites can produce divergent successors. The system represents
those alternatives, but a workflow or user still has to choose between them
or reconcile them.
That is distinct from an intentional split, where several successors
collectively replace one change. Convergence preserves the evidence needed
to make the distinction; it does not silently decide which competing
implementation is correct. (05:34:10-05:35:10)
Rebuilding Git?
Speaker: Patrick Steinhardt (GitLab).
Recording: 05:36:20-06:13:18, including Q&A.
Modernization rather than replacement
The question mark mattered: Steinhardt described upstream-first refactoring,
not a wholesale rewrite, fork, or generated replacement.
Git's data model and protocol remained largely usable. The pressure came from
local-filesystem assumptions dating to 2005 meeting large repositories,
concurrent pushes, CI clone fan-out, and growing agent workloads. GitLab's
work sought replaceable reference and object storage while retaining a shared
Git foundation. (05:36:20-05:41:52)
References behind a common storage interface
Loose references encode names as filesystem paths, exposing differences in
case handling, Unicode, and naming rules. Packed references avoid some costs,
but deleting an entry rewrites the file; Steinhardt cited an example exceeding
2 GB. He also identified the lack of consistent all-reference snapshots and
atomic multi-reference updates as limitations of the traditional design.
(05:41:52-05:43:54)
Reftable uses immutable, sorted, prefix-compressed block tables. The active
tables.list identifies the current tables; publication adds a table and
atomically replaces that list. Tombstones represent deletions. This changes
the immediate update cost, without eliminating future compaction work.
(05:43:54-05:44:56)
The broader achievement was a common reference-backend abstraction,
replacing callers' direct access to .git/refs rather than merely adding
another format alongside hardcoded filesystem assumptions.
(05:44:56-05:45:29)
Separating object storage from repository globals
Steinhardt regarded loose objects and packfiles as useful formats. The
obstacle to alternatives was their integration: hardcoded callers and global
repository dependencies.
The refactoring made components self-contained, including the multi-pack
index, packfiles, loose objects, and commit-graph. A single
object_database entry point and an odb_source virtual-method interface
provided the separation needed for different sources. (05:46:00-05:49:40)
The roadmap shown at 05:51:35 included:
Git version
Milestone as presented
2.54
odb_source, with files and loose-object sources
2.55
In-memory source and transactions
2.56
Maintenance and fsck; release due at the end of September
2.57
Planned: make alternates an implementation detail of the files backend
That final separation, together with a repository extension, would enable
full object-backend selection. The complete roadmap was not presented as
already released.
Demonstrations are not production architectures
The interfaces open storage choices such as SQL databases, remote object
stores, content-defined chunking, and transactions. Steinhardt did not argue
that every possible backend would be a good design.
An instructional SQLite demonstration stored both references and objects in
one database and supported conventional commits. Its purpose was to
demonstrate the abstraction, not propose a production deployment.
(05:51:44-05:54:50)
A deliberately naive S3 demonstration stored each object and reference under
a separate key, then recreated the local repository skeleton while retaining
the remote data. It functioned, but exposed a poor fit at scale: millisecond
requests replace microsecond local-access assumptions, request counts and
costs grow, pack delta compression disappears, and listing references or
updating several atomically becomes problematic.
Those limitations belonged to that demonstration, not to the intended
architecture. (05:54:50-05:57:54)
Immutable artifacts, manifests, and local Git access
The intended design retains packfiles and reftables. Immutable manifests list
active artifacts, while generations identify the published manifest. Object
storage is the durable source of truth.
Gitaly nodes cache artifacts locally and can serve any repository. Git
itself accesses only local disk; Gitaly handles remote uploads and
downloads. This separates durable storage from an elastic compute fleet
whose local repository data can be treated as cache.
(05:57:54-05:59:27)
Steinhardt contrasted this with an earlier attempt to create full repository
snapshots using hard links for each operation, which proved too expensive.
(06:00:58-06:01:28)
The multiversion concurrency-control sequence was:
Pin each remote procedure call to a manifest, giving it a consistent
snapshot.
Let a writer create new packfiles and reftables privately and prepare a
replacement manifest.
Have Gitaly upload the new artifacts, then atomically publish the new
manifest as the active generation.
Keep existing readers on their original snapshot. If another writer
publishes first, reconcile the stale writer's changes against the new
state before retrying where possible.
(06:01:28-06:03:30)
The Q&A clarified that immutable object additions can be combined. Reference
updates require a three-way comparison of the base, proposed, and current
values. Disjoint updates can merge; conflicting concurrent updates to the
same reference fail the operation or push. This is not automatic combination
of users' commits. (06:06:33-06:07:36)
The foundations were upstream work. An upstream contribution of the
multiversion backend itself remained conditional on interest, not a claim
that it had landed. (06:04:01-06:05:32)
Compatibility and the unfinished plugin boundary
Other implementations, including libgit2 and gitoxide, need support for
repository formats they encounter. Repository extensions and the option to
choose legacy storage provide a compatibility boundary, not automatic
interoperability with every existing tool.
Steinhardt noted that the reftable library is MIT-licensed. That did not imply
wholesale relicensing of Git.
The demonstrated custom backends required compiled-in implementations.
Loading shared objects or using an external-service protocol without
recompiling remained future design work. An interface shared across Git
implementations was an aspiration, not an existing plugin ABI.
Steinhardt's goal was varied storage implementations on one shared
foundation, rather than isolated vendor forks. He also credited competition
from JJ with helping encourage Git's modernization. (06:07:36-06:12:47)
The Git Forge Must Be Hackable
Speaker: Anirudh Oppiliappan (Tangled).
Recording: 06:58:05-07:21:05, including Q&A.
Open interfaces and selective self-hosting
Oppiliappan framed hackability as the ability to replace or personalize parts
of a forge without losing participation in its network. Open formats and
interfaces should let users and agents build tools that do not depend on one
company's software. Agent-friendly operation and scalability were design
aspirations. (06:59:10-07:00:42)
Self-hosting should not be an all-or-nothing decision. Users should be able
to host all, some, or none of the system themselves. Separating identity,
repository hosting, indexing, presentation, and automation makes those
choices possible without requiring everyone to operate an entire forge.
(07:16:14-07:18:16)
Portable identity and the record pipeline
The Authenticated Transfer (AT) Protocol supplies account-associated records
and portable identity. Personal data servers (PDSs) hold signed, structured
records, while identities can move between hosts. Tangled's record types
include org.tangled.feed.star and org.tangled.actor.profile.
Relays collect the record firehose. Indexers and app views select relevant
record types and build materialized views, which clients query through APIs.
In this model, OAuth-authorized writes go directly to the user's PDS.
Self-verifying records separate authenticity from trust in the intermediary
carrying them. A client need not trust a relay to establish a record's
authenticity; that does not mean the relay guarantees availability.
(07:00:42-07:03:47)
Repository-owned collaboration and replaceable components
Tangled divides its implementation into several roles:
Component
Role
Knots
Headless Git remotes, hosting repositories and collaborative data
Spindles
CI and webhooks
Bobbin
Diskless, in-memory API index and identity cache
Other app-view services
Repository mirroring, code search, and notifications
Client
A Svelte UI over those interfaces
(07:04:19-07:09:58)
Issues, pull requests, collaborators, and access-control lists are represented
as Git-backed collaborative objects (COBs). Oppiliappan credited Radicle as
prior art.
This storage choice addresses an ownership problem in the earlier design.
Keeping issues and pull requests in their authors' PDSs allowed those authors
to delete them entirely. Oppiliappan argued that contributions should persist
under repository control even when a contributor disowns them. Moving
collaborative data into the knot separates that requirement from personal
profile and social records, which retain the PDS ownership model.
(07:04:50-07:07:25)
The new Rust knot implementation was described as avoiding server-side
invocation of the git binary. It projects collaborative data into an AT
Protocol repository represented as a Merkle search tree, behaving
sufficiently like a PDS for relays to ingest it. (07:06:54-07:07:55)
Bobbin's serving state is reconstructable. Recalling figures from memory,
Oppiliappan estimated that its then-current metadata index fit within roughly
100 MB, with cold hydration taking under 30 seconds. Those estimates
concerned the index, not all Git repository contents or performance at
arbitrary scale. (07:08:25-07:09:26)
Interchangeable storage and automation
The XRPC HTTP interface defines a knot's contract without prescribing its
storage implementation. A demonstration used a knot on Cloudflare container
infrastructure backed by an external code-storage service. The ordinary
Tangled repository view still interoperated with it.
(07:09:58-07:12:04)
An agent created a repository and populated commits through the open
interfaces, while CI operated separately. A spindle need not use Tangled's
NixOS microVM scheduler: external automation can participate instead.
Mitchell Hashimoto's tack illustrated connecting another CI system, using
Buildkite as the example. (07:14:40-07:15:43)
The architectural point was substitutability: a compatible implementation
can change how repositories or automation are hosted without requiring a
different client.
A preview of change-oriented review
Oppiliappan previewed a revised UI with stacked pull requests, selectable
change ranges, successive patchset revisions, and interdiffs. It also
exposed JJ change IDs, while retaining commit-based review.
These features belonged to the UI revamp shown during the talk. They should
not be read as a statement that every demonstrated capability was already
generally available. (07:12:36-07:14:08)
Custom clients and Q&A
Lightweight APIs allow personal frontends or custom forges to retain identity
and participation in the wider network. Agents can use those APIs, PDS data,
and the firehose rather than depending on the hosted UI.
(07:16:14-07:18:16)
Asked whether creating a repository was simply a PDS write, Oppiliappan
distinguished two operations: a declaration record lives in the PDS, and a
separate knot call creates the repository. Static or custom frontends can use
knot APIs. (07:18:16-07:19:53)
There was no near-term plan for direct JJ integration in the Rust knot. The
JJ-aware review preview was therefore distinct from native backend
integration. (07:20:33)
Can we design a Git smartlog?
Speaker: Toon Claes (GitLab).
Recording: 07:21:50-07:56:03, including Q&A.
A view of work in flight, not just history
Claes started with a familiar failure mode: git log --graph --oneline can
become very wide, especially as broader ref selection brings unrelated work
into view. A history browser and a working-tree status display answer
different questions. A smartlog should make ongoing work easy to locate.
(07:22:43-07:27:21)
JJ, Sapling, and GitButler demonstrate useful defaults, but differ in what
they consider relevant and how they present uncommitted work or metadata.
There is no single definition of a smartlog implied by those examples.
(07:24:15-07:26:51)
Git already offered partial mechanisms. --first-parent simplifies
traversal but hides merged-side commits. The existing
--graph-lane-limit=<n> option in Git 2.55 caps graph width. Neither alone
supplies the proposed work-focused view; these existing options are distinct
from the new proposals that followed. (07:27:21-07:29:25)
Proposed composable options
None of the following proposed options had been upstreamed at the time of
the talk. Names and semantics remained open to debate.
Proposed option
Intent
Timestamp
--untangle
Cut redundant long connecting lanes below merged topics while retaining commit entries, unlike first-parent traversal.
07:31:35
--mine
In the example, select branch tips authored by the configured user; not a settled definition of branch ownership.
07:34:07-07:35:09
--fold-others
Collapse intervening commits by other people into ellipses and counts.
07:35:50
--down-to-upstream
Stop each branch's traversal at its own upstream rather than continuing through history to the root.
07:36:45
--ahead
Annotate both ahead and behind counts.
07:38:20
--worktrees
Show checkout directories so parallel work, including work by agents, can be located.
07:39:25
--decoration-hook
Let a user script supply CI, review, or signature metadata without binding Git to one forge.
07:41:15
--unicode
Use box-drawing characters instead of ASCII graph glyphs.
07:44:50
These mechanisms address different parts of the display: relevance,
traversal boundaries, graph layout, and contextual metadata. Composing them
would allow a useful view without making every choice inseparable from the
others.
Incremental implementation versus a fixed policy
Claes initially favored upstreaming individual log options, then documenting
how users could compose them into a git smartlog alias. A section 7 manual
page was one possible home for that recipe.
This would let independently useful pieces arrive incrementally without
establishing one workflow as a permanent default. It was a proposed delivery
route, not a released command or a ready-to-use alias composed from available
options. (07:41:27-07:45:36)
Q&A: defaults can improve without forbidding customization
Audience members challenged the documentation-first approach. A configurable
default could provide a useful out-of-the-box view and let upgrades improve
it, rather than requiring users to maintain an alias copied from a manual.
Choosing defaults need not prohibit customization or impose an unchangeable
workflow.
Claes was receptive, but no final decision on delivery or default policy
emerged. JJ's configurable revsets and templates, along with Sapling's
approach, were offered as prior art for combining useful defaults with user
control. (07:48:15-07:48:47; 07:51:22-07:53:25; 07:54:58-07:55:28)
Q&A: an upstream is not necessarily an integration boundary
The most consequential semantic objection concerned --down-to-upstream.
A branch's tracking upstream is not necessarily the integration or default
branch. If tracking-upstream history is hidden, pushing a branch can make its
work disappear from the display even though it has not been merged.
Participants suggested that a distinct integration boundary, such as
origin/main, may better represent completed work. Audience members also
noted that a server's symbolic HEAD often already advertises its default
branch.
The discussion did not settle how smartlog should choose this boundary.
Treating a branch's tracking upstream as universally correct would conflate
synchronization with integration. (07:48:47-07:50:52; 07:54:58-07:55:28)
Q&A: graph layout is more than a glyph choice
Participants suggested exploring Rust graph-rendering work from Sapling or
JJ to reduce diagonals and improve layout. That is a separate task from
replacing ASCII characters with Unicode box-drawing glyphs.
No renderer integration was presented as completed. The suggestion broadened
the design discussion from limiting width or changing characters to making
the graph itself easier to follow. (07:52:24-07:54:58)
SHA-256 at a Hyperscaler
Speaker: Emily Shaffer (Google, JJ team lead).
Recording: 07:57:00-08:28:30, including Q&A.
Content identity and the reason to migrate
Shaffer started from Git's content-addressed Merkle structure: object
identities depend on their contents, and references connect objects into
history. Collision resistance makes matching IDs useful evidence of matching
content and history across clones; finite hashes do not guarantee that
different inputs can never collide.
SHA-1 collision attacks motivate moving to a stronger replacement. Shaffer
distinguished known collisions from useful malicious Git payloads, which she
described as harder to construct. (07:59:14-08:01:52)
SHA-1 object IDs contain 40 hexadecimal characters; SHA-256 IDs contain
64. Git already supported SHA-256 repositories, but forge adoption was
uneven. Shaffer updated the presentation to note GitHub's private preview
announced that morning, not general availability. (08:02:15-08:02:54)
Conversion requires the objects, not just their old names
A corresponding SHA-256 ID cannot be computed from the old SHA-1 string alone.
Conversion requires the source objects, and references embedded in those
objects also change.
Switching only new commits does not secure a repository whose old history
and dependencies remain identified through SHA-1. The rejected design is a
mixed SHA-1/SHA-256 object graph. Shaffer called for wholesale conversion of
known-correct history, including referenced objects and dependencies such as
submodules.
Distributed forks further complicate choosing a shared cutover point. The
migration must account for relationships across repositories, not merely
change the algorithm used for the next local commit.
(08:02:15-08:06:35)
Object IDs escape the repository
Commit and object IDs appear in CI and release pipelines, vulnerability and
dependency scans, tickets, email, chat, and signed-tag payloads. Changing
repository identities leaves those references needing a way to find the
corresponding objects.
Tools must accept longer IDs; truncation is not migration. Existing
signatures also require explicit handling. Rebuilding the object database
does not automatically repair the surrounding ecosystem.
The talk identified historical-ID lookup and signature handling as migration
requirements, not finished infrastructure. (08:07:00-08:09:03)
Google's pressures and temporary bridges
Google cannot simply let every early adopter choose new source storage:
approved-storage policy limits that freedom. Shaffer described several
pressures on the schedule.
The planned SHA-256 default in Git 3.0 for newly initialized repositories
would create demand without converting existing repositories. External
SHA-256 dependencies require import and contribution paths. A practical
malicious SHA-1 break would make the remaining work urgent, rather than being
a prerequisite for starting it. (08:09:33-08:12:07)
On managed platforms, system Git configuration could temporarily retain
SHA-1 as the initialization default. A Copybara extension had already been
written to import SHA-256 sources while converting them to SHA-1 for legacy
internal tooling. That is an interoperability bridge, not a security upgrade
to the internal SHA-1 representation.
Dependency updates and cross-references remain difficult. Recording a source
hash in a commit message can carry useful metadata, but does not provide
robust lookup infrastructure. (08:12:07-08:15:40)
Two coherent server views, not a mixed object graph
Forthcoming server interoperability work by brian m. carlson would keep
SHA-256 as the canonical source of truth and derive a SHA-1 view. Clients
could request either format from the same repository URL. This work was
explicitly not expected in Git 3.0. (08:15:40-08:16:42)
Android illustrates the coordination problem: its many repositories,
dependencies, vendors, and forks make a synchronized flag day impractical.
Shaffer described four stages:
The network uses SHA-1.
Servers throughout the network enable interoperability, making SHA-256
canonical while most clients still use SHA-1.
Tools and clients migrate over time, with both representations available.
Once SHA-1 users are gone, the compatibility view can be removed.
Shaffer cautioned that the transition could take years. Each view would
remain internally coherent. Supporting two representations does not mean
mixing SHA-1 and SHA-256 references within one object graph.
(08:16:42-08:18:15)
The benefit begins before the final stage: retaining a known-good SHA-256
representation can expose changed content that still has the same SHA-1 ID.
Reconversion or comparison with independent SHA-256 mirrors can reveal the
disagreement. That provides additional evidence during migration, rather
than automatically making every legacy client safe.
(08:18:15-08:19:15)
Q&A: missing history and durable cross-references
A historical submodule pointer exposes a hard limit. If its target
repository has disappeared, the old hash is insufficient to derive the new
one. Rehashing or archiving helps only if the underlying contents can be
recovered.
The discussion did not resolve the case of a completely lost target.
It did not establish deleting or replacing the pointer as a completed
migration solution. (08:19:15-08:20:16)
For old IDs embedded in external records, participants proposed archived
mappings and lookup infrastructure. These remained proposals, not a
finalized protocol for preserving historical references.
(08:20:16-08:20:47)
Q&A: verifying bytes does not authenticate a historical mapping
An audience member challenged the initial suggestion that rehashing
downloaded data would detect a malicious mapping. Two different contents
that collide under SHA-1 can each have valid, different SHA-256 IDs.
Checking downloaded bytes against an advertised SHA-256 hash therefore does
not establish which contents an old SHA-1 reference was intended to identify.
It verifies the bytes against the supplied new identity, not the correctness
of the old-to-new association.
Independent mirrors were suggested as a mitigation. Comparing them could
expose different contents associated with one SHA-1 ID. But the discussion
did not specify a trust protocol for deciding which mapping is authoritative.
A server's SHA-256 representation does not automatically provide complete
protection to legacy clients. (08:22:29-08:25:33)
Shaffer knew of no malicious Git collision injection and cited SHA1DC, the
collision-detecting SHA-1 implementation, as a mitigation. That was not a
claim of complete immunity. (08:21:27-08:21:58)
Q&A: algorithm identification, future migrations, and performance
An abbreviated hexadecimal object ID does not identify its hash algorithm;
carlson confirmed that no algorithm prefix is added.
(08:25:33-08:26:04)
Future hash changes were expected to be easier inside Git because tests and
infrastructure had been abstracted away from hardcoded SHA-1 IDs. That
improvement does not remove the difficulty of migrating external tools,
dependencies, and references. (08:26:04-08:27:07)
The performance discussion also corrected the comparator: SHA-256 was
described as slower than plain SHA-1, but faster than SHA1DC with its
additional collision-detection work. This was a distinction made in the Q&A,
not a universal benchmark or a numerical performance guarantee.
(08:27:07-08:28:10)
Edited from the supplied English subtitles, following the session boundaries
in the accompanying summary. Rolling-caption duplication has been removed,
and obvious verbal filler and false starts have been trimmed; spelling and technical terminology
have been corrected using the summary, context, and selected video frames.
The talks and substantive Q&A remain in spoken order, including introductions,
puns, and meaningful reactions. Music, breaks, and routine logistics are omitted.
Timestamps are approximate offsets into the recording, not conference clock
times. Chapters begin before the listed talk starts where necessary to retain
introductions and jokes. Unidentified voices are left unattributed rather
than guessed. Square brackets mark editorial clarification or wording that
could not be recovered confidently. Speakers' claims and opinions remain
their own; this is not an audio-verified verbatim transcript.
[00:29:13] Hello. Hello. Hello. Welcome to Git Merge.
[00:29:19] I just like people clapping for me.
[00:29:21] Fits you well. So, for those of you who don't know, does anybody know what year the
first Git Merge was?
[00:29:29] 2013. So, it's been 13 years of Git Merges, and I'm excited to have everybody in
this, I think, the best venue we've ever done.
[00:29:38] Fantastic. Yeah.
[00:29:39] Taylor Blau is supposed to be here to help co-host with me. I think most of you
probably know Taylor. Unfortunately, he couldn't make it. So, I'd like everybody to give a round of
applause and thank you to Mini Taylor. To Caleb, yeah, a friend and fellow GitButler who will be
introducing everybody. Most of you were here for Jujutsu day yesterday.
[00:30:19] And we have an awesome afterparty that's sponsored by GitLab. We're all going to
move, sort of like a mother duck with all of my ducklings, over there so that nobody gets lost.
[00:32:12] We've got an excellent coffee station which weighs the beans, if that's what you're
into.
[00:32:19] So, a big round of applause to the Pierre Computer Company. And we've got East River
Source Control.
[00:32:33] It's Google. It's pronounced "ursk", I learned yesterday. So, if you see them around,
you're like:
[00:32:42] Google.
[00:32:47] And the most fantastic speakers' dinner, if you were there, was wonderful. Sponsored
by OpenAI/Cursor. So, check them out.
[00:32:58] Did you say Open? No. SpaceX.
[00:32:59] Did I say SpaceX? Right. Okay. [Laughter.]
[00:33:02] Speaking of SpaceX, if you're wondering why some of us are speaking a little bit
slowly and have drooped eyelids, it's because of the wonderful speaker dinner last night. So, that
being said, I'd like to introduce Tomas Reimers. Oh, no -- Reimers [correcting the pronunciation].
I'm saying it the German way. Our first speaker will be talking about the history of code review. So
let's give it up for Tomas.
The history of code review
[00:33:31] Amazing. Cool. Hi everyone. My name is Tomas. A quick background on myself, because
I'm speaking to a room full of people who work on source control: I am one of the founders of
Graphite, if that name means anything. We were acquired by Cursor and subsequently by SpaceX earlier
this year. We have been focused on building new interfaces for code review for quite some time. I
think we're going to spend a lot of today talking about what comes next and what we're building on.
But I was talking with Scott, and I think there's no better way to start that conversation than by
talking about what came before. It's really important to think about the history of what we're
building on top of, the tradition we're continuing, and the questions that we now see happening.
[00:34:21] I want to start with an observation that I think some people will resonate with and
some people will not: coding is easy. But the immediate thing I want to add is that building
software is hard. I think we've seen a lot of people embrace this idea that, with coding, I can now
build anything. I can go to my laptop, I can ask an AI agent to produce any fantastical thing and it
will create it. And where it becomes problematic is when I need to collaborate with others.
[00:34:54] And really, I think that's the challenge that we see at the heart of source control
is it's a way that developers collaborate on projects together. And that that is the foundation for
what we build. I want to talk around then the history of code review. I want to talk around why this
collaboration I think is breaking right now. And I want to leave you with three thoughts on what I
think must happen next. I will make a small pitch for what we are building at the end. I will start
out by saying that but that is not the focus of this talk and I want to give you more generalized
lessons that at least I am currently looking at a lot of our innovation through.
[00:35:30] So, first things first: how do teams collaborate today? People create pull requests.
Teammates review those pull requests. Individuals merge those pull requests. I want to start by
calling out that I'm talking primarily about closed-source teams. I know there's a lot that happens
in open source. I know that pull request and open source can mean something different. Often on
small teams of developers that collaborate with each other day in, day out, what we end up seeing is
I create a pull request, one of you reviews my pull request. Once I get one or more approvals, I
then go ahead and merge that pull request.
[00:36:02] What we may be less familiar with is where this process comes from. The earliest
recorded history we can find of this process actually starts with Michael Fagan back at IBM in 1976.
Back then he had this belief: hold on, before my developers contribute source code back to our one
shared concept, we should all proofread it. What he literally made his team do at the time was they
would take the code that they were writing, they would print it out, and then they would hand it to
each other, and they'd mark it up the way a grade school teacher might mark up an essay.
[00:36:41] And so, you'd go ahead, you'd print out the code, and then you'd read it. Because of
that, you ended up with a lot of the early advantages of code review, right? You started writing
code as if someone else might read it, because they certainly will. You start to increase bus
factor. You start to think: why did they do it this way as opposed to this other way? This process
quickly spread -- actually, I should say slowly and then quickly -- within IBM to different groups.
At some point, IBM did a study to ask: is this actually making our developers more productive? They
found that groups that did Fagan inspections, compared with groups that did not, had about 30% fewer
reverts. I thought that was a really interesting statistic, and some of the earliest telemetry we
have in this space of why we care about code review at all.
[00:37:32] And the practice then start to spread to the rest of the industry. The first
evolution in source control happens when we decided we probably should stop printing these things
and so we start to move them on to email. This is a practice I hope many of us are familiar with. I
think a lot of us contribute to open source itself. And the idea was, well, what if we were to take
that same idea of messages and quite literally email it to each other. And at the time the idea was,
well, I'll go ahead and email lines.
[00:38:03] I can't mark it up with pen anymore because that's just not a concept. But what I can
do is I can go ahead I can see some lines in a diff or a patch that I don't agree with. I can just
go ahead in my email editor click new line, new line, new line, be like, nit, why did you do it this
way? And then go ahead and send that back to you. And so from there, you start to develop this
concept of comments as we know it today. You start to develop this concept of approvers when people
had to formally respond to an email chain and be like, "Yeah, this looks good to me.
[00:38:33] Go ahead and merge it." you had to formally start to develop these these concepts
that that today are sort of like very well-known. From that point in a lot of teams and open source
frameworks still do code review this way for what it's worth. The next evolution to this one at
least the next one I'm going to talk around there's a lot of divergence here but I'm going to try
and stay linear for the sake of the talk and the audience. The next one comes in the early 2000s.
[00:39:04] Guido van Rossum -- yes, the creator of Python -- did this at Google as his intern
project, which is a crazy bananas idea to me. At the time they were using this piece of technology
they developed called Gmail to email patches back and forth. He had this idea: could we make Gmail
more friendly for the vast majority of emails that I read, which are code reviews? At the time there
were questions like: it would be really nice to have syntax highlighting -- a crazy concept we
haven't had so far. It'd be nice to click into the codebase, to be able to click to go to a
reference. It'd be nice if clicking Enter didn't simply feel like editing the code, but actually
created something new.
[00:39:43] It'd be nice to see the history of these things in a way that wasn't threaded
replies. And so he creates Mondrian. I have a potato quality video because that is all that exists
on YouTube of this. But you start to see the core ideas that now underlie modern code-review UIs.
Things like Critique at Google. Before that it was well it's gone through a few it's gone through a
few variants. Critique at Google, Phabricator at Facebook, GitHub, etc. Kind of all started to take
ideas that at least first for Git were found here and then started to started to expand outwards.
[00:40:22] And so that leads me to the final evolution, which was the proliferation of tools
we've seen making their way into the public tool chain and making this practice of code review
really a thing that we can have a conference around. I think before then it was actually quite odd.
It would the concept would have been quite niche or quite odd. And I think now nearly every
developer in the world uses this practice and wonders what what does code review mean? What does it
mean to be a good code reviewer? What why are we doing this or how did we get here?
[00:40:52] And I think what's so special about this moment today, right now, this year, is that
we're on a precipice again where we feel the technology must evolve yet again. For those of you who
have lived under a rock, software development has changed immensely in the past three years. It
feels like every day something new is happening. And there's a lot of cracks starting to show in our
current code review. In both on the source control layer but also on sort of like the interface, the
UI, the practice side of why why are we even doing this?
[00:41:22] I think the tough question that a lot of people ask right now is: with AI, do we even
need code review? Is this even a relevant practice anymore? Or now that we have the computer gods,
can we get rid of this practice entirely? The way I like to answer that question is I actually start
by looking at the numbers. This is the number of bot comments across a handful of open-source repos:
how much are automated bots commenting on pull requests? Before 2023, that number, I think, was
sub-percent. It was [exact figure unclear].
Validation, change management, and shared understanding
[00:41:56] And if you look at today, it's about 25% around the turn of the year, and I think
today we're at somewhere in the high thirties. That's a lot, just as as a concept to say a third of
all pull requests have a bot that's commenting on them is a wild statement. And it is the statement
that we have today. But I do not think it is I do not think that means code review is over. And I do
not think that is the end-all. I think it is the first reason we do code review.
[00:42:21] We do validation. When I started Graphite, I remember a question I had to contend
with a lot was what why do we do code review? And at the time it was a fun philosophy question and
now I think it's actually a very pragmatic real question that we talk around. It's a conversation
I'd love to talk with you about lunch. It's something I'm personally passionate around. What the
three reasons I have are one is validation. We want to check that if it's right going back to like
Fagan's original goal. It is the moral equivalent of you hand me a print out and I mark it up
looking for errors.
[00:42:49] The second reason, which I think is more interesting, is change management. I think
ignoring all of the proofreading benefits you get out of code review, I think an incredibly
important benefit you get out of code review is that if something goes wrong, we can revert a thing,
and there is one thing to revert. If you go ahead and send me a change that has 15 different things,
I'm going to ask you what's going on here. Why does it have 15 different things? If you send me a
pull request that looks massive, I'll tell you to break it up.
[00:43:17] And so the second one is actually a very practical sort of like DevOps kind of thing
where when we merged development and when we merged ops we needed to start to think more carefully
around what our changes look like. Tied into that is the concept of CI/CD and continuous delivery.
And the third one for me is I wrote alignment. What I originally wanted to write on this slide was
increasing bus factor because I think that's really what it is. It's if I write piece of software
that we are now mutually responsible for, I need someone who is not me to understand how that piece
of software works.
[00:43:46] And so those are the three reasons why I think today we do code review. Now AI has
really only solved the first one. It solve I think it has solved and arguably solved the validation
one. For people who work on complicated systems, you probably have not seen this at all. I think
that for a multitude of reasons, AI really high performance and high high complexity systems resist
vibe coding in an interesting way. That said, I like to think about it more for marketing sites. I
think about it for mobile apps. I think about it for the vast variety of glue code that that us and
our co-workers write.
[00:44:23] I think actually validation has largely been solved. I've seen AI call out everything
from the regex you're using to validate license plates is incorrect. That was a crazy one when I saw
that. I've also gone ahead and see, hey, like this function that you think is safe because you're
working in Python and it's untyped. Does not assume it can take nulls, but actually if you search
the codebase, you can find that this codebase might pass it nullish values. That's going to create a
bug. And it's crazy to me that we can see this level of reasoning, but we do.
[00:44:50] But that said, I am nowhere closer to solving the change management problem of, okay,
why are we making this change? Why did you make this change? What was it intended to do? And I'm no
closer to solving the alignment problem because I understand it. AI doesn't actually help my
co-workers understand it any better. And so where these two really break down what the reason why we
still need code review to me for these two really sums up to this. It's this still exists, right? I
can show this on the screen. Everyone immediately understands what this means.
[00:45:22] We're all horrified by this number. You're like, there's no chance in hell I'm going
to read that. And we look at it and it's and it's super real. And so I think the problem we have
right now is actually twofold. It's that we now have to do change management but we have to do it at
superhuman volume. Because what happened before statistics I did not include in the slide was change
change management has gone from I think in the old world on teams we saw that developers had to tend
to an average of four open pull requests at once.
[00:45:53] I think now that number is somewhere in the mid-twenties. That's literally how much
we've seen people expand. And so the challenge you have there is what used to happen. I would wake
up in the morning as a developer. I'd grab coffee and I'd sit down and I'd be like, "Okay, what are
my four pull requests? What do I have to do?" Right? And I'd be like, "This one is ready to merge.
Great. This one failed CI. Okay, I'll go ahead and fix it. This one the reviewers haven't responded
and I'd need to either go swing by their desk or ping them aggressively on Slack being like, "Please
respond to my messages.
[00:46:22] I need to get this merged. But the as we've expanded from like four to 20 pull
requests, I think the expansion here is actually quite quite significant. I think that is the I
think we've crossed the threshold where I can no longer be responsible for making sure that every
one of these is correct. The second thing that's happened is we have superhuman speed. When
previously when we had alignment there was a natural rate limiting factor which is you and I are not
working at the same time next to each other at least very infrequently are we and so what would
happen would be I would be like okay like maybe we should reconsider this and then I would go away
and it would take you a day to respond and then we'd come back and so there was this natural
debounce on the like rate of iteration what has happened now is I can be like hey like I actually
think that this entire architecture is messed up you should do instead and within seconds an AI can
come back and be like I've rewritten it for you.
[00:47:18] Okay, what am I supposed to do with that? Am I going to respond to this again? And so
we have two this slide to me summarizes the challenges that code review faces today as a workflow
which is we're trading we no longer have to do validation. I think we can largely be freed from that
responsibility, but we need to do alignment and change management much much faster and at much much
larger numbers than we ever have had to before. And so here's a graph of lines change per day by
developers. Again, same open source project.
[00:47:51] You can see we've kind of started to exponentiate since the rise of agents. I think
two things are happening to that. Sometime in 2023, we all got access to like or starting 2020, we
all got access to tab complete. 2023 tab complete got really good. 2024 we had agents. 2025 we're
starting to see background agents. Now I often find myself asking for features and developers don't
even write it themselves. They can go to an agent and they can be like yeah ask me for this feature.
Just go implement it and just spews out PRs that I have to review later.
[00:48:21] And so the ability for a single developer or agents masquerading as that developer to
write code has gone way the heck up. And this is scary. And so we're dealing with more code being
created, but that definitionally means that more code needs to be both reviewed and merged. And so
that is what is happening today. And we are now the bottleneck. And so that that leads me to sort of
the three conclusions that I'll leave you on as we think around going into the future of code
review, the workflow, the technology, the underlying source control.
What needs to change
[00:48:55] So thing number one, I think that the AI is going to need to be able to talk to the
forge directly. I mean this both as like an API problem of if I have to tend to 20 pull requests, I
don't think I can be responsible for it. I spend a lot of time watching people do code review. That
is part of my job in building the code review software is I will literally sit behind you kind of
breathe down your neck and watch you do it. And what I will see is is bananas to me.
[00:49:21] I'll watch people literally go to the forge. They'll be like, "Cool, this PR has bug.
This PR has comments." And then they'll open up Cursor and they'll be like, "Pr number 356 has bugs.
Please solve them." And this step that the human is doing, the meat proxying, this is not useful.
This is a waste of time and it's actually slowing us down. And if we are the bottleneck, we need to
get ourselves out of the bottleneck. And so part of the reason I think that doesn't happen is
because AI agents when they talk to be it GitHub or GitLab or whatever you are using don't have the
same privileges as humans. Both from an API perspective of being able to do all the same things that
the humans can do or being able to ingest the context in as efficiently a way as humans can or
literally as identity and identity and authorization that they can.
[00:50:05] I think one of the things that we are struggling with as as sort of like a developer
collective is when an AI agent writes code, does it write it as the agent or does it write it as you
or does it co-author it or is there a third door that we haven't even thought about yet? And so this
is challenge number one that we are faced with as we think around what comes next in the future of
code review. Challenge number two, our attention must be guided. I talked before around we had four
PRs, now I have 20 PRs.
[00:50:32] I think if I have 20 PRs, the idea that I'll the name I give to the workflow I see a
lot of developers doing is called gardening. That's how I'll describe it to my team. I think about
it a lot this way of I will every morning and usually in the morning and just after lunch you'll
watch developers go to their garden be like I have four flowers. Let me go check what those four
flowers need. And that that is actually the practical workflow you see in telemetry of what like how
developers do code review.
[00:50:59] They'll go check in on these PRs. Every so often someone else will ping them and be
like you need to look at this very specific PR but in terms of when you actually iterate through
them it's not as often as you think it's like twice a day and as that garden has expanded from four
to 20 flowers suddenly my attention needs to be guided I am the precious resource in the sense of
like before I was like ah I can kind of just do all of them now I am the world's slowest bottleneck
to these agents I need to find a way to get myself out and so when I show up to ideally the list of
pull requests and needs to tell me these are the ones that immediately need your the attention.
[00:51:34] Here are the ones I automatically took action on. These are the ones that immediately
need your attention. But then also for any individual pull request, I think some of that attention
needs to start to be guided. When I look at a pull request that when I look at a pull request that
is even call it 500 lines, I think there's work that could be happening ahead of that of these are
the important parts that I need you to think about. These are the key architectural decisions made.
Here's some context around the codebase that would help you get to decision faster.
[00:52:01] This is whatever else. This attention has to happen. And then the last one for me is
reviewing outputs is going to become as important as reviewing inputs. I think for a wide variety of
reasons, we're starting to watch sort of like code review and user acceptance testing start to merge
where we're getting to a point where a lot of code that we write so some code is clearly very
important. I think actually a lot around like Git itself very important very performance sensitive
creates a lot of downstream problems if it's wrong.
[00:52:31] A lot of code we write is glue code. I think around code for internal tools. I think
around code that does the my like obvious one is like CSS on marketing sites where so long as the
marketing site looks good, I actually don't really care that much about the quality of the code
because it's going to live for like 3 months. So, who who really cares? I don't need to overengineer
it. And I think in those cases, it's not clear to me that the right artifact to review is code. I
really do wonder if the thing we want to start reviewing is do we want to start reviewing
screenshots?
[00:53:01] Do we want to start reviewing videos? Do we want to start reviewing terminal output?
Especially as we think around utility functions that we're writing or CLIs that might be internal
and I think that right now that doesn't really have a place in our forge sometimes sometimes we see
really good citizens be like I've attached a screenshot of the change I made to the description but
this is infrequent and I think that as we continue to move forward this will have to become this
will have to become more more commonplace this is what we're working on with Origin Origin is what
Graphite has transformed to since we were acquired by Cursor our goal is to build the familiar Git
Forge.
[00:53:35] We have it released. I promised I'd keep this pitch short. And so I'll I'll leave it
there. But the takeaway I will leave you with is when you look at sort of the history of code review
or at least the linear one we walked through of sort of Fagan to email to Mondrian to patches, none
of these have been cataclysmic breaking changes. They've all been deeply rooted in the past of
here's the workflow that developers were familiar with and here's the small set of edits we're going
to make to bring it into the new generation.
[00:54:06] And I suspect that as we look forward to sort of AI code review or AI changes, it
will look very very familiar to us. It will feel fundamentally different and the work it will reduce
will be enormous. But I think that it will first and foremost look very familiar. Anyway, that's
what I leave you with. I'll pass on here to I don't know who comes next. Scott, do I pass it back to
you or
[00:54:29] You can do questions for a bit?
[00:54:31] Great.
[00:54:35] Let's do questions for a bit and then I will pass on to the next speaker. I'm going
to start white shirt here. I'm happy to repeat the question and then
[00:54:45] Or there's Mike. Beautiful.
Q&A
[00:54:48] Yeah. Yeah. Hi there. Thank you. So you mentioned the three main reasons why code
review existed in the first place. There's validation alignment and then
[00:54:58] Change management
[00:54:58] Change management right
[00:55:00] So okay fine validation is a problem that's solved change management is that like
something that's at the review surface or is something do you think that belongs more at like at the
CI layer or the integration layer because I'm a bit confused how like code review is a surface where
you know the atomicity of where you land changes and have the ability to revert those changes are
relevant And on the alignment piece if the again should the alignment happen while you have already
reached the code state or should it be more alignment on the design step maybe even the prompts and
the agent trajectory is that the thing that is reviewed and that helps set the alignment so that
when it gets to the code actual coding stage it's your agent adding the code reviewer's agent
reviewing it and then ensuring that it conforms to the spec that was dictated by the prompt and the
trajectory.
[00:55:54] Oh, wonderful questions. I love them both. I'm going to answer them separately. So,
question number one, I think yeah, question number one is should change management really be
happening at the code review UI layer. I want to clarify when I talk around code review here, I mean
more the code review workflow, the practice of it, the concept that we have code review at all. I
think that code review is often what enforces change management hygiene. I think that when you see a
lot of junior developers or open source, let's say less invested open source developers come to you,
one of the things the hallmarks of it will be that 100,000 line PR and I think code review is where
you say absolutely not.
[00:56:29] This is not what's going to get merged. And so that's part of why I think around it
as the workflow. I think that without code review, you would maybe be able to automate it. I think
it's actually been quite resistant to automation because sometimes in 100,000 line PR is correct. I
mean that sincerely. I remember I when I was at Meta and I see two Meta source control engineers in
the front. I remember stamping million line changes sometimes when we had to do large codebase
refactors because we renamed a function, and that is semantically one one change.
[00:57:00] It probably should be one change. I have no concerns with it being one change. Most
million-line PRs I don't even bother to look at. The second question which I think is more
interesting is alignment. Should alignment happen before or after code review? I've spent a lot of
time thinking about this because I think it's a really good question of is code review the right
place for alignment to happen at all. I think the answer is yes. And my reason why is very specific.
When I'll actually draw an analogy to a part of product development which is my designer will hand
me pixel perfect mocks and that's wonderful and I will implement them and they will still review
what I implement.
[00:57:36] And I like to think it's not because I'm a bad engineer. I like to think it's because
language isn't perfect. In the sense of despite how perfect their mocks were, they failed to specify
transition states. They failed to specify error cases. They failed to specify whatever else. And so
review is really where we get to be on the same page of you said one thing and I gave you another,
are these close enough? And I think that that will probably always be true in some limit of so long
as the prompt I give you cannot so long as the prompt I give you is not super specific and the only
super specific language that we have developed as a society is code.
[00:58:10] And so so long as I'm not giving you code, we will need to have a review step later
to make sure that the way that you interpreted the ambiguities latent in my statement are aligned
with the way I wanted you to interpret them. And so that's what I think around alignment there. But
more than alignment, it's more knowledge sharing more than anything, right? It's the I built X what
here here's how it works. Here's how I implemented it. And again, I think it's I think so long as
humans are not just English to code compilers, you will need that step because there will be natural
ambiguities and we can agree by the way we should probably implement a database of like this thing.
[00:58:49] What was the format of that database, right? What kind of query patterns did you
optimize for? Which ones did you not optimize for? Why did you make those decisions? That's the kind
of stuff that needs to be in your head so that when I go on vacation, I don't get paged when it goes
down. Right. Cool. Blue shirt.
[00:59:06] Yeah. Thanks, Tomas. Really good talk. I have a really quick question. I agree that
not all code is created equal. So, some would require code review and for some just reviewing the
artifact is enough. Where do you think is the right place to record this information? Is it in the
code? Is it in the Phabricator project or do you have a guidance on that as where do we record that
information?
[00:59:28] Now I get to go off script entirely. Wonderful question. The greatest tragedy of Git
to me is that we forgot to build code review into it. I will say that as a statement. I hope someone
here is working on that. I am I think Scott might be so we'll wait for his talk later. I think that
there is where should it be recorded alongside the code and fortunately we have this very robust key
value store that we all happen to use called Git that could record this knowledge.
[00:59:53] I think the challenge we have is how do we standardize that technology? I think that
there are I think code review in general should happen to be there. I think because it is not baked
into Git we all rely on forges to store this metadata alongside our Git repository and share this
data. That's fine. Ideally, we could standardize this into one centralized protocol. We don't have
one yet. For now, my answer is wherever your Forge chooses to put it. When you really think around
sort of like what is the job of the software forge, it is to allow you to share state with your
collaborators.
[01:00:27] That is the job. They happen to mostly speak the same language called Git. They
largely attach metadata onto that Git because they found Git insufficient in whatever ways. Over
time I would hope that most of that can flow back into Git such that we have one established
protocol that we can all share
[01:00:48] Questions.
[01:00:50] Yes.
[01:00:50] U first of all a big fan of Graphite. Just
[01:00:55] Something I was curious about though was during the acquisition, was there a friction
in like
[01:01:00] Graphite's philosophy on code review versus Cursor or xAI's philosophy on code
review? And kind of following up on that question, what do you think about like the long-term kind
of evolution of stacked PRs because I thought that was probably, Graphite's most prominent feature.
[01:01:18] It was certainly one of the reasons we came into existence. Having so for those of
you that didn't know, Graphite was a company that built code review interfaces on top of GitHub. Our
long-term aspiration was always to build our own forge. It was hard for us to get there as quickly
as we could have post being acquired. First question, were there differences in philosophy between
code review cultures? Yes, in so far as any two organizations would have differences in philosophy
at code review cultures. I'd say that there were differences between Meta and Graphite also when I
left.
[01:01:48] And so, yes, nothing insurmountable. I think the way we think around it is, changes.
I'd also call out that within an org, there's usually differences in philosophy. The way that the
infrastructure team looks at, I always point at infrastructure and product because they are two
extremes of a spectrum to me. The way that infrastructure teams look at code review is quite
different than when product teams look at it. When I've spent time on infrastructure teams, the kind
of culture I have is like actually the most extreme version I had was a team which won't be named
where the belief was the approver of the PR that causes the SEV is the one who has to present the
postmortem at SEV review.
[01:02:21] Which I thought was a great culture honestly it led to like when you approve
something you're like am I ready to present the postmortem on this? And I thought that was a really
phenomenal culture. Product teams mostly like cover their eyes and click approve because they're
under tight deadlines. I say that somewhat jokingly, not really that jokingly. And so, differences
in culture, yes, if you're interested in the specifics, I'd love to talk with you about them at
lunch. I think they're interesting because they're illustrative. Stacking, I think stacking is
becoming popular, thank god.
[01:02:51] So when we started Graphite, the origin story for us actually was we I left Facebook
with my two co-founders. We started our own company, we actually did nothing related to source
control at all. We found ourselves rapidly missing Facebook's internal code review tool code review
tools Phabricator. We tried to stand up open source Phabricator. There is a far cry between open
source Phabricator and Facebook's version of Phabricator for those of you who have ever looked into
that. We then tried to stand up Gerrit. We really struggled with the ecosystem where all of the
modern tools we want to plug in didn't.
[01:03:21] And so then we started to hack in hack in sort of like the functionality we wanted on
top of GitHub. At the time we were kind of crazy. I remember we would tell people like no we have
all these great workflow improvements. Probably the one that caught on most of all of them was
stacking. I'd say there was a lot in there generally helping monorepos a lot which we still want to
bring to the industry. I think stacking is finally becoming popular. GitHub finally has support for
it.
[01:03:47] JJ I think has done more on stacking than it has done so much on stacking that we
wanted to do that we never got a chance to. I think that there's a like wide proliferation of tools.
And I think most importantly, when I say stacked PRs to a room full of people, you all nod and know
what I'm saying. Whereas before I would say stacked PRs and you would look at me like, what you mean
like a branch? And I'd be like, "No, not like a branch. Kind of like a branch.
[01:04:12] It's only a branch because GitHub decided to make branches a unit of PRs." they
should have been commits. But I think they're actually becoming popular. For now, we are committed
to supporting it. We will continue to support Graphite for the long haul. Our goal is to try and
make that a ubiquitous practice because I think it is simply a better way for development. And when
it gets to change management, I think so many of us believe in feature branches. The idea of many
features smooshed into one branch as a way to speed us up because the tooling is bad, not because we
actually want that workflow.
[01:04:44] Thank you. Of course, question. I wanted to ask is validation actually solved and
maybe you'll elaborate that the things I'm going to bring up are the other two.
[01:04:54] I'll start with no, but I would love to hear I'd love to hear.
[01:04:57] It's like it the agents are very good at telling me here's an edge case you didn't
consider from like 17 files transitively away but not good at telling me here's a better data
structure or data model that will like cut out the entire class of issue. So like
[01:05:15] Is yeah that's is that alignment is that something else what how do you think about
that
[01:05:19] Perhaps I've drank the Kool-Aid so validation is not solved I think I see the path
towards solving it I really have seen as as I keep looking at the model jumps between what was here
two years ago a year ago now three month what will be here in three months I really think that
validation is getting solved. I think that as you start to look at things like Mythos and their
ability to like really find flaws in your code or to adversarially attack or interrogate it, it's
really really good. And I think what gets left are cases in validation which are really masquerading
as alignment issues like oh yes this data structure would solve it because it doesn't handle this
generalized case but turns out we never have that generalized case because of this other business
context that isn't encoded anywhere.
[01:06:00] I think that that is really I think that validation will get solved. It is not fully
solved and I think that AI is part of the story but I think that between AI and CI we actually are
starting to get a wrangle on something that was previously intractable. Second question.
[01:06:15] Yeah. Very briefly is like so in your mind things like API taste is alignment rather
than validation because that's the thing that's really for me lacking.
[01:06:24] Validation to me don't see on a path to being solved yet.
[01:06:26] Yeah. For me validation is the how as in like does it work? Does it more than does it
work? Does it do the thing it's purported to do? Alignment is what are we building? So how are we
building? What are we building? And what is a question of like are we building the right thing? Do
people want this thing? Is there a different thing that people would want more? That's kind of how I
break up the two in my mind. Of course.
[01:06:51] Yes. Now that validation or like the code review is moving a little bit away from
code I see a trend of like that the preview environment or like the environment in which to test
your code is becoming more important. How do you think that how does this relate to the
responsibility of a forge?
[01:07:11] For everyone on the live stream there was just wild applause. Everyone agreed with
that statement. I am so glad we're all aligned on that. Yes I believe that this is going to be true
for quite some time. I find myself often sharing the social network meme of "Drop the 'the.'
Facebook is cleaner". I often say "Drop the 'code.' Review is cleaner". I think we are moving to a
world where review is the important thing. I agree that preview environments are key. I think
preview environments are actually key for a few reasons.
[01:07:36] So previously they were key for a whole bunch of reasons around like it's great that
our developers have a standardized environment and that we don't have to clone our codebases and
that they can just be on and they can parallelize development. I think Google's Cider here is
absolutely incredible and really good at what it does. I don't really think we've gotten a tool like
this in the public tool chain yet and I would really like us to. The challenge you have is that from
a to take a developer hat on and put a business hat on.
[01:08:08] The challenge you have is that no one in the organization feels like an acute
business need around this. You're like and what if like developer onboarding time were slightly
faster and people are like at a seems like a lot of pain to maintain this. I think it is now
becoming more of an issue actually because AI agents need to code and I think that preview
environments happen to be the best way to say like, oh, I need this agent to be able to add this
feature and obviously it's going to need to be able to validate that feature and so it should do
that, right?
[01:08:32] And so that's making it more powerful. And as AI agents start to be able to do that
because they need to write code, I think they're starting to be able to do that in order to validate
code. One of the personal automations I have set up at Cursor is on any PR which makes a user-facing
change run the app run an instance of the app before and after and take a video of that change and
suddenly all of my PRs I'm wasting a bajillion tokens on this but all of my PRs only have a
screenshot of before and after and it has made review a lot easier.
[01:09:03] I think something like that will become more common and the obvious improvement being
and run the website if it's a website. The only reason I call it the reason I think that preview
environments may not just be websites is because we also have a large variety of other software
terminal scripts, mobile apps, desktop apps, all sorts of things which probably need something else.
The best solution I've seen here is is being able to remote desktop into your agents machine. And
with computer use agents, I think it's actually becoming quite viable.
[01:09:32] And I don't know if I care between opening the like I used to be very skeptical of
that. Now, having done it a few times, I don't know if I care between opening DevTools on my local
version of Chrome or if I have a perfect like remote desktop connection, opening DevTools there, it
doesn't really matter to me. And so there there's something interesting there that's happening. Yes,
I think preview environments are certainly going to become a thing. I think they'll become a thing
more for agents than for validation. They need it to code.
[01:09:58] But I'm excited either way. There will certainly be a tie into code review.
[01:10:05] Cool. One last question, Scott. Or Great. One last question.
[01:10:08] So I wanted to say if you have any thoughts on code review of the existing codebase.
Most of us work in large codebases with millions and millions of lines of code and as we sort of
adopt stacked diffs more and more each individual diff gets smaller and smaller and the codebase is
rather built up over time as this sediment and I think we can all agree that our codebase is
probably not the optimal architecture. It could be improved sort of how do you work with reviewing
the existing codebase over time rather than just each sort of diff at the point in time it's added.
[01:10:48] That is a wonderful question and one I've not been asked before. So kudos. I don't
know. I think this is a cultural issue more than it is a workflow one in that I think as companies
we are often so focused on the new that we don't really go back to maintain or consider the old
unless there are immediate problems coming from it. I think we even have engineering adages around
it of like it's not broken don't touch it who knows if it's load-bearing and I suspect that we will
start to fix that actually again by the way of agents I suspect the place where I'm seeing this most
fixed right now is security where we do live in a world where actually overnight multiple things
scan my codebase find security vulnerabilities fix and either propose fixes or tell me that it needs
to be fixed.
[01:11:36] And so I think that is becoming quite a concept. If you were to tell me that we now
start to do the same thing with like performance or general architecture, I would start to believe
you. I think security will probably chart the way here just because there's so much economic value
in unlocking that first. But we will see what the future holds. Agreed. This is an open problem. I
think it is distinct from code review and I would not muddy the interface to try and do both at
once. At least not yet.
[01:12:02] Cool. Passing it back to Scott.
[01:12:05] Amazing. Thank you all. [Applause.] Do I take this?
[01:12:12] Also, amazing sense of fashion. I feel like I don't know about that. All right. So,
our next speaker is brian m. carlson. So, I think a lot of you also know brian. He's spoken at a lot
of Git Merge conferences. He is a core contributor and has been for a very long time. He has been
working on the SHA-256 migration stuff. So, for those of you that know the incredible pain in the
ass that that is, when it gets done, you largely have brian to thank for a lot of that work. And he
will be talking about Rust in Git.
[01:12:43] So, as he gets set up, please give a nice round of applause for brian. [Applause.]
[01:13:56] Everything's kind of set up to go beforehand. In the meantime, would you guys like
some dad jokes?
[01:14:03] All right. What did Spartacus say when a lion ate his wife? Nothing. He was glad he
ate her. You're welcome. Let's see.
[01:14:17] Okay.
[01:14:18] All right. Oh, are we ready to go?
[01:14:20] All right. You guys are saved from more, for now.
Why Rust in Git?
[01:14:27] Okay. I'm brian m. carlson and this is Rust in Git: Status, Internals, and Features.
So let's talk about why we want Rust in Git. So Rust provides memory safety as a lot of you know and
a decent number of Git security issues are memory safety issues. We have a variety of buffer
overflows and pointer arithmetic problems and you know just if you've seen the recent security we've
had and with an increase in AI looking for vulnerabilities this tends to be a problem. And Rust is
easier to write correctly than C as well.
[01:15:06] We have you know I've been writing C since I was 10. It's still easy to forget to
free memory. It's still easy to forget to you know you can still overrun a buffer. So the compiler
is going to catch a lot of these memory safety and thread safety issues that you would normally have
before you before you run your code and it's easier to test for them. So Rust also has good FFI with
C. So if you're if we want to build things incrementally, we can port over just the pieces we want
to build and continue to use the C for the rest of things.
[01:15:44] So we're not required to do a full rewrite at once because that's never we all know
that's never going to happen. And Rust is a really popular language. So we can bring in new
contributors that way and we can have them start off pretty easily writing better quality code since
a lot of people find C to be rather difficult. I know there are some people who really like C, but
it is it can be a little difficult. So, one of the other things that this brings to our codebase is
it brings sensible iteration.
[01:16:15] Git provides multiple different methods of iteration. It provides macros for each
functions and a variety of other options. And so, depending on the data structure, you need to look
for the proper iteration tool. And sometimes the one that you have is not the one that you want. In
Rust, we have a sensible way to do that and the compiler compiles that down to an efficient form of
execution. We have better data structures. There are maps and sets in the built-in language. So we
don't have to implement those ourselves. We have our own hashmap in Git.
[01:16:52] We don't really need to have that. When I worked I work at GitHub and we have we
replaced some of our C code with Rust code and we saved a huge amount of memory with one of our
hashmaps because the built-in Rust data structure used memory more efficiently than our custom data
structure. We also have to initialize all of our data in Rust. You can't rely on zero initialization
and zero initialization causes a lot of subtle dependencies on globals in our C code. For example,
struct object_id has an algorithm member which is an integer.
[01:17:31] And if that is zero, then when we compare object IDs, we depend on the global
constant the_hash_algo, which is a dependency on the repository. And that's basically that
basically limits us to one repository in the codebase at a time that we don't really want to have a
dependency on. We also have less dependency on globals in general because mutable static variables
are unsafe in Rust. So we don't use them in our Rust code and this helps achieve our goal of
libification. So we have some potential pitfalls that with adopting Rust and Git and the one of the
major ones is that it's not as portable as C.
Portability and compiler support
[01:18:18] We have systems like Linux ia64 or alpha despite the fact that those are no longer
sold. They're still very popular in some segments and distros like Gentoo support them. So there's
also NonStop which is an OS on Hewlett Packard Enterprise hardware and Git runs there but it does
not have Rust and Rust effectively requires LLVM and not everybody wants to port LLVM to their
platform. Another problem is that Rust has a lifetime of six weeks. So we're not going to only
target the latest stable release, but there are some projects which do.
[01:19:02] And so that's a potential problem is that we maybe want our software to have a
lifetime of, maybe more than six weeks. People don't want to update Git every every couple of weeks.
The good news is that there are some possible portability improvements we can consider. So there are
alternative compilers. There's gccrs, which is an in progress Rust frontend for GCC. And it's
currently targeting Rust 1.49 which was released in summer 2020. I think we're up to 1.98.1 right
now upstream and it's a potential candidate for supporting other platforms. mrustc is a Rust
compiler written in C++ and it was originally written to compile Rust on new platforms because in
order to compile each version of Rust you must compile the previous version and so bootstrapping a
new platform can be difficult.
[01:19:59] With mrustc you can build Rust from the from just a C++ compiler. So I'm exploring
the possibility of using mrustc to compile code. It still needs some patches to support static
libraries which we use for our Rust code and Git and mrustc also has a very limited Cargo version.
So we're going to need to do some fiddling there. We'll also need to support updates to libcore
and libstd for the new platforms but it removes the dependency on LLVM. The problem there is that
upstream Rust doesn't carry library support for third party compilers.
[01:20:40] So this may require distros to carry portability patches. However, this that's better
than not having Git at all. So let's talk about adoption. So right now we're targeting Rust versions
in Git that are using 1.49 to allow for gccrs. So my proposal is going to be the version in Debian
stable plus when a new stable comes out one year of oldstable. So when for Debian 14 comes out, we
support Debian 13's compiler for one more year. That's effectively 3 years of support for a
compiler, which is pretty decent for modern software.
[01:21:21] And many distros offer a more up-to-date version for compiling Chromium and Firefox,
which now require Rust. So that most operating systems will be able to support that kind of version.
But many crates aggressively bump versions and some of them only support the latest stable Rust. So
we're going to need to be very careful about dependencies. Where we are now is that in 2.52 Rust is
auto detected by Meson and it's disabled by default in the Makefile. So depending on how you build
it will either be auto detected and auto used or not used. In 2.55 it's enabled by default.
Adoption and code already upstream
[01:22:03] So unless you turn it off, you will compile with Rust and the build will fail if you
don't have Rust. In Git 3.0, which we expect to come out relatively soon, it will be mandatory and
will require that for all new versions. The last version before Git 3.0 will be an LTS release to
support non- Rust using systems. So let's talk about where the code is now. So we have some code
already upstream. The first piece of code was varint which is our variable length integer parser
and thanks to Patrick Steinhardt that was the first piece of code ported.
[01:22:41] We have a hash algorithm abstraction. So you can choose a hash algorithm and you can
look up things like the size in bytes or hex characters. You can look up the empty blob or empty
tree ID for that algorithm and you can hash with that algorithm. There's also an object ID structure
which matches the one in C so it can be passed between the two algorithm between the two codebases
and there's a checksum-file implementation when you have a pack an index or one of the
interoperability object maps they all end with a hash and there's special code to write files and
hash them at the same time and then fsync them at the end and this implementation is also
supported in Rust one of the things that I've written as part of the interoperability work is an
error type.
Interoperability and the C/Rust boundary
[01:23:29] So, Rust returns a result in a lot of cases, which has either an Ok, which is the
successful thing with a T inside of it, whatever your T is, your successful state, or an Err with
an E, some sort of error state inside of it. And we want to be able to expose that in C. So,
there's an error type that can be converted between C and Rust. And the C version is two 64-bit
words. It's exactly 128 bits in size because most APIs support returning two registers to support
the div function. So that it those are an error code which is zero for success plus some flags such
as like did we allocate for the pointer integer value or did we allocate in C or in Rust or is this
a special errno value and then we have that pointer or integer value that's specific to the kind
of data we're handling.
[01:24:26] The benefit to this is that we can also format our error codes in a localized way
because we can only really do that in C due to limitations in Rust and support for gettext. So by
putting that localization in C, we have the benefit of maintaining a localized codebase which Rust
doesn't support natively. Git also has an internal string buffer type called struct strbuf. And
this basically is a byte array with a NUL termination. It's always NUL-terminated. And so we use
this a lot for moving data for serializing data between different parts of the codebase.
[01:25:12] And so this the Rust code can now use that for interoperability with C or it can use
a Vec<u8> which is this typical non-NUL-terminated buffer as appropriate. And this uses the Rust
Write trait. So basically anytime you want to format text or you want to write bytes into a buffer
you can use the Write trait. And so all of the native Rust formatting works with strbuf. It just
magically works. So we don't need a lot of the C formatting functions with the var like printf
-style functions that we have in C because we can just use the native Rust ones.
[01:25:48] There's also a lot of code that's already written in the SHA-256 interoperability
work that uses Rust. The object maps are used for storing maps between the different algorithms in
for loose objects and submodules and shallow points. And this is important because normally for
packs we store that in the pack index, but obviously we can't do that for loose objects or
submodules because they're not in the pack. So this object map approach also allows us to build a
trait API that says this is the API that we use for mapping objects and we can do different things
with that which also allows us to do in-memory testing that we don't have to create a repository. We
can just test our code more in a unit style unit test style.
[01:26:36] And we also have a parser for trees. So you can actually parse tree objects in Rust
and iterate over them. We have object file conversion which was originally written in C but it used
a lot of pointer arithmetic and with the interoperability work that is now done when you when you
pull over the protocol. So it becomes a protocol sensitive security sensitive boundary that we need
to deal with and I was uncomfortable with all of the weird pointer arithmetic we were doing. So that
is now written in Rust and as part of doing that we need to support parsing header based objects
like tags and commits.
[01:27:11] For example, a commit will have tree and then object ID. It'll have parent and then a
parent. And parsing those formats it can is now done in Rust as well. And we use zero copy
operations when possible. So whenever so if as long as we're not actually parsing out a signature
which is a multi-line context we can continue to parse it without actually allocating any memory.
And as part of that we also need to parse signatures to map them between algorithms and parsing
signatures is can now also be done in Rust. The final things that we have that are already written
are hex object ID parsing and serialization.
[01:27:51] So we can now parse and serialize hex object IDs. For object ids, we just use a
Display trait. So anywhere you just format it, it magically turns it in text value which is super
easy. And in it's traditional for me to write code the day before Git Merge. So I have started
writing some of the base85 code and which is used in base in binary diffs and that should also be in
Rust relatively soon. So things I want to support in the future. Here we have index parsing and
serialization and packfile parsing and serialization.
Future work
[01:28:30] And these are both security sensitive code because they're used when you're cloning
or fetching. And so they are a source of we get a decent number of security reports around these
kinds of things. Not all of them are valid but at least in Git security model but we'd also like to
have fewer reports of this nature and if we do less kind of suspicious behavior and have more
dependable behavior that will really help us. I also have intention to build a tool for hash
algorithm migration. So as we move from as we take a repository in one format and move it to another
that will that tool will be written primarily in Rust which is why I've written a lot of the tree
and header parsing and that place will also that will basically allow us to move from one algorithm
or multiple algorithms and then I have some intention to support incremental tree parsing with a
kind of with a kind of a binary search in future approaches to deal with very large trees and that
is one of the things that I'd like to write in Rust because I think it will be much faster.
[01:29:48] So that's mostly it but I do have one more thing. The slides here are available at
this URL. I wanted to point out that this is the SHA-256 repository. So GitHub now has that in
private preview for those of you who are interested. It is not publicly available so you can't
create new SHA-256 repositories. I'm seeing some reaction from the audience here but it's coming and
we're planning to ship it soon. So that is hopefully good news for everybody and that's it. Thank
you very much.
Q&A
[01:30:31] Are there any questions? Yeah, sorry. So you mentioned there's support for like
various exotic architectures or kind of less frequently used architectures and OSes which kind of
made this challenging and so I'm curious like how would you describe like was that support
originally intentional or it just so happened to work because Git was using C and like the distros
they supported C and so the C compiler it supported targeting. So, was that support accidental or
was that like a deliberate choice? And how do you think about that now?
[01:31:11] I think Git has long been since since well before probably early on in the life of
Git, we chose intentionally support of wide variety of systems, especially with a port to Windows.
So we're looking at so they it was kind of intentional to support a wide variety of systems but the
mostly it's the fact that most compiler most systems have a compiler written C. So when you can
compile C code you can compile pretty much anything. The problem is that Rust most of the compilers
that we have for like especially Linux systems and other Unix systems like the BSDs are GCC
traditionally and so LLVM does not support the number of architectures that GCC does and that's why
Rust doesn't support them.
[01:32:03] So we do hopefully want to get to the point where we can compile on those. It's a
priority for me to see if we can find some alternative but ultimately it's up to porters to decide
whether they are willing to maintain LLVM for their platform and we can't support every platform on
the planet without somebody's assistance.
[01:32:26] Yeah, thanks brian. Really good talk. For are you looking for contributors? I'm
particularly interested in index packing and serialization. Are you looking for contributors or
there's someone already working on that I can speak to.
[01:32:39] There's not already somebody working on that. It's something I will pick up
eventually, but if there are people who want to start porting that into Rust, that certainly would
be welcome. There's a variety of code that could be sent upstream and if someone wants to send
patches to help help work on that, I'm sure it'll be very very gracefully accepted.
[01:32:58] Cool. Thank you. I think that right at the start though like the upstreaming of Rust
will be slower probably compared to everything else also because like the Git community also needs
to come up to speed like we don't have a lot of Rust reviewers yet and so we will also have to build
up capacity over there so it will be a little bit more bumpy along the road at least initially I
think until we have picked up steam
[01:33:23] I tend to agree I think for a lot of people who have written C Rust will feel
familiar because of the memory management and lifetime handling, but it still can be a little bit
tricky to come up to speed. I know even though I've written C for many years, fighting with a borrow
checker is a time-honored tradition for learning Rust. So it's a thing that people just have to kind
of get familiar with.
[01:33:50] You talked about Rust's potential to bring new contributors into the project. Because
a lot of people like you said are more comfortable with Rust than in C. Which I thought was amazing.
Are there areas of the project do you think that like where like external contributors are most
likely to be interested in contributing to and is there discussion about like moving some of those
areas to Rust? I think a lot of the there's a lot of work these days on performance and performance
improvements and that's not a great place for new contributors to start because it's very it's kind
of tricky and so but I think a lot of it there's a lot of cases where we could build a lot of the
new a lot of new somebody came in and for example just added some more git var options to help
provide information that we didn't have before and so So in a lot of cases that code could partially
move to Rust and then be updated.
[01:34:47] There's a lot of places where people build small new features that could actually be
mostly built in Rust and then called from the existing C code to wire them up. So I think we have a
lot of like one-off contributors that do small incremental pieces that are really valuable and could
be done very well in Rust. I don't think a lot of the existing systems being ported are super ideal
for new contributors just because they're very complex. Like when I was working on the
interoperability work, I learned a lot about the pack format and wrote a lot of documentation for it
and I've been working on Git for probably 10 years now.
[01:35:24] So it's still a learning skill. There's still a lot of learning for me in that area.
So I think it's really up to kind of us to encourage people to build new things in Rust that are
coming along.
[01:35:41] Yeah, I agree with that. Like building new stuff in Rust is probably going to be the
easiest way, but the second easiest way is also to like have a look at sub subsystems in Git itself
that are already like fully self-contained and that don't have a lot of external dependencies. Like
for example, the reference subsystems are fully pluggable nowadays. So those could potentially be
areas where you could already convert those into Rust in standalone units. And once object databases
are also going to be like pluggable, then those could also be like a good subsystem to like convert
to Rust itself because like they're also going to become more self-contained going forward.
[01:36:27] I was curious we most of us probably saw the Bun project rewrite the entire codebase
to Rust in a matter of weeks or months. I was curious one is it your ambition to rewrite the entire
codebase into Rust over time and two to what degree are you personally and the Git project
leveraging agents to do this porting? So I my goal is that most of the code will be written in Rust.
We have a lot of portable C that for whatever reasons needs to paper the POSIX API on various
platforms including on Windows and so those pieces are probably going to stay in C.
[01:37:11] My goal is that most of the main pieces of the code will live in Rust but that will
be a long time coming. Git does not allow most AI contributions. So we're not using agents for that
purpose. The concern is about copyright provenance and the ability to complete the Developer
Certificate of Origin. So we're not using AI agents to do that. But I think in a lot of cases some
people are definitely using AI for things like code review or like checking the correctness of code
and that should be in compliance with the our policy.
[01:37:50] Thank you.
[01:37:59] Is the code that you're writing here is it packageable like into crates or libraries
that can be used externally like outside of Git? I think at this point there's a lot of there's not
that much code and it could theoretically be packaged into a library for a lot of that purpose. We'd
have to probably split some of it out, but we could certainly do a lot of that. I think right now
it's not super useful because libgit.a, which is the internal static library that we use for C and
the Rust library depend on each other.
[01:38:29] So for example, the hash algorithm abstraction to handle the hashing calls into C and
some of the C code calls into Rust. So it's not super useful right now because it depends they
depend on each other. But I'm hoping that in the future there could be enough self-contained pieces.
I think we're going to keep for example SHA1DC with the collision detecting SHA-1 implementation
which as far as I know is written C. So we're going to still have some of those dependencies but
they could theoretically be packaged separately. Hey brian, thanks so much for all this work.
[01:39:04] It's very takes a long time is very hard to do. And I've been a fan watching this
happened from the outside. So my question is as part of that with the kernel adopting Rust it was
let's say famously a little contentious to happen and it seems like it was a lot less contentious in
Git and I'm curious what social difficulties right this is as much a social problem as it a
technical problem and so you talked a lot about the technical side but I'm curious how this went
socially to get adopted upstream
[01:39:31] Well I think originally we discussed the desire to adopt it at the contributor summit
in 2024 in Berlin and there was a widespread a lot of people were in favor of it. I don't know that
anyone's really against it. I think the concerns were about portability. We've had concerns from
like the NonStop developers.
[01:39:50] Yeah. So I think the major thing was that we recently had somebody contribute who
decided to handle the xdiff code that we use for diffs in Rust and that didn't get adopted as is
but it got the discussion happening and so we started to move towards that adoption. Now I'm not the
only person who's been pushing for Rust and Git. There's been a lot of people, but I think a lot of
there's been that was this impetus we needed to get there and to start having those discussions and
because we knew Git 3.0 was coming up, that was part of the adoption.
[01:40:25] There's going to be some definitely some tricky pieces. I think Junio was not as
familiar with Rust as maybe some of us are. But I think there's an agreement that this will be
valuable for the project. And I think anybody who's been on the security list recently has seen,
anyone who's doing security for an open source project has just seen a huge increase in security
issues. And it's not that some of them, again, some of them are valid, some of them are not. There's
probably a lot of them don't fit our security model that involve like, working with untrusted
working trees that we don't support.
[01:40:57] But I think as much as possible if we can eliminate those classes of vulnerabilities
there's a huge appetite to do that from the core contributors.
[01:41:13] The exotic platforms that you mentioned this ia64 and the NonStop and so on. Are they
actually upgrading the Git versions? So, do they actually care to build the latest versions of Git?
[01:41:28] As far as I know, Gentoo supports both alpha and ia64, and they do routinely ship new
versions. Debian has those in unstable. So, they are not in a stable release anymore, but they are
in unstable and they do they sometimes lag a little bit behind, but they typically do still run. So,
there are people still using those now. They are not something that you can buy new these days. They
haven't been for a long time. So it's not and the ia64 was almost removed from the Linux kernel
recently. So they're probably not long for this world, but you also have some like NetBSD supports a
huge variety of architectures and they only support Rust on a handful of those architectures.
[01:42:12] So it's also about whether Git is available on things like the BSDs and you know
other and other different systems. You can actually query the Rust compiler for what platforms it
supports and there is a huge like Apache NuttX which is an embedded kernel actually does support a
lot of these. So there's a lot of systems that do support it but it also requires effort from the
porters for those platforms and sometimes there's just not a lot of them. So it can be difficult.
[02:02:24] All right, as we wait for people to settle in, I've got one dad joke prepared, so
we'll give this a go. I was walking down the street and I saw two ducks. I thought they were
speaking Spanish. Turns out they were Portu-geese. That was all the jokes. [Laughter.]
[02:02:52] Oh man.
[02:03:09] Just got a few people coming in. All right. Here to take us on a Git Meta adventure,
we have the sort of CEO of GitButler, co-founder of GitHub, and a certified wine expert. Let's give
a round of applause to Scott Chacon.
Why metadata needs a new approach
[02:03:39][Applause.] Thank you, sir. Thank you. All right. So, this is a new one. I don't
know if you guys have seen any talks that I've given, but I have this sort of disease of creating
new custom slideshow presentation software for almost every talk. This time I did it in Unreal
Engine. So, we'll see how it goes. Okay, so I'm going to talk about Git Meta. Does anybody here know
anything about the Git Meta project? Have you ever heard of it? Okay, great. So this is an
introduction to Git Meta.
[02:04:13] So this is essentially what is metadata? Why do we need a new approach to metadata?
There's lots of different ways of doing metadata. What is metadata in the context of Git? So Tomas
was kind of talking about this at the end of his talk. So I think that segues nicely into this. So
what is metadata? There's actually lots of different ways that we capture information about your
code that is not your code currently. The commit by itself is basically a big blob of metadata,
right? Like it tells you provenance, it tells you author, the committer, the time that it was
written.
[02:04:49] It helps you figure out who this came from. There's trailers in the commit message.
The commit message itself is metadata. There's trailers that can do signoff information, things like
that. There's security. You can SSH or GPG sign individual commits, right? And so there's lots of
sort of metadata in the commit. There's also metadata in files sometimes. So CODEOWNERS -type
files. The readme is really it's not really code, right? It's really for you as metadata about the
project. And we also have commit notes. So systems like Git AI, if you've used that, or Entire will
use notes or some notes like feature to add metadata to commits or to files.
[02:05:29] Usually commits. So this is metadata. So I want to talk about a new project that I
have been working on a spec for which is the Git Meta project in order to solve some of the problems
that we've had with git notes or commit messages and things like that. So what are the goals of
this particular project? One is granularity. So one of the things that git notes can't do very
well is you can't really attach metadata to lots of different things. It's hard to attach metadata
to a branch, for example. It's hard to attach metadata to a path, other than putting a file in the
path; change IDs; the project as a whole. There's lots of granularity, but most of the data is
commit-based, if there is metadata at all. Another one is performance: if you've used git notes at
large scale, it doesn't perform very well, which is one reason a lot of people don't use it for
that. Querying and mutating locally is also a goal of the project, in addition to having lots and
lots of keys and still being performant.
[02:06:32] Another one is exchange. So, I don't wanted I did not want to create a new API,
right, a new system. I wanted everything to use the Git systems that the transport protocols and
methods that you can already use in essentially every Git server. So, exchange is done via Git in
this in this project. And then the last one is merge behavior. So if you have ever used git notes
for anything, you can add sort of one note, you can add JSON if you want lots of different sort of
values, but then if somebody else mutates that you have awkward sort of merge semantics in getting
these JSON or having a lot of different refs with different types of notes on them or whatever.
[02:07:12] So I wanted to have CRDT like behavior. All right, so what are the new use cases? The
one of the reasons that this comes up is because there's a lot of new use cases for metadata that
that people are getting more and more needing to use. So what does that look like? So one is trust.
So this is actually one of the bigger things is trying to attest to identity who wrote this like and
that kind of is hand in hand with provenance which is one of the ones that I'm seeing more and more
a lot of systems trying to do which is was this written by an AI was it written by a human if it was
written by some sort of AI system what was the model what were the prompts what were the prompts
that that produced that code what are the transcripts, maybe tool calls like some people want all of
this data, some people want to be able to use that to figure out how this was written or whatever.
[02:08:10] The other that Tomas talked about is review of being able to put review alongside
your code rather than in some you know s you know having things in GitHub issues or you know PR
review data or something like that to be able to pull it down and have it alongside which means it
needs to be you know mutable. You have to be able to add comments to the system and so again notes
is really difficult at doing that type of thing in parallel. There's also stuff like evolution logs,
history, what is the previous version of this commit if you rewrite it, right?
[02:08:41] So, like having that sort of data. So, if you guys were at the Jujutsu conference,
they can do this type of thing in their metadata. But can you sort of push that over the wire,
right? Like you can't really push that over the wire currently. So, this is one of the use cases
that we're looking at. Again, licensing using AI systems. What license does it fall under? Can you
select parts of the code? Can you update that? Does it have to live in the source code itself?
Testing if you run tests on on a codebase can you do some sort of attestation that says that's some
signed attestation that says this pass CI right and you can do it in some other system and attach it
to the metadata of the code.
[02:09:20] And then working with binary files so locks and hints and things like that. Okay, so
that's what we're trying to do. So let's look at what we've had so far and how those systems kind of
fall down and what we're trying to avoid, right? So existing solutions are things like commit
message trailers which I talked about a bit. So one of the problems with putting metadata in the
commit message, right, or in trailers in the commit message. One is you have full visibility. So if
you want to put a transcript, you attach a transcript to a commit or test results to a commit or
whatever you have essentially git log becomes really useless. Like you can't have large data sets
in there for using the commit message for that.
[02:10:01] You also it's a little difficult to have independently addressable fields. You can
kind of do this with Git trailers in a programmable way. But it again is ugly and is all over the
place and it's difficult to mutate. So anytime you change or add sort of data to this, you have to
change the SHA of the commit which has to change the SHA of all the commits above it. Commit headers
are another way of doing this that solves the visibility problem. So, GitButler does this, Jujutsu
does this with change ID.
[02:10:28] So, we'll inject a change ID header, into the commit headers, which means if you run
git log, it's not shown. But otherwise, it's also very hard to find that information. So you have
to actually, open up the object and look through the headers. There's no way of seeing it with the
normal, user interface. It's also difficult to mutate. If you change something in a commit header,
it also changes the SHA and it also changes the SHA of everything above it. And the other thing with
commit headers that that you know we found is that it's not copied by rebase.
[02:10:57] So if you run git rebase, it will not copy those headers over. Anything that's
non-standard, replay will. So if you hit sort of a rebase button on GitHub, it will do it, right?
Because it's not running git rebase, it's running git replay. And the Git history commands and
stuff, I think, use the replay mechanisms as well. And so they will copy it, but normal rebases will
not. Problems with CODEOWNERS -type files. So this is also common to put metadata in the file
system itself, but obviously you have to do a new commit every time that you change it that doesn't
actually change your code.
[02:11:30] And it just pollutes the file tree unnecessarily. And then the last one is
git notes. So how many of you have used git notes in a in a project? That's less than I thought.
GitHub actually used to have support for git notes. I wrote the renderer where if you looked at a
commit that had a Git note, it would show it for a while and they ripped it out at some point
because it just people weren't using it and it wasn't very performant. Not because I wrote it, but
because actually it was probably because I wrote it, but yeah, so people historically it's also had
very poor scaling characteristics.
[02:12:02] They started doing fanouts now and that helps with some of the read and write
operations. But you still have to kind of clone the whole thing every time you're doing something.
And so if you have millions and millions of keys, it does get quite slow. All right. So, Git Meta.
What is the Git Meta project? So, this is a specification. I have a reference implementation which
has is in Rust and has a Rust library. So, you can do stuff with this. But the point is not
necessarily you need to use my Git Meta implementation.
Targets, keys, and values
[02:12:34] I know some companies that are doing this in Go and have their own. You can just kind
of point at the spec. The concept is to try to come up with a way a standard so that if we're adding
if you're adding tools that add metadata to your projects that we all do it the same way right so
that means fine-grained targets which means you can target things like branches you can target
things like change ids right the project as a whole namespaced keys so you can say sort of
agent:model or you know have sort of lots of different namespacing so you can get all of the keys
on a commit that have to do with an agent and keep multiple values underneath those namespaces.
[02:13:11] It's CRDT-shaped, so I'll show you a little bit what that means, but merge semantics
are quite easy and use normal Git merge semantics. And it uses only normal Git transport mechanisms
that have been around for at least 10 years in most of these these instances. So if you want to
learn more about it, you can go to gitmeta.com after this and I'll show you a quick sort of demo of
what this looks like. So the reference implementation you can run git meta setup forget about that
and then you target something.
[02:13:42] So for example, you can say set commit:HEAD and it will figure out what the sha of
that is and then it will write into the metadata and in a local key value store what that is and you
can say agent:model astra for example agent:provider openaiagent:prompt here's a thing you
can you know put in lists of lots of data whatever and then you can get them back out of the system
or in JSON or whatever right and so this is a relatively simple obviously thing but this is kind of
the types of ways of setting and getting these that you can do.
[02:14:15] You can also do paths. So for example, you can say src/main and I can say here's a
string that I want to associate with this path. And it doesn't change, right? So this is not a path
at a specific commit or something. This is simply a path in your here's data about this path. You
can also do different types other than strings. So I have lists and sets as well. So if you for
example want to do comments on something you can say you know sort of push to this and then get and
let's see the JSON is probably better and get this list out and you can keep pushing onto that that
list right and multiple people can do it in multiple places and then these lists will merge cleanly
every time and same with sets.
[02:14:57] So if I want to have code owners for example I can add to this sort of set of code
owners relatively easily and then get that set back out. And as you can see, because it's
namespaced, I can say, give me all of the metadata from the path src. So, anything underneath
that, and it will say, okay, here's all the files underneath that that have metadata associated, and
here's what the metadata is. So, that's roughly what it looks like. So, storage, I want to tell you
how we're doing this and what the actual standard is I'm looking for.
Storage and exchange
[02:15:30] So it's sort of like Git in that Git has sort of a working directory and you can
modify files and mutate files relatively easily and then you commit and then that writes it to an
object database that you can then use transfer transport protocol. So I implemented the key value
store locally as SQLite in the reference implementation you can do anything it doesn't really matter
the point is how you serialize and how you transfer the data. But you serialize to normal Git simple
Git trees and then we can use normal Git merge strategies to merge them.
[02:16:02] So this is roughly what it looks like. So we have for example some some targets here.
So I'm going to do the main branch and I want to put this string onto deploy:environment or
something. Right? And so or I have this commit and I want to do it's PR42 in case you need to look
up the pull request number or something like that or pull request data or whatever. And then when I
serialize, what it does is it writes a Git tree and it updates this refs/meta/local/main.
[02:16:31] So you can have multiple refs as well. You can put in filters that say everything it
starts with, OpenAI: is like an internal company stuff or Google: or whatever is internal
company data. So materialize that or serialize that to a different reference and I can push that to
a different place, right? So I can have sort of semi-private data as well. And the this is kind of
what the tree looks like. So for example, if I have this PR, it stores it as a tree, right? And so
the commit it does a it puts it under commit as a top level namespace then does a fan out.
[02:17:08] So the trees don't get too big. And then for example the PR value is stored as a blob
right and so this is kind of what the path looks like. It's commit and then the fan out and then the
SHA and then at that point you can start putting the key in. So it's review. So everything under
review: kind of goes under that and then PR and then __ means okay this is a special thing and
you can stop and you can get the blob out. So you can get an idea.
[02:17:36] If I add more metadata, for example, I put some tag set or something in here and I
serialize that, it writes another commit on top of that that has that as a parent and just adds to
the tree, right? So you just keep sort of growing the tree of values. And so yeah, that's more or
less how that and then the other thing is that lists work this way. So instead of __value as a
blob, it has __list as a tree. And then it does a timestamp and partial SHA of whatever the value
is that you're putting in there.
[02:18:07] And so that way if multiple people are writing to the same set or the same list, then
when you pull it down, we can just do a merge, just a normal in-memory merge of the trees and it
will interleave. And there is it's almost impossible to have collisions that way. And we'll talk
about what happens when you both modify a string in a second. So exchange. So now we have this tree
that has these values in it. So how do we actually exchange them? What how do we know if we're sort
of writing the same value or something like that?
[02:18:36] So let's say that we have, a string dino name. We have a set apples and ferns of
feeding foods. We have a list of feeding times that we can sort of append to and then we push it to
a server, right? And then the second user does a fetch and materializes those those values and then
we both mutate them. Right? So we both change the name. He adds moss, I add leaves, puts in extra
feeding time, I put an extra feeding time and then keeper two pushes. So these are the new values.
[02:19:11] It's whatever they had. Keeper three or keeper one tries to push and it gets blocked.
So the semantic here is is we use this for the CRDT sort of eventual consistency of just saying it's
always [fast-forward-only], right? So if you mutate something and then you serialize it and you push
to the server and the server says that's a non-fast forward, then what you have to do is you have to
rebase and then you have to try it again. And if it says sorry, somebody pushed since in the small
amount of time it took you to rebase and you just keep doing over and over until you win, right?
[02:19:42] And that way everybody is eventually consistent. And so now what you have to do is
pull this in and merge it, right? So you just do an ours strategy. So whatever is local if in the
case of a string it takes the local one because that way we still have the other one in the history
that we can go back and see that it was changed. We don't lose data. But the sets and the lists will
just union and then you can go ahead and push that up.
[02:20:08] So that's how the exchange protocol sort of works for this, right? And since we're
not doing anything very special, it's really easy to look through the trees. It's really easy to
inspect the values. And so everything's relatively simple. So scaling. So one of the problems was
scaling. So how does this work? Right? So the way there's a couple different interesting things that
that I wanted to do. One is let's say you set two keys and then you serialize it. So it'll do a
commit and the commit says here are the keys I modified in this commit right so it's sort of a diff
stat like a cached diff and I'll show you why why that's interesting later and then it writes the
tree in that says okay here's these values right and you have the two blobs now you add two more
keys and you serialize again it will say okay I modified three and four oh actually I even wrote
this in here I modified three and four it writes a whole tree right so it just keeps adding adding
to the tree to say here's all the values I have and then writes those blobs in. If this starts
getting really really really big, right?
Scaling and working sets
[02:21:14] If you start having the tip be larger than you want, you can set pruning values and
it sets with some auto pruning values. And so when you serialize, if it sees you're over a limit, it
will prune the tree to only have, let's say you had, 250,000 keys or whatever, it'll write a new
tree that doesn't introduce any new values that just prunes the tree to say there's only 10,000 now,
right? And so it just and once it gets back up to 250, it prunes it back down to 10. You can set
those limits wherever you want.
[02:21:44] And then you add more keys and then the tree starts growing again, right? And so how
does this work? That's interesting. So if you push all this somewhere and then a second person
fetches what we do or what git meta setup will do by default is it'll do a blobless clone right.
So if you guys have ever done blobless clones it will create a packfile that is just the commits and
trees and none of the values none of the blobs and it'll send all that over the wire which tends to
be relatively small.
[02:22:10] Also most of these delta all the trees delta compress fairly well next to each other
and then it brings it down and then it'll hydrate. So the second step is it will do essentially what
a blobless clone does, which is a second fetch that gets all of the blob all the blobs that it needs
in order to populate a working directory. And it does that. And so you can set these to whatever you
want, right? But the point is that you get a working set that's local so that most of the things you
probably would want to fetch are already there and it does that automatically.
[02:22:40] But the other ones are not. And so if you then say, "Hey, I want the value for K1."
And it's not local. It is it just uses promisor remote, right? And so it will just go to GitHub or
whatever and say I need these values. And you can also say I want everything under the you know
everything for this commit for example, right? And it'll say okay there's 47 sort of blobs that
you're going to need to get all the metadata for this commit. And so it can do one round trip to say
don't negotiate.
[02:23:09] It does the pro the whole promisor thing. Don't negotiate. Here's all the blobs I
need. And then the server will send you just the packfile of those blobs and then you have them all
locally. And voila. So now you can just kind of use GitHub or whatever as a as a promisor, right? As
a key value store. The other thing you can do if it gets really really really big, there's actually
two different things you can do. One is you could prune it in or you could take some point and then
push it to like a history ref and then add that and fetch that down and get those values after the
fact, right?
[02:23:42] And then just have a shorter history as a working set. The other thing you can do is
you can do a shallow clone, right? So you can say in the Git Meta configuration depth is 10,000 or
something like that. And it'll just fetch down the last 10,000 commits and trees. And then if you
want to you can deepen it, right? But if you look for K1 at this point, it'll say I don't know what
that is. I haven't seen that. And you're like, oh yeah, how about we deepen it and then it can
deepen it more and you can get those values again.
[02:24:10] And the nice thing about this, the interesting thing about this is that if I'm
looking for a key, the way that I know that the key is in there is that I can walk through the
linear history of commit data and each one of them says here's the keys that I modified. They sort
of have this cached diff. And so I can find it pretty quickly where the last time that was modified.
And then look into the tree to see what is the actual SHA that I'm looking for and then a ask for
that blob SHA.
[02:24:37] So speedwise, I tried to make large sort of relatively normal shaped metadata
repositories by taking metadata that I've had around and then trying to say make millions and
millions of these in this shape. And try to get an idea of what these different things take. So for
100,000 keys in my tests, it was taking about 12 seconds. And so that is the fetch takes a couple
seconds, the hydration takes a couple seconds. And these are relatively large tip sizes as well. A
million takes a little over a minute to do that. 5 million takes a pretty long time, but if you do
it shallow it's still a minute and a half or something like that.
[02:25:12] So for you know 5 million keys it's not I feel like this is within the realm of okay
and clearly this can be made faster. But again this is just getting the commits and the trees and
then looking for what are the values that I need. All right. So, one last sort of example. In the
reference implementation, if you're wondering if I if I used OpenAI to model all of these things in
Blender, the answer is yes. So one last thing, if you want to see kind of an example of what you can
do with this, you can download the Git Meta reference implementation in Rust and you can run a
command that imports all of your pull request data into Git Meta.
Pull-request-aware blame
[02:25:55] And then I wrote a blame mechanism that uses the PR data rather than the commit data.
So which I find actually a little bit more useful. Right? So what it'll do if you do import instead
of a blame like this for a file, you can run a blame like this and it will say okay for these the
last time that something touched these lines was not this particular commit but this PR, right? And
so if you have a whole bunch of lines that were modified by different commits in the same PR, it'll
group them into one PR and then it give you a URL to go see that if you'd like to.
[02:26:28] But again, this is all running just out of the local metadata that I have that was
imported from GitHub. And I can push that somewhere if I'd like to. Okay, so that is it. Are there
any questions? Thank you. Hello. Yeah thanks for the great talk Scott. I just have a question. I do
understand the need to have additional context being associated with comments because and a big push
for that is because of AI. But Git forges and have typically struggled with even if we optimize
storage and the footprint of the repo on the client side forges themselves struggle with like more
and more data.
Q&A
[02:27:16] The examples that you give are lots and lots of key value pairs but key value pairs.
But if you look at how agents get used some examples of how agent gets used today is like big big
plans getting written for some project that has been done that is achieved through a multitude of
commits right and these are massive textual documents that sometimes have no overlap with each
other. So do you do you have any concerns that using Git as not just for association but as the
storage and transport model for things that are large and do not compress well delta compress well
with each other could be an issue going forward with the strategy.
[02:27:52] Yeah. There's a hundred differently shaped problems, right? It really depends on what
you're trying to do. If you wanted to try this out, I pushed the Unreal Engine slideshow
presentation that I made to [repository name unclear]. You can clone that down and run the Git Meta
stuff and get all of the values. I put all the metadata in here about the sessions that I did and
the transcripts and stuff like that.
[02:28:25] Which I think is mostly what people want to do. When I did it, it had a bunch of
image values in there because it would, fire this up, take a screenshot, look at the screenshot, and
it kept all of that in the transcripts. And that made the metadata really large. And so, the way
that in this implementation, I can clear it by saying local, if you do a key as local:, it just
won't serialize it. And so I said, just make all the images local.
[02:28:49] So I still have them in my metadata. I can still reference them. I could serialize
them and push them to another ref if you want to pull them down, but I won't do it by default. So I
think it just depends on what it is that you're wanting to add as metadata because you can also use
this in combination with like LFS or something if you wanted to do that, right? Like it just depends
on are you doing a lot of binary non-diffable or not not easily delta compressible images and things
like that. For the most part, what I've seen people want to do metadata stuff with that notes and,
existing systems fall over with like Entire or like Git AI or whatever is is agent data, right?
[02:29:30] That's most of the stuff that I'm seeing right now. But I've found lots. There's
another project I have at [repository name unclear] that does issue tracking this way. There's a
little TUI, and I can write all my issues locally, synchronize them with Git Meta, and pull them
down somewhere else. Again, I can mutate state and stuff like that. It's a nice way of doing
offline-ish ticketing. For some of the experiments I've been doing, it's nice for agents as well,
because they don't have to use GitHub APIs to look at issues or whatever.
[02:30:08] They can see everything locally. They can mutate state locally and then they just
synchronize when they synchronize and then everything kind of works out fine. So it depends on what
you're trying to do with it, right? Are you doing are you keeping transcripts? Are you trying to do
a lot of image data? Are you, trying to keep snapshots of VMs or something like that? That would be
a different problem set, right? But that's mostly the shapes that I'm seeing is is something that is
notes is falling over. What is a better way of doing this type of thing?
[02:30:31] And I instead of everybody reinventing their own ways of doing it, which is mostly
what I'm seeing, I think it would be nice for us to be like if we're going to do something this
shaped like this is a relatively straightforward way of doing it.
[02:30:44] Yeah. The for just for context, the example that I had in mind was code review. And
like the agent trajectory along with the conversations directly feed into how you do the code
review. So yeah, I think we can be selective about what you what you upload and what you locally
materialize. So
[02:30:59] Yeah.
[02:31:01] Yeah. Oh
[02:31:04] Yeah. More of a technical question. What are the like limitations regarding the keys?
From what I could if I understood it correctly that if you have a colon it just nests it and but
what is like are is there an upper bound or also regarding the key portion length so to speak
[02:31:24] It's only the double underscore and there there is I forget exactly how I did it but
there is a escape mechanism so you can put underscore two underscores at the beginning of a thing
but there's some escape that that will pull them out so there's not really any limitations other
than it needs to be writable as a Git tree file name. So there are some like some general
limitations that way of what you can encode as a tree file name.
[02:31:51] I don't know.
[02:31:51] So I've been I've been
[02:31:54] Yeah I've been looking at at Git Meta Git Meta for how I'm wanting to try and
implement metadata for Jujutsu and at least for as far as transport goes. And I'm wondering if you
foresee any issues with how Git Meta does scaling and how maybe that might interact with how
Jiujutsu does things. I've I have little experience in the implementation of Jujutsu, so I'm trying
to come at this a little bit blind. But
[02:32:32] Yeah, let's talk let's talk after the fact. I you know the scaling things that I've
seen I think can generally be solved by the shallow or the sparse sort of ways of just using Git
mechanisms that are built for scaling for scaler or like general scaling stuff. So if it works in
those in those ways then it should be able to work for this as well. But yeah I'm just taking
advantage of what Git does from a scaling perspective in the first place.
[02:32:59] Totally. I see that you say that it's like fast forward only. So it's a linear
history. I feel like it's require some centralized repo that do all the ordering and I see many way
where you might want something more distributed. A simple example would be if you clone a open
source repo but you have an internal version of it. You might want to pull change from the open
source repo and then push the change that are private to your company to the to your private repo.
And so how do you reconcile all the notes from the public history and the private history?
[02:33:40] So there's a filter that you put in that says if it matches this then serialize it to
this other head. So the way theoretically it would work is you would say everything that starts with
GitHub: or whatever is private data that goes to like the private internal team or internal team
data or something like that. And that's you can put that in the metadata itself so that everybody
uses the same filtering mechanisms. And then anything you want to make sure it doesn't go to the
public thing, you put it in with that as a key or some some match that matches the expression,
right?
[02:34:11] And then it will then yes, you have two different places that you're doing, but for
each of those reference, it still should be that fast forward rule, right? Like you don't
non-fast-forward it. But as far as I've I haven't done extensive testing internally with real world
stuff for that because we don't really have that that use case. But that's the design.
[02:34:35] Yeah.
[02:34:36] I've got two questions actually if that's okay. So the first question is so the keys
are you got like project commit branch. Can you have multiple keys on a value? Like can you say like
this is a code review comment for commit x path y line 74 to 75?
[02:34:50] Theoretically yes, but I didn't put that in this version of the spec. I mean, really
it's just the serialization specification, right? So if there's a way I mean I think the easy way to
do that would be to have a commit and then have the value be a path, right? Like you can do that
that type of thing.
[02:35:11] But the value would want to be like the code review comment itself. So if you want to
have a code review but yeah I feel like I had that conversation when I was talking to the Google
people about this and we Oh wait did we have this conversation?
[02:35:26] No Facebook. [Laughter.]
[02:35:29] Yeah I mean if you have a good idea for how to do that like I think you we can it's
more of a what is a serialization standard that you want to be able to do for comments. I feel like
you would just put it as a comment and then you know have have the comment be a list where it has
something in it that says this is what I'm commenting on because it is kind of like
[02:35:49] So associated with a commit and then have like a list of this path this line range
this com right I mean if people start using if Jujutsu starts using this if entire I was talking to
some of the Entire people they were thinking about starting to use this for what they're trying to
do rather than their own hand thing the next If people are using this and it works out really well
like it's worked out great for the stuff that I've been using it but I've been using it for smaller
time stuff then I think it'd be nice to standardize on some of these things how do you do code
review comments like what does that what are what is the key like because it can be anything right
like I have a ticketing system that I wrote and I'm like this is a standard now I guess like I you
know who knows right
[02:36:30] Second question first right is not related to the first one so the it seems like
there's kind of two parts to this right there's the key value store the standardization of keys and
you had like your reference implementation has it stored in SQLite you could store it anywhere else
locally and the second part is essentially kind of abusing Git trees as a transport and storage
mechanism because there is no nothing better is there scope for protocol extensions where you could
have a forge that actually just stores it as key value stores in S3 or something rather than having
to muck around with all these trees and values and everything.
[02:37:06] Yeah, certainly. I mean, there's you can also, transfer Git data not using Git push,
right? Like you could like create a packfile and then have a special API that you uploaded and make
it resumable or something like that. I say that because I did that. But the question is like what I
what I think would be really valuable is like essentially every Git forge already supports all of
these protocols, right, this way. And so it would theoretically work with almost anything anything
you stand up like you don't we don't need anything new and it doesn't need to have vendor lock in
for some new API or something it could be built into Git right like like we could say okay here's a
better way of doing this within Git but now you have to wait a couple years for that to be widely
implemented everywhere right so
[02:37:51] You have like a yes I guess I sort of talks years ago having like proof protocol v2
where you have like a capabilities things you could have like it could be if if we could get that
over the line we could have like a capability of like I can actually do get Git Meta the simple way
rather than
[02:38:05] Yeah I mean I guess the question then is what are you gaining like like what is more
interesting about a special protocol than abusing Git trees like abusing Git trees actually works
relatively well for a lot of the things that I'm trying to do this way so it seems I mean it's and
plus I love abusing Git stuff so [Laughter.] it's one of my favorite things to do
[02:38:26] I think the thing would probably be scale like you've got okay sure it works with
hundreds of thousands of keys but if you have billions of keys then that's going to be an issue.
[02:38:33] Yeah. I mean again there's lots of different ways that you can approach that like you
can I mean and it's the same thing for I mean it's exactly the same thing for codebases as well
right like like the if you get to that type of scale you're probably making something custom
anyways. And or if you're doing you know like any codebase and it has millions and millions of
commits you're already doing you know shallow clones and partial clones and stuff like that. So like
it's just trying to use the same mechanisms that you're probably already using internally at that
scale anyhow, right?
[02:39:08] Yeah.
[02:39:08] I could not have teed myself up better with that question. How would you like the Git
forges to handle Git Meta? Would you like us to just show it? Is it not meant to be user-facing?
Should it be user-facing? How do you think around what you want to what would be your ideal
experience there?
[02:39:23] I think it depends on I mean I'm more interested in your use case, right? Like what
do you want to have at the forge level like a lot of your talk I think
[02:39:36] Tomas yeah
[02:39:37] That was good
[02:39:38] Everybody's I was like you're talk yes it was your talk is yeah being a like for
example this is not going to be good for stuff like videos right like if you have video as metadata
like you do want that to be more of a LFS type thing or more likely you upload it somewhere that
hosts it I mean the same way that kind of GitHub does, right? Like even in their issue data, it's a
it's a URL that they're hosting separately. And so like I think but I think the metadata about this
is it would be kind of nice to be able to bring it down.
[02:40:07] The question is like what do you want to be able to show? And if you want to be able
to show agent transcripts or you want to be able to show sort of, this type of provenance
information and you have it in this format, then tools that can read and write that, become
relatively easy to do. So I think it's both ways. It's not what do I want to put in metadata. It's
what do you want to show on the forge and then can we agree that okay that's how that's the key
space that we'll use for agent transcripts like this is how we will put sort of the JSON transcripts
into there no matter like if it's you know Codex or Claude or whatever like I think one is how do
you move store and move the data around and the other is what are the standards of what we want to
keep and can we all try to keep the same types of information right
[02:40:54] Heard and aligned I guess my question is more what do you expect to be the same
default here. For example, like we should certainly special case how we show comments, we should
special case how we show stack data. When when GitButler adds a new key, would the expectation be
that it's user-facing, that it's not user-facing? Would it what would be a sane default for us to do
with that key or should we just ignore it entirely and be like it's get it's get metadata? Who
cares? Like no one should see it.
[02:41:20] That's a great question. I don't know. We should talk about it.
[02:41:22] Cool. [Laughter.] hi. I have a sort of technical question about the serialization
which is specifically about the fan out and the pros and cons of it because now you have a target
dependent fan out scheme which means that targets are not an extension point. You can't add more
targets without it being part of the spec. And it also seems like the fan out is mostly useful if
you actually materialize the tree, which you probably aren't doing in normal use cases into the file
system where the fan out like speeds up directory iteration.
[02:41:59] So like have you like is there like a stronger rationale that I'm missing somehow
that to break this extension point of people adding their own targets? So you can always add your
own targets under the project one because there is no fan out right. So when I do when I do like
ticketing is not there's no target for it right so I just do it under project and then I come up
with my I came up with my own fanout scheme for how I'm doing ids and stuff like that.
[02:42:28] I mean you want fan out so that updating the tree with new values when it gets really
large is fast right is relatively fast. And so you don't have huge huge trees that you're deltifying
and pushing over the wire. I did a bunch of tests with different fanout schemes and it turns out
that the one that git notes uses is actually the best one for these shapes, right? Or get object
database uses, right, in loose format. So there's turns out there was a reason for that. But yeah so
I think if we came up with other keys this is most of the I mean if anybody has ideas for other you
know targets to have these are the ones I could think of that made sense anything else can go under
project for now right unless it becomes widely used like I was really debating issues or things like
that other types of metadata but that's not really a target right so so yeah I'm not I'm not sure I
would love to have conversations with any of you about ideas that you have about that type of thing
too.
[02:43:24] A small related question which is is there any way to say that a specific key should
be persisted through prunes so that a very slowly changing key doesn't have to require someone to
fetch all the way back in time.
[02:43:37] That is a great there is not in the reference implementation. I mean again that's an
implementation detail like there could be a thing that says like anything under this name space
never prune right or things that match this this thing like this regular expression or whatever
never prune those but yeah then you can get to the thing if you have too many and then you go over
the limit and then you can't like it would auto prune every time so I want to be careful about that
type of thing but yeah it would be possible Rodrigo Hi, thank thanks for the great presentation
though.
[02:44:13] Please stop using make me a sandwich as a commit message. Yeah, my actual question
was was similar to the one about prune is like how do you think about garbage collection and
preserving some data that you really care about long term like code review versus things that you
don't care about like your transcripts are probably useless two years from now.
[02:44:33] Yeah. Oh, so taking them out of the metadata entirely.
[02:44:36] Yeah. Like actually garbage collecting some of it from from the repo versus not,
[02:44:41] Right? I'm curious how much of a difference it makes in most applications because
since it does the shallow clone, it's just never going to look for those blobs. So the I don't know
if it's a worth I mean, you'd have to go back and it's like removing a file from your code history,
right? Like you'd have to go back and rebase everything. So if you did a split where it's like,
okay, this is the new local set. I'm going to go back. Some people do this with code, right? Like
I'll go back a thousand commits and then make a new empty blank one and then rebase everything and
then use that as my new sort of working set and then have this as a history that I can graft on.
[02:45:16] So you could do that type of thing, right? And filter out stuff you don't want
anymore or something like that. But yeah, I think generally it doesn't really matter because you're
just going to the only thing is you get some more tree data like during the blobless clone and then
you do the you never run those promisor calls anymore, right? So then it's just it's bits on disk,
but it's like it really depends on how problematic that is for you, I suppose. Okay, I think maybe
one more.
[02:45:47] Oh, sorry. We pointed at one more and then I'll I swear I'll give up the stage.
[02:45:52] Hello. You mentioned that you were using CRDTs for the merge conflict resolution.
Well, you don't have merge conflicts. So I was curious what was the motivation were you was the
motivation only because git notes do it in a weird way where you get in a weird state in Git or
was the motivation also because you wanted it to be specifically lightweight.
[02:46:16] So I'm not doing CRDTs. It's just CRDT-shaped, right? It's doing the type of thing
that CRDT systems do as well: it's eventually consistent to the same point, right? No matter what
order of operations you do, if people are offline, when everybody gets back online, eventually they
get to the same data state. And I'm just abusing Git trees by sort of writing what looks like files
essentially and using files and directories and then just pretending that a merge is like a merge of
somebody added code here and somebody added code in the same directory, right?
[02:46:53] You're not going to have a directory conflict. And it's the same way of sort of
making sure you don't have key conflicts that way. So it's CRDT like in that it's eventually
consistent to the same point. It's not actually implementing a CRDT, right? It's not using Automerge
or whatever. Okay, thank you very much. [Applause.]
[02:47:17] All right, up next we've got Elijah Newren who is famously known for the
git-filter-repo and the sort of merge-ort strategy that's been the default in Git for the
longest time now. So, let's give him a big warm round of applause.
Scaling without replacing Git
[02:47:35] Thank you. As you said, my name is Elijah Newren. I work at GitHub for the last few
years. And today I'm going to be talking about scaling Git for the AI era. Going to start out with
just a little bit of introduction. And I think most people in the room have experienced or seen
this. Our chief technology officer at GitHub actually put this particular graph on a blog post a few
months ago just showing how we've got a lot more commits coming in. We've got a lot more pull
requests being merged. The scale of everything is just increasing dramatically and it's at such a
scale where we're struggling to support the largest repositories out there. And there's a lot of
areas where that goes into but even Git itself has a number of places where we need to make changes
in order to be able to scale to these large repositories.
[02:48:29] One of the things that Vlad put into this blog post that I really liked but I'm going
to nitpick slightly is he said at high scale small inefficiencies compound and the thing I want to
nitpick about that is that big inefficiencies also compound. Not just the small ones. And I want to
talk about a few of those that we found and are working on. And some of these things will be server
side, some will be client side. I'm going to start off one that with one that's actually both of
those. So merge-base finding the common ancestors in history.
Walking less history for merge bases
[02:49:02] So merge bases can be used to figure out okay what part of this pull request is
relevant compared to the main branch. You know, what do I need to look at for the diff? Is a cached
merge still reusable? It's also used for merging. Merging needs to know what the common point of
history is as well. And this actually uses a function in the code is called paint_down_to_common.
And the basic idea is just walk backward through history to find what the best common ancestors are.
And I'm actually going to demonstrate that starting out with a really simple example that has just a
single merge base.
[02:49:41] So here we've got all of our commits. We got two branches, main and feature. We start
out by just coloring the commits that we're looking at. So E and F are the newest commits. And we
just queue up those two one in blue, one in red. And the idea of this algorithm is basically take
each commit and paint its parents. So if we start out with the newest commit on this will be F. It
will paint its parent C with the same color red. Then we've got of the two commits remaining in the
queue.
[02:50:14] We take E. It's the newest. It paints his parent D. D is now the newest. We paint his
parent B. C paints B as well. And where red and blue first meet, you've got a merge base. If you
walk beyond that point, then you've got a stale commit. And once all of your nodes in the queue are
stale, then you've finished and you can say, "Okay, what merge bases did I find?" Of course, this is
a simple case. Typical histories, you're going to have a lot more merging and diverging. So, let me
go over one that's just a little bit more complex, but it's the same algorithm.
[02:50:49] So, we got two commits in the queue with the dark circles around them, Q and R.
They're the two tips of our branches. The newest one is R. So it will paint its parent O with red.
Now Q is the newest. It paints its parent P. P is a merge commit. It has two parents. So it's going
to paint both of those. Both L and N will become blue. Now the newest is O. It will paint M. N is
the newest. It paints both I and K with blue. M will paint both J and K with red.
[02:51:21] And now we get a merge base because red and blue met for the first time. Now the
newest is L. It'll paint F with blue. K will paint H with purple. We're still we've already gone
behind a merge base. Then J will paint G. I will paint G. And we've got a second merge base. And
then we continue on from there. H paints D. G paints E. And we can't stop yet because F, our three
commits in the queue are D, E, and F. And one of them is not stale. So we f paints D but that one
was already stale.
[02:51:57] And when we end up with just stale nodes then we know our algorithm is done and we
can say okay these are the merge bases that we found. I was looking at a particular case with this
with a large monorepository and in one case I found they had just their PR was one commit off of the
main branch and so I can simplify it a little bit because this one demonstrates what the issue was
with that algorithm. So if I look at the main branch it's got the newest commit. You'll notice I've
got subscripts there because n will be however many commits this is advanced.
[02:52:40] It might be hundreds of thousands or millions. Main will paint its parents. It keeps
on walking back through several. We get to some merge. It paints both of its parents and at some
point we finally get to where D is the newest commit. It paints its parent. The commits on main go
back. We find a merge base and then we have to keep on walking back from m_n minus 8 all the way
until we get everything to be stale. One thing you may notice when you look at this algorithm that I
was asking about is why did I have to walk through those extra hundreds of thousands of commits on
the main branch to get back to the common point of history.
[02:53:21] Could we have stopped earlier? And if you look at this, the thing that I mentioned
earlier was the key to it. You can only get a merge base where red and blue meet for the first time.
So if you've run out of one of the two colors in the queue, you can stop. And the interesting thing
about this was is basically someone had some other branch, this bug fix bug fix branch that was
based on a really old version of history. It got merged in ahead of where your PR was based. And
then it made all the merge bases with your branch slow and that was what was happening inside this
repository.
[02:54:00] So just changing this to exit a little bit earlier in some repositories provided a
pretty massive speed up. The repository that I was looking at a lot of their cases all became 70
times faster. There was another monorepo where not only was it some cases that got 70 times faster,
but everything sped up so much that their average merge-base run was 20 times faster. One
interesting thing about this algorithm, this little change was it was discovered three different
people within about a month to a month and a half. I discovered it, deployed it at GitHub.
[02:54:38] A little while later, Patrick Lühne discovered it, contacted me and was like, "Hey, I
found the same thing." He's also at GitHub. And then Kristofer Karlsson from Spotify emailed the
list and said, "Hey, I've been looking at this. I think there's something we can do here." And I'm
like, "Yeah, actually I did that recently. Here's some patches. I'm about to go on vacation, but
please extend them and I'll come back and review." And so he went and did that. Did a great job on
all those patches. I reviewed them.
[02:55:04] And so the next release of Git is going to have this improved algorithm. One
interesting thing though this algorithm depends on commit graphs. So sometimes even when you write
new algorithm and the algorithm is correct and works well but it depends on data not being corrupt
and at scale we found out that we actually had a bug where some commit graphs were corrupt and we
had to go through and fix them and I sent a patch off to the list to fix it up so that we wouldn't
get more of those. But it had just gone dormant for a while because nothing was triggering it.
[02:55:41] So that was an optimization that I was working on that was both server and the
client. Next one is going to be just the server side. So in Git all of its objects are stored either
loosely or in packfiles. Packfiles can contain an arbitrary number of objects. So, for example, if
you have a repository with 10,000 objects, they could all be stored in one pack, or you could have a
thousand packs with 10 objects each, but these behave very differently. Within a packfile, you can
do a binary search for the object that you're looking for.
Aggregating packfiles
[02:56:18] But you have to look across and find the right pack that has it. So the difference in
time for a number of probes that you need to do in these two different cases to find the object
you're looking for in the worst case is going to be a factor of 250 250 times different. So we don't
want to have lots of packfiles. We want to have few packfiles with all the objects in them. Loose
objects have a similar issue where the size of the directory keeps on increasing and so there's a
linear search to try to find the file that you're looking at.
[02:56:51] So we don't want to have a lot of loose objects. We don't want to have a lot of
packfiles. We want to have a few packfiles. And the obvious thing is well if I've got a whole bunch
of packfiles then we want to repack to fix this up. We may have had a previous repacking represented
by this blue thing where we've already put all of those packfiles into a nice thing. We've done all
the delta searching. We've got additional things on top of it, multi-pack indexes, bitmaps, and so
on. But now we want to go and repack that whole thing.
[02:57:23] Part of the issue is that while we're repacking, some pushes come in, some more come
in, and they keep coming in so that by the time that we finish repacking, we actually end up with
just as many packs as we started with or maybe even more. And in a large monor monorepos today, if
you're pushing once a second, you get a new pack every second. And if the repacking takes 20 to 30
minutes, then at your best, you're down to 1 to 2,000 packs. So, one of the things that I thought
about was how much would it help if we could somehow repack just the new stuff faster somehow?
[02:58:04] So, we took this picture and we didn't repack everything, but we took new packs and
were somehow able to get it down to a smaller set. Of course, I wanted to measure first and say,
well, how much of a benefit would that be? So, I went and compared it in an example monorepo. So, it
had 1,400 packfiles before I did this, five packs after I just aggregated the new ones, and then I
measured the timings of a number of different operations. And this particular monorepo was unusual
compared to others in that it spends a requires a lot more time for diffing and merging but still
otherwise seems to be similar to a lot of the other monorepos and when I did this I found that
merging and diffing you do get a small improvement but since they were so heavy on merging and
diffing in that particular monorepo you don't see too much of an improvement but a lot of other
operations checking for ancestors ahead behind object existence.
[02:59:05] Some of those speed up rather dramatically when you do this. And 1400 packs is what I
called a normal pack count for some of these repositories. Sometimes they get up to the point where
there's several thousand. And when there's several thousand and you aggregate it, you see even more
dramatic speed ups for a lot of those operations. And even merging and diffing can go quicker. So
that says that there are potential wins if we can somehow aggregate things quicker. Problem is even
if you use the fast kind of repacking geometric that can still take 20 to 30 minutes for these
repositories.
[02:59:45] So like I said with my idea what if we can aggregate the small packs and leave the
large ones alone and maybe also cut out a lot of the typical work that is done on these. So
repacking usually does delta search, cruft handling, bitmaps and MIDX updates to commit graphs and
what if we just toss that all out for these. I tested first on a few hundred packs within that
repository and found that yes indeed we can get that done a lot quicker. It'll be a few minutes. But
then I went and looked and found a few optimizations as well and could actually get those few
hundred packs done fairly quickly.
[03:00:27] So then I wanted to scale it up and say okay what about these other cases where we
had a few thousand packs as well as include the loose objects. Now how long does it take to get down
to a smaller number of packfiles? And in the case where I called the normal where we got 1,400
packfiles and 13,000 loose objects, it took just over 6 seconds. Where we've got a lot more, that
took around 45 seconds. But that gives us a way of at least saying, hey, I can get back to some kind
of better maintained state.
[03:01:00] So I've got a starter better starting point before repacking. So the idea then is
I've got a picture of what currently happens on the top where we repack directly versus doing some
kind of aggregation and then repacking on the bottom. So currently today we've got a full repack of
everything and new strategy is let's try aggregating just those other ones. The aggregating will
still take a little bit of time as I mentioned on the previous slides. There might be some new
pushes, hopefully not too many that exist in that time. And then the aggregating finishes and that
pack isn't nearly as optimized as the blue one.
[03:01:38] It's missing a number of other things, but we can leave that around for later and
say, okay, no, notify the next following repack that it's got to do more work on that particular
pack. And as soon as we finish the aggregating, then we can say, okay, now we're going to repack
everything, including that one new push that came in. And then both continue to get new packfiles.
But one thing you'll notice here is the total number of packfiles in the top versus the bottom is
about a factor of two at the end.
[03:02:06] And early on it's much bigger ratio. There's only a few packfiles at the start there.
But that brings a follow-on question as well after repacking on these two actually finishes near the
same time. But another question is, can we keep those new pushes compact as well? So if we started
at this point where we got a big heavy repacking happening, we've got new pushes coming in. Could we
aggregate those as they come in? So this is something that I've been working on. I've got series
about 20 patches just about ready to send off to the mailing list.
[03:02:49] But I did a whole bunch of testing on this as well. When I first did it, found a
number of race conditions. They're certainly there, but interestingly, most of the races that I
found already existed within Git. And some of them we were actually hitting in production, just not
as often, but it let me find a few of these. And so we can fix things up and make it so that
repacking is better in general. But after I got through all the fixes, I had to do multiple rounds
and loops of test this on a server.
[03:03:21] Ran for 24 hours with thousands of repacking and aggregations while concurrently
doing a lot of pushes, merges, and reader ops in this repository and eventually after all the fixes
got through the time with no command failures, no missing objects. So I'm hoping that this will give
us a way to be able to handle some maintenance a little bit better. In the near term. Next one I
want to discuss now I'm moving back to the client side because a lot of these monorepos they don't
just use our services on the server end but if they want to be able to continue using Git their
operations on their side need to be relatively quick as well and it actually touches on something
that I was interested in for many years back in 2010 I posted some patches to the mailing list to do
what I was then calling a sparse clone I didn't get it finished off I had the initial prototype and
ideas there, but people weren't interested in funding it at the time.
Batching partial-clone fetches
[03:04:24] But I've always been interested in how can we make this kind of a thing work better.
And in the intervening years, we've gained a number of capabilities that help in this direction.
We've got partial clones, we've got shallow clones, we've got sparse checkouts, and sparse checkouts
even with sparse indices. And these are great features and things that customers will want to be
able to use together. But it turns out that not only are they not fully coherent, but we've got some
rough edges and some incompleteness in these. So I want to talk first of all within this thing about
partial clones.
[03:05:02] So in order to describe partial clones, I want to do a quick review of Git object
model. People are familiar with the fact that commits have a hash associated with them but the trees
and the blobs within Git also have hashes with them. So a commit denoted by the purple circle has a
top-level tree. The top level tree has zero or more subtrees and zero or more files directly within
it and then it just recurses all the way down. And here I wanted to represent a large repository.
But of course slides only have a small amount of space.
[03:05:37] So imagine that I've got a million unchanged files and directories in the bottom up
there. Now if someone makes a small change to this repository, maybe they go and edit a.py, then the
way this works in Git is of course you'll have a new a.py. It will have a new hash. Because it has a
new hash, you need a new source subdirectory that has a new hash that includes it. And that will
need a new top level tree and then you'll need a new commit. So with this one change, you've got
four new objects and the rest of the million or so objects are all shared.
[03:06:18] And the idea behind a partial clone is that you get the commits and the trees, but
you defer all the files until later. So you don't have the files to start with. And this is I think
a couple other people already mentioned this common command before with clone you use the
--filter=blob:none. It's often used together with --sparse so that you have a subset of files to
start with and then Git will go on demand download the files that are needed so that you have those
and not necessarily all the other objects.
[03:06:55] One of the issues with partial clones is that the backstop to this is it needs to go
get files on demand. So it may be that some Git command says hey I need file one and says okay I'll
go fetch that then it says oh I need file two as well and goes and gets that and then says I need
file three as well and gets that and you can have a massive roundtrip overhead. What's a lot nicer
is if the Git command can say oh I already know all the files that I'm going to need. Let me send a
batch request up to the server.
[03:07:26] When partial clones were first introduced in 2017, there was only a single command
that would send a part batch request up to the server and that was checkout. Over time, additional
commands and low-level functions have added support for it. But it actually went quiet for a little
while and then earlier this year I added a few more subcommands, cherry and grep. Some customers
who are using partial clones to try to get things to be quick were running into issues where things
were taking really long because git cherry when they ran it just repeatedly sent single objects
requests up to the server and you can make it hundreds of times faster by just batching that
request.
[03:08:08] There's also a special one I want to talk about new this year with backfill. Stolee
introduced git backfill I think last year but it was for downloading everything that you're
missing initially. So the idea behind backfill is sometimes you may want to run multiple Git
commands, and you just know I'm going to be working on this set of commits or ranges. Or maybe you
know that you have a command that's going to end up doing multiple batch requests. So for example,
let's take log. If you're asking for diffs with logs, so git log -p, and you've got a number of
commits to work over, then when you go and say, okay, I want to see the diff between D and C, it'll
say, great, I know how to get those.
[03:08:59] I'll batch that request and get all the files within C and D that I need. And then
when it goes to C, it'll be like, oh, okay, now I need to go get all the files between B and C that
I need. And so you'll end up with one batch request per revision going back. The nice idea of
backfill is just hey I know I'm going to be doing a number of operations user can say I need commits
within a certain range. One of the issues and Stolee added that earlier this year I noticed it and
said commands like git log need things that are just slightly outside that revision.
[03:09:32] So A..D means all the commits in the history of d that aren't in the history of a.
But when you're doing log -p and you want to look at commit b, it needs to compare to its parent A
and A is outside the range. So we tweaked that. I went and tweaked that backfill command to include
edges by default. And now if you're going to run multiple commands, whether it's log, diff, cherry,
but you know you're going to operate on a certain range, you can proactively go and pre-request the
blobs that you need.
[03:10:08] But we're still not done. Even though we've improved things and we've partial clones
will help us batch missing objects and now we have a way to select history range and the new
backfill command will follow a sparse checkout. There are still other commands that don't have this
support and for the really large repositories is going to really hurt them whenever they hit one of
those commands. There's also a couple of issues there. Found one case where partial clone will
repeatedly go and download the same set of objects in a batch request.
Pushing less from shallow clones
[03:10:42] So, we need to work on fixing some of these things up. Partial clones aren't the only
thing where we've got an issue. We've also got an issue with shallow clones. So, shallow clones,
I'll show the same basic object model. The idea behind them is to cut history at a certain point in
the commit graph. So in other words, take this new commit C2. It's our shallow commit. This is with
the --depth flag. So often used with --depth=1, but could be some bigger number. And then Git
locally knows that C2 is a shallow graft.
[03:11:25] It doesn't know what its parents. It knows that in some ways that the code runs it
will treat it as this is the very beginning of history. In some places it'll be like okay I know
that there's other history I just don't know what it is. I need to stop at this point. And in a
shallow clone, if you go and make some small change like readme file, then same thing that I
mentioned in a previous graph, you've got just a few new objects in order to create just a small new
commit that changes one file.
[03:11:58] And you would like it to be the case that if you go to push within this shallow clone
that you just send those new objects up back to the server. We got a small problem though. Git
doesn't record that C2 came from the server that we cloned from. And at push time, what happens is
the server will say, "Here's all of the branches and tags that I have and the commits they point
directly to. If the branch that used to point to C2 that you did the shallow clone from has since
moved on, nothing will point at that and there won't be an older commit for it to be able to know
that it has in common.
[03:12:34] So when you go to push, in this case, Git says, oh, we don't have anything in common.
I need to push everything. And so what you may be thinking is, I thought I was only going to push a
few kilobytes of information, these three new objects, but I'm instead pushing potentially gigabytes
of information. And you're probably thinking, what a deal. Sign me up. I want that. But wait,
there's more. It gets even better. If you combine shallow clones with partial clones and sparse
checkouts, then we can double your network usage for free.
[03:13:11] Let me show you how this happens. So if you've got a partial clone, that means you
don't have all the objects right now. And so the push side of Git says, "Oh, I need to push all of
the history we have because we don't have anything in common." And then it says, "Oh, I'm missing
some of those objects. Let me go ask the partial clone machinery. Please go get all those objects
for me." And the partial clone side says, "Oh, I know how to get those objects. I'll just ask the
server for them." and it will go download all those gigabytes of information so that it can push all
that exact information right back to the same server.
[03:13:48] These are things that we need to fix, but this is a limited time offer. I've got some
patches on the mailing list. So, if they get accepted or something similar to them, you may not be
facing this anymore. But one of the ideas here is we really want to focus and change push just the
new stuff. If we assume that the server has C2, then we can push just a smaller set, we do run the
risk that the server won't have C2, in which case the server will just say you don't have you
haven't pushed all the objects that are necessary and you'll get an error message.
[03:14:24] But that's actually very similar to the assumption we're already making. Pushing
inside a shallow clone, the server has to have C1 and any other parents of C2 and all those other
bits of history. If it doesn't have those, it's going to send an error saying you didn't push enough
for me to know what how to connect things up together. And so all we're doing is shifting that
assumption by a single commit from C1 over to C2. So short version of my slides, walk less, open
fewer packs, batch and scope fetches, push less, add some hardening, but we've got a lot of things
in Git that we need to improve to make it be able to scale to large repositories that we have today.
[03:15:10] There's a lot of work to do. Any questions? [Applause.] hi there thanks for a great
talk. It's great that we are now solving this problem about the lazy fetching and we can batch them.
And like the batch the prefetch of a range seems like quite useful but I think what I've observed is
the CI pipelines that have been written have existed for a while and then the repo grows large over
time. So it's transparently switched from the full repo to a partial clone version of the repo.
Q&A
[03:15:47] And there are commands like git log embedded in it. So it's quite difficult to have
like all the consumers go back and then hey go ahead and do a prefetch beforehand. Is there a
potential for you know to for git log and operations like this to recognize that there is going to
be like if I did a prefetch here I will be able to optimize and make this much more faster or is
that like too complex?
[03:16:11] That's a good question. There might be I'm not sure exactly because CI can do so many
different things. What are the particular things they're interested in? What's the range of
revisions? One thing I didn't quite get this let me slide back real quick. Interestingly
git backfill when it's given a range of revisions is going to default to just getting the things
within your sparse checkout. But interestingly, things like git log aren't going to pay attention
to that sparse checkout. And so then it starts downloading all those files outside.
[03:16:47] I actually put a spec out a number of years ago, I think it was 2022, saying, you
know what, we should have other modes for sparsity. It shouldn't just be the checkout. It should
also be history related operations at least as an option. Some people like having all the commands
that operate on history operate on the full thing despite the checkout being sparse,
[03:17:08] But we should have a number of options there. Unfortunately at the time that work was
defunded. I was at a previous company, but I think we need to pull that back out and add those kinds
of things.
[03:17:21] Yeah. I have another question. The, you mentioned like corrupted commit-graphs,
right?
[03:17:28] What?
[03:17:29] Corrupted commit graphs. Yeah. So I think one problem that I've seen is a weird case
that happens with Git shallow where you do a Git shallow with a specific depth and again it's one of
those cases where you know the users of the repo on that environment are not aware if the repo is
full or shallow or partial and then they end up fetching specific commits by giving the commit hash
itself that's further down in the history and then the commit graph that gets generated out of this
is like the top three commits an entire stuff missing from the middle and then like one bottom
commit.
[03:18:04] So, do you think that you know there and this kind of broken commit graph can then
lead to other problems as well? Do you think there is a potential for recognizing that and then like
kind of any kind of fetch that happens this way that it automatically causes a unshallowing of the
repo happening to the point of the fetched commit or like preventing the user from getting this this
kind of state or do you think this is like an okay situation to be in and it's not a problem at all?
[03:18:32] I think there's a lot of sharp edges with shallow clones that we need to be able to
look at particularly as we get to larger repositories. A simpler one that Patrick brought up on the
mailing list, it was a really good point was just what happens when you go and amend a shallow
commit and Git is happy to suddenly turn that into a new root commit which is almost certainly not
what users want and will create later maintenance problems. But it's not just amending, it's
cherry-picking, rebasing. I've got some patches to fix that up that I'm hoping to send in soon.
[03:19:01] But there's those. As I've been working on that as well, we I found a number of error
messages that are just awful that will make you think things are corrupt even if they're not. And so
it's like, okay, we've got to fix up a number of these sharp edges. Shallow clone in some ways can
feel half-baked, I guess. And I think it's an area that we need to extend and improve.
[03:19:28] Yeah, thanks Elijah. Like we hit most of the issues that you described in the talk.
So really really appreciate your work in this area. I actually have a question about h how to get
these improvements to customers the fastest. Given we talked about server side, I'll actually focus
more on the client side. For example, this faster the merge-base algorithm that you described which
is clearly helps many of your customers. The how are we getting this in the customer's hand? Is it
will it take months and years until it's merged or there is a way to have a patched Git clients that
the GitHub is distributing to the customer so that they more immediately feel the benefits of it.
[03:20:11] So can you talk about like rollout strategy both on the server and the client side
improvements? The server side's a little bit easier because those who are running the server control
it and can say hey I want like I rolled out that improvement fairly quickly. The client side is a
little bit harder but hopefully those who have the largest repositories will be updating more
frequently but there is certainly the issue of that merge-base improvement. No one really has it yet
unless they're working on a development version of Git version that's going to be released later
this month will have it.
[03:20:45] And then there's a question of how soon do they update and we don't have control over
their environment. We haven't investigated trying to run our own fork. I don't think we want to do
that. We'd rather work closely with upstream and have them send out the releases and everyone pull
from there so that we're all still collaborating because we've got great people working from a lot
of different companies. We want to continue working together.
[03:21:14] Thanks.
[03:21:15] Regarding getting these features out to customers more quickly: at
least for the part where we are pushing too large a pack because we did not realize that the server
contains commit that we have.
[03:21:31] Yep.
[03:21:31] There is a feature:
[03:21:34]push.negotiate=true.
[03:21:36] Sorry.
[03:21:37]push.negotiate=true. Is that what you're referring to?
[03:21:39] Push negotiation.
[03:21:39] It ameliorates this problem, but there are at least two cases
where it doesn't solve it.
[03:21:45] Mhm.
[03:21:45] Well, there's a bunch of additional cases where we could eventually fix to help it
solve in those cases. You have to have a v2 fetch server as well, and there's various issues with
that. But if you have a case where you're pushing to the wrong server, you want to be able to get
back an error message saying, "I don't have the necessary objects. This won't work." And you don't
want to have to push gigabytes of information in order to get that error back. And push.negotiate's
never going to solve that. Second, another issue is actually with quick object lookups.
[03:22:24] So, if you're repacking on the server and you put everything into smaller number of
packs, upload-pack is going to do quick object lookup saying only the packs that I knew when my
process started so it doesn't have to go relook for a whole bunch of packfiles. And the idea is it
might miss some of the commits that you have in common, but that's okay because it'll just push a
little bit more history is all that's needed. But the problem is with shallow clones, you miss those
commits that you have in common. So just the fact that you have this quick object lookup idea inside
git upload-pack means we're missing something.
[03:23:02] It's not going to work always with push.negotiate=true. So I think we need a
backstop. But great question.
[03:23:10] In fact, everyone I've talked to about this so far has brought up push.negotiate. I brought
it up. I originally thought it was a solution, but each time I've gone to dig deeper, I found that
there are some issues with it.
[03:23:22] I think that's time for questions. Thank you very much. [Applause.]
[04:28:24] Joke time. All right, for those of you here, welcome back. I'm just going to tell a
couple of dad jokes because sometimes if it sounds like the conference is starting, people come back
faster. So, I kind of want to make sure that my voice gets heard just generally in the main room.
And, look, you can already see people going faster. Oh, yeah. Actually, that's a great one.
Everybody pretend that I just said something awesome and do a huge round of applause. Yeah. You're
welcome. You're welcome. I know. God, I hope nobody missed that.
[04:29:00][Laughter.] let's see. My friend Tony told me not to say his name backwards. I
said, "Y not?" It's Tony backwards. That's Tony backwards. It's a thinker. Maybe better visually.
Let's see. I'll go a little bit more racy this time. What is worse than having ants in your pants?
Having uncles in there?
[04:29:34] No. Is that one not okay? [Laughter.] That's fine. Sorry about that.
[04:29:41] Oh, is it? That's fine. All right. I'll give it one more minute. I'll give it one
more joke and then we'll get we'll get started here. Can't tell that one. What did the sushi say to
the bee? "Wassup, bee?" I like that one. I like that one. I think that one's cute. All right, we're
going to get started. A couple people still filing in, I guess. The lunch was quite heavy on people.
So, first of all, I want to introduce Christian Schilling. Christian was a great help.
[04:30:21] When Taylor texted me on Sunday morning saying I can't make it, I lost a co-host, but
also I lost a speaker and so it's a little bit difficult with three days to go to find a speaker
with a talk that you really want to listen to. Christian had applied to speak and I reached out like
two days ago and I was like so here's the thing. [Laughter.] How fast can you make slides?
[04:31:01] And he said, "Yep, no problem. I'm there." So a huge round of applause for Christian
for coming in at the last minute and helping us out. Thank you so much.
Why Josh exists
[04:31:12] Hello, welcome to the test run of my talk. It's about an open-source project I've
been working on for about 10 years now. The project is called Josh. I'm not. A bit of background on
what I'm doing: I'm mostly an embedded software engineer. I work on C++ codebases for automotive
embedded software, for electronic control units that are more resource-constrained: real-time
operating systems, device drivers, protocols. Most notably, recently a lot of the embedded code has
been open-sourced through the Eclipse Foundation as Eclipse OpenBSW, under the software-defined
vehicle umbrella.
[04:32:14] But I also work on developer tools in Rust. And the big thing I'm looking there is
the Josh project which I'm talking about today. So what is it? So it's was created as a like
solution to a problem we had working as a supplier to like a bunch of different car companies. And
the problem was we really had like our own codebase and really wanted to use a monorepo to be like
efficient in working in this codebase. I think don't need to explain why this why this is good.
[04:32:53] But at the same time we also wanted to share part of that code with all those
different like customers or like collaborators. And yeah, we tried different things that didn't
really work so well. And in the end, I said like git subtree seems to be close to what I want, but
it's like so slow. So I rewrote it and discovered some things afterwards. And as it turns out, this
like tension between making one big repo and making multiple repos is found in many places, not just
in this like automotive bubble.
[04:33:34] And like the biggest case study you can like look up online is the actually the Rust
project. So they had the same kind of similar problem. So the Rust compiler team works in addition
to the compiler also on a bunch of tools like there's like Miri there's rust-analyzer there's like a
bunch of them and they all they also really want to use the monorepo to do like changes spanning all
the tools but they also want to have like autonomy of the different teams that that work on those
tools. So they also tried other things like submodules subtrees eventually also figured out it's too
slow.
[04:34:24] Kind of impressive how far it they took it. They even patched subtree at one point.
But still eventually hit a wall where it was would just not work anymore. Oops. So Ralf the
maintainer of Miri actually tried tried out Josh and it worked quite well for them since and yeah
they basically switched one one of one project after the other over and still switching over some
more projects. On the flip side, this has also been a really cool source of like edge and corner
cases and like performance problems that we found because the Rust history is not just big but also
has lots of interesting cases and like how the history looks like.
[04:35:17] It's very wide. They use branches quite a lot. And there's a blog post that they
wrote. You can read it up. So what does it do? So in the end it's really we like virtualizing the
actual graph of Git. So we like we know like Git is like this content addressable DAG. And what Josh
really is is a set of transforms that forms like an algebra to transform the DAG as a whole. So
we're not just doing tree transforms. Like treating the whole DAG as an object that can be
transformed in one way or the other.
Reversible projections of Git history
[04:35:59] That means the result of such a transform is a completely self-contained history. It
the nothing that we do here has anything to do with really working trees or index or something. And
because those like products like virtual DAGs are self-contained, we don't we don't we don't like
recentralize Git. Like there's a lot of approaches where like you download part of the repo and
still have references to files that you haven't downloaded, but that kind of means you still need
the like server to eventually get those objects.
[04:36:43] So the rep is kind of incomplete. So we avoid this and what you could call this is
actually materialized views that are like Git native and distributable. So you can actually share
those projections with others. They can work on those contribute back and yeah we have like
formalized the bidirectional relationship between the differently shaped graphs. So in this example,
we say there's like a monorepo on the on the one side and the project repo might look like this and
we can exchange the changes. To do this we have a filter DSL.
[04:37:25] So like kind of made up a language. There's like four operations I'm showing here.
There's like we can select folders while changing the location in the tree. We can do globs. We can
excludes. We can like compose different operations. We have about 30 something basic operations that
you can like compose more complex ones off which seems like a lot. Some reason we ended up having
them. The key is really this like reversible. So we can take an expression in this language and say
invert this and you get the an expression like in the same language but that just describes how you
how you need to map back to your original.
[04:38:17] We're experimenting or we it actually works but we still need to commit to final
design of using checked in Starlark files or Wasm blobs to compute those those filters on demand
based on content in the repo. So maybe it wants to look at build files to determine dependencies to
see like what you want your your projection to look like. We have wrapped this in a CLI to use
locally and we have a Git proxy so you can basically have all the filtering done remotely. So if you
actually want to avoid downloading stuff by filtering you need the proxy.
[04:39:04] Like I said there's a bit of algebra you can do with those expressions like like a
subtree of a subtree is the same as doing them separately. Yeah we can compose there's some like
rules like you they're associative and distributive they're not commutative interestingly and we
have this whole like inversion thing. So the basically the first line of invert is like kind of how
we define it like if you apply the filter, its inverse, and the filter again you end up with a
filter.
[04:39:43] And there are some rules how to invert like composite filters that we use. So this is
one of the things I'm getting asked most often about how do this how does this invert like this
reverse path actually work. I'm just talking about trees here. So we have like like a like a
monorepo and there's a tree like abuse the what's it called when it's in the up the superscript for
the leg monorepo and the subscript for the projection. So we have a tree we apply filter to it then
somebody modifies it so we get like a new tree and the question is how do we get the corresponding
tree in the monorepo from from that.
Mapping changes back
[04:40:45] So the one idea would be we could just invert the projection we did. That doesn't
work because the like filter just selects a part. So the actual the rest of the tree is actually
missing. So we would would we create a commit here that that yeah misses most of the tree. So we
need to combine it with the tree we already have like the previous tree in the monorepo. Also
doesn't quite work yet because if the commit if the new commit actually deletes files then we also
need to delete them in our new mono commit.
[04:41:30] So what we're actually doing is we're taking the commit from the monorepo, excluding
all the files that the projection could have modified and then just adding back the ones that it
actually still contains. So this is the way this is a this is like I said just the tree. There's
quite a bit of more logic needed to get the like topology of the commit graph right for this. But
yeah, it's a bit one of those things where we where we uncovered interesting edge cases while
working with Rust repo.
Performance and incremental filtering
[04:42:13] So I said like we it's much faster. So the question is how much faster is it than
what other what's out there like git filter-branch is kind of the old everybody knows it's slow
but I just wanted to have it here as a like for reference. We sometimes get compared to Copybara. I
don't think the comparison really makes sense if you like this is a logarithmic scale. So we are
like three orders of magnitude away from this. Like the only tool that is actually in a similar
realm is is the one Eli wrote like the git-filter-repo.
[04:42:53] But this is like a small repo. This is our own. It's like two and a half thousand
commits. And if you like extrapolate this to like really big repos, it's still not quite fast
enough. So what we really need is is actually incremental filtering. So that the work we do is not
dependent on the repo size or the history size at all anymore, but just on the amount of new commits
that are coming in. So we kind of do a cold start once and from that point on we can always continue
where we left off and yeah this is not fair because git-filter-repo does not have incremental mode
but it's [Laughter.] it's just the yeah the amount of time you save by doing this incremental is
really big.
[04:43:45] So this is on the Git repo which is like around 80,000 commits and incremental here
means I filtered up to like 100 commits before head and then the rest so to do like a catch up. So
the question how how did we make it so fast? And I can't stress this enough the way really to get it
fast and this is I think also why git-filter-repo is fast. We really this is very specific to Git
the way we implemented this. So it's fully embracing the model and the data structures how Git works
and there's like no attempt to somehow abstract it or make it backend agnostic or anything of this.
And this is yeah how we can avoid most like a lot of work and yeah apart from that it's processing
things incrementally having having caches in places using algorithms with like like we found also
with going to bigger repos things like quadratic explosion and stuff and fix this.
[04:45:13] Yeah, just to like avoid looking at as much of the graph as possible for whatever we
do. Then we have kind of collection of like known fast operations that we like decompose the more
complex operations into. So it's kind of almost like I showed you filter algebra. It's kind of like
almost tree algebra below it. And everything is implemented directly on the DAG and no no
abstraction above that. And then the third pillar, this is what we did like a lot of this actually
in the last year is really going like low on the lower levels of performance doing more things just
in memory.
[04:46:02] Deferring IO looking at like where we like don't need to decompress objects because
we're just interested in the header like stuff like this. Oh, I'm way ahead of time. [Laughter.]
so I'm already at the recap. Yeah, so we are shipping a really fast reversible Git filtering engine.
We have created a language to generalize this beyond just simple subtrees of a repo. We have with
rust-lang a pretty good benchmark and proof that this actually works for repos of interesting size.
[04:47:00] And yeah, we know about some deployments that use way way more complicated filters
which also drove the recent work we did on performance on repos that are way way bigger than what
Rust is. So I can't tell you which one. The so what are we looking at to do kind of next? So what
we're thinking is that this having this like formalized relationships between the like differently
shaped repos or graphs we can really rethink how distributed collaboration work because the like
assumptions that were made before that this is really expensive like led to a lot of decisions down
the road like how forges work, how CI works like a lot of things are based on the assumption that we
have an expensive thing here and it's not expensive anymore.
New collaboration possibilities
[04:48:10] Then we've seen Scott's talk and it's quite interesting that a lot of there's like a
trend to move more data into Git. And as soon as the data is in Git we can actually apply everything
is built to that data as well just as well as to code and there are a lot of other things of or
other kinds of data that we want to like collaborate on be not just code. And lifting some of the
restrictions that that Git kind of imposed on the like collaboration model. We might unlock ways to
use it for completely new kinds of data, completely new domains.
[04:48:58] So for closing, I'm using like a quote I really like from Linus. It's from from his
talk about about Git that probably everybody has seen. He gave it back in the very beginning at
Google. Basically what he said back there is that like previous systems had this in previous systems
like branching and merging was really expensive. So people like avoided it at all costs like it was
like you have to plan for it and do it only when you really need it and stuff like this. And Git
made branching like really cheap.
[04:49:32] So we have lots of branches now. In a similar way actually he said it in the same
talk filtering is also very used to be very expensive so it would always be avoided nobody wants to
do it. But once the thing that used to be expensive becomes almost free. You can suddenly like use
it in a different way for different purposes. And that is kind of I think where we going here. And
I'm already done.
Q&A
[04:50:21] Thank you. Sorry I this may be a stupid question because I have not used Josh. I
understand the algebra and the fil like the sort of filtering logic, but I'm curious if you could
tell us just quickly how is this used? Like is this a server side thing that you clone from a
different URL to get a filtered subtree that you've set up the logic on and then you push back to it
and it does the reverse to the main repos. But it's a server like this entire or is there local
tooling that you're using?
[04:50:49] We have both.
[04:50:49] Okay. So it used to be just a server thing for the first I don't know six seven years
or even more but we now have a local CLI as well. What the CLI does not do yet, what we definitely
want to implement is doing this like partial clone things before the filtering so that when you like
like currently when you when you do the local filtering you would download the whole repo apply the
filter and then get like a ref which is the is the filtered view of it. But if we already know what
the filter is, we could use like Git mechanisms to avoid downloading parts.
[04:51:32] So currently if you want to save save the space, not download stuff, you would need
the server side, which we also have, which is basically a proxy server like you point it at a URL
and it will just basically we use the URL where you say server, repo.git and then colon then the
filter expression Git again and then you kind of address inside to the inside of the repo.
[04:52:03] Cool. Thank you.
[04:52:05] Hi. Very excited about the project. I follow it from day one. You guys open source
it. Very good project. I do have a couple of question. First of all, I'm very excited about the
Starlark experimental work. I was wondering if you guys can comment anything about how this would
integrate with CI, right? Because obviously you are composing multiple repo into a bigger repo and
then you decompose the bigger repo into multiple repos when you do that CI in the other repo that
you are syncing to can break. Yes.
[04:52:41] Right. If you can comment on that and then the second thing I want to ask is have you
guys explored transformation that involve generative AI as opposed to coding agent right because all
you have here is imperative transformation and that's may be nice but you know there are some use
cases that I think can be unlocked with generative transformation thanks
[04:53:07] Okay so two questions so the first one about CI is we it's another experimental
feature that we currently use and we use it for the CI of the project itself where we like define
the different parts of the CI in terms of filters. So basically we say like you filter the tree in
this way and you get this this is like the working set of a particular job in the CI and then you
make a graph of this. You could think about it as like more coarse-grained thing than a build
system, but kind of similar.
[04:53:40] So, yeah, we have this already. The other question, generative AI, I'm not sure I
really understand what you want. [Laughter.] so the more complex filters we have and also like the
users we know of that use more complex I think is mostly generated by AI already. So basically
[04:54:11] The use case I was having in my head is let's say Stripe were to update the API right
and then I can have a coding agent follow a set of imperative prompt that I want to apply the
transformation into my monorepo right and I just want to listen to the event from GitHub that hey
Stripe is publishing a new set of API do something about it right and I do want to leverage
something like Josh to do those transformations. That's what I was thinking about.
[04:54:41] Yeah. I mean, I think there's like two sides. One is you would want an AI to be able
to query like a subset of the repo in this way.
[04:54:53] Yes. If you think if you think of you know an external repo as part of your monorepo,
right? And then a change is coming into that directory. Let's say third-party Stripe API right you
got that change you want to take that change into your monorepo but at the same time you want to
transform that change into your application code yes
[04:55:18] Right to react to that event I think Josh would be a very good fit but
[04:55:23] I'm not entirely sure how it going to fit into the architecture at least for now
[Laughter.] so I'm just throwing it out there so that you can consider it but yeah thanks
[04:55:32] Yeah I mean what you're describing is exactly what it's used for. And yeah, it can be
combined with a generative AI. It's just not we don't need AI inside of the project. You know, it's
like we you can generate the filters with AI. The agent can use filtered representations of repos.
Which is also one interesting use case because you can like shape like context shape by doing this
like subsets yeah but we don't need to have AI inside the filter that's what I mean I knew about the
service part before but like the client side how does it differ from git-filter-repo
[04:56:21] What do you mean
[04:56:22] Like the client side Josh
[04:56:24] Yeah like how does it differ from git-filter-repo?
[04:56:28] How does it differ?
[04:56:29] Yes.
[04:56:31] They have a different purpose. So git-filter-repo is more for like it is in the
things that it can do. It is more general because it can do basically anything. You can even call
out to a script. So the idea of git-filter-repo is really you have a repo you want to fix
something about it. You filter it and basically throw away the original. That's kind of the thing
what this was made for. Whereas what we're doing is like you have a repo and you apply a filter and
you just get a ref in the same repo and the ref represents the filtered version and then you can
work on that ref and like put your changes back to the original.
[04:57:22] Yeah, so it's the whole like having both both both the original and the filtered one
at the same time. That's something that git-filter-repo doesn't do in exchange for some other
things that we can't?
[04:57:46] Okay, good.
[04:57:49] Thank you very much.
Checking out another branch of Version Control
Speakers: Raphaël Gomès and Pierre-Yves David.
Recording:04:57:50.
Introduction
[04:57:50] Thank you. [Applause.] Oh, now we have two mics. I'm double-miking it. You could
probably take this back. All right. Our next speakers are Raphaël and Pierre. Oh, they're on their
way up setting up. So, I met Raphaël and Pierre, I think, at FOSDEM a couple of years ago. And it
was one of the fun things that I hope you guys are having these types of conversations here. It's
why this is my favorite conference to do: there's a whole bunch of Git nerds that love talking about
Git, and that's relatively rare.
[04:58:27] So at FOSDEM there are lots of nerds. But there's only a few Git nerds and so we
found each other at a bar from some of the afterparty thing or something and it was us and who was
the other guy the patch
[04:58:43] From Pijul
[04:58:45] What is it? Pijul,
[04:58:51] It's easy to pronounce. Anyways, theory of patches. And so yeah, it was us just
sitting in a corner booth drinking beer talking about nerdy Git stuff and they will tell you who
they are, but they have been maintaining the Mercurial project for a long time. At Git Merge, I love
trying to get people in who are not just "I use Git all the time" -- to talk about Jujutsu, to talk
about other systems out there, to give us good ideas of what we could possibly steal. So with that
said, please a big round of applause and welcome.
A parallel universe of version control
[04:59:30] Thank you. Hi everyone. So I'm Raphaël Gomès. This is Pierre-Yves David. He's my
coworker. We come from a parallel universe of Mercurial. So, thanks again to the organizers, Scott
and the others, for letting us speak a bit about Mercurial and how, it differs from Git and why we
think you should care. We are a small consulting company called Octobus. We specialize in Mercurial.
So I will give a very brief overview then, give it back to Pierre-Yves. We're going to have like a
tiny glimpse into the Mercurial model and a couple of differences with Git.
[05:00:10] We only have so much we can fit in 30 minutes. There's no way we can talk about every
single thing. The differences we're going to talk about are small but have pretty major
consequences. Given that both projects were started at around exactly the same time, by around
exactly the same people, it's interesting to see how much it has exploded and cascaded. We're going
to talk about the workflows that the model enables that are unique to Mercurial. We're going to
hammer this word all the time: you're going to be sick of hearing about CRDTs. Maybe you already
are, because other people have been talking about CRDTs, but we are going to talk about them a whole
bunch. In the second part we're going to talk about scaling insights: some stuff that Mercurial is
quite good at and might be interesting for Git people to look at. We sometimes outperform Git 1 to
80 in content storage for certain cases. We can fit a repo with more than 20 million -- actually,
it's like 25-something million now -- file-commit revisions on a single dev box, and that's just
fine.
[05:01:25] We have petabytes of data that achieve that kind of compression in certain cases and
we can do microsecond level diffing of the aforementioned data. So this is kind of a small spoiler.
I will hand it back to Pierre-Yves to talk about the data model now.
Distributed models and CRDTs
[05:01:46] Yeah. So I'm going to talk about data model. I'm going to cover stuff that you will
be familiar with like the Git data model, but I'm going to probably cover it in an angle that you're
not familiar with. So it's going to be kind of theoretical, but please go on and stay awake. So plot
twist, let's talk about Subversion first. So go back in 2010 for the one that could actually hold a
keyboard at that time. And in Subversion you have a central server with a linear history. Your clone
are just a working copy.
[05:02:22] And when you do a commit it's happen directly on the server. You can have multiple
clones. So you have concurrent edits and because you can have concurrent edits you have to deal with
merging. Doesn't mean that you will have conflict. You can have workflow where you don't have
conflict but you have to do merging at some point. When people do concurrent addition it's an
important point when you enable new things. There is consequence you need to deal with and so in
Subversion you have a central server. So you have a pre-commit merging model where if you lag behind
you have change you need first to merge with the things that have been put upstream and then you can
finally do your commit and things like that.
[05:03:00] It was nice in 2010, but everybody here knows that we can do much better nowadays,
because we have a better data model. Let's jump back a bit to 2005, when Git and Mercurial appeared.
Now everybody has a full copy of the repo. The commits are no longer created on the server: they're
created locally, and you're going to merge after you've committed. Here I made a local commit; there
is a remote commit. When I fetch, I get them, I'm going to do a merge, and it's going to be fine.
[05:03:33] In early Git, the UI was not very different from SVN. It was a different data model,
but the UI didn't have to change to deal with a richer data model. git pull is literally update
and merge: get the changes and merge them at the same time, in a single command. But because you
have a richer data model that can express more things, you don't really care how you do that. You
could do a commit then a pull, or a commit, a fetch and a merge, or a fetch, then a commit and a
merge. It doesn't matter from the end-result perspective. That flexibility unlocks a lot of things
that we know nowadays: new workflows that were not meant to exist when Git and Mercurial were
created, but were unlocked by better capabilities.
[05:04:24] For example, you could merge later. You could do like string of commits that makes
sense, rebase, history rewriting in general and stuff like that. And it's all exist because the data
model and to some extent CRDT CRDT is an fancy name like it's conflict-free replicated data type for
something that have existed for a long time. It basically describe distributed database that have
three key property. So each replica can have concurrent updates. Here you have the left one that
adds something and the right one that adds something else independently. And if there is
inconsistency like different updates you have an automated way to not break like it's not going to
crash the system.
[05:05:10] You can still do synchronization. Whatever the order of things, and however you
synchronize everything, every replica will converge to the same state. You're familiar with that
already: in Git, the object storage is a CRDT. The commits and all the objects associated with them
are created in a distributed way, on everyone's machines or anywhere. You can extend them in any
order; it doesn't matter where you got a commit first.
[05:05:45] The storage is fine with duplicated objects and stuff like that. In Git, the object
storage is fine. But if you only look at the objects, you have a soup of commits that you cannot
really make sense of. At some point you want to name things, to do code review or deployment, so you
need a way to look at the soup of commits in the object storage.
[05:06:10] This is where Git goes off-road regarding CRDTs, because Git branches are not CRDTs.
The way you see commits is by pointers that reference them. Those are only a local mapping. When you
exchange them, the last write wins, which is definitely not CRDT, because it depends on the order.
The way you exchange them is very configuration-dependent: the name you get depends on how you name
your remote, on tracking, on various things you do. They also have properties that you don't
synchronize. You have a branch that tracks something else; if I pull that branch, I don't pull the
tracking information either.
[05:06:54] Git branches do a lot of nice things, but they are not CRDTs. On the other hand, in
Mercurial both the commit graph and the branches are CRDTs, because branches are inside the commit
graph. The branch information in Mercurial is part of the commit itself. It's part of the commit
identity. It's part of the data, and therefore identity. It's basically graph coloring: you assign
commits to branches. You don't have a pointer that says "here is the head of the branch"; you
compute the head of the branch by looking at the graph. It also means that a bunch of weird things
that you have in Git, you don't have. For example, I can move anywhere in the branch and it's fine.
I'm not in a detached HEAD situation, which simplifies things for worktrees and things like that.
Branches, divergence, and convergence
[05:07:47] It also means the content of a branch is explicit. If I have multiple branches
stacked on each other, for example for doing merge requests in Git, if it's linear it's fine. In
Mercurial, the first one has a color and the other one has a different branch label, another color.
But when you start to update things naturally -- for example, someone makes more changes on the
foo branch in Git -- you have an ambiguity. Is the base commit in the foo branch or the bar
branch? Is foo on top of bar, or bar on top of foo? How do you organize them? This is clear
in Mercurial: what's in the branch, and the relationship between the branches. It also means you can
do partial synchronization at the commit level.
[05:08:46] The basic unit in Mercurial is not the branch; it's the commit. With the repo I was
talking about, I can send just the bar branch to the left repo when it makes sense, and the
content of bar has not changed. There were two commits in it; there are still two commits in it. I
can also just push part of my branch. Maybe some changes are not ready to be shared, or I want CI on
a specific substate. I can do that, and it all makes sense at the model level. Nothing is freaking
out. Since this is all CRDT, at some point everything will converge to something that makes sense.
There is no bad state; you can represent all of that.
[05:09:30] The last small thing is like what happen when people do do concurrent work on the
same branch then you just get two heads like you compute the heads and you realize that you have
multiple heads on that branch which is fine the model is perfectly fine with it and you can solve it
the simple way is to do a merge because now you only have one head remaining on that branch and it
maps reality quite well because in practice you already had two head on that branch. It's just that
they were not in the same repo.
[05:10:03] There was two heads on the branch that were in different repo and now the your repo
see both of them. So in Git you would have two head with two branch with one head and in you can
have a branch with any number of heads and things like that. It's fine from the model point of view.
So you can have that locally you could push that. So, if you don't know how to do that merge or
you're about to jump in a plane, which doesn't really work anymore, but like you can push whatever
state you have, it makes sense.
[05:10:33] You can exchange it with people. Of course, head a branch with multiple heads is not
going to be great for like working with it. If you want to do CI review, landing, and stuff like
that. So, as a as someone using source control, you want to fix it at some point, but you don't have
to fix it right away. Like you're not in a in a panic state where it needs to be fixed. And why does
that matter? It's because you have many way to get into multiple head. Maybe you pulled an extra
head and you already had one or maybe you pulled multiple head already or maybe you moved in the
middle of your stack and did a commit there.
[05:11:10] You could have work trees with multiple things like you're working in multiple on the
same branch and different work trees. You could remove a merge like I'm not going to go into all of
them there is many way to get multiple heads and we keep adding new workflows and stuff like that
and this keep adding new way to get into that state and we also keep adding way to get out of that
state like we you can do merge but you could also like fold the two commit into one commit or prune
one of the two ones do rebase move another one on another branch like you have many way to get out
of this situation And if you have a specific concept to handle each of the way you get into multiple
heads and multiple way to get out of them the amount of case you have to handle is growing
exponentially whenever the things become more complex.
[05:12:07] Well if your model have a simple way to represent something but have multiple way to
happen like you just have multiple head on your branch. This mean that you can keep adding way to
get in that state. It doesn't make things more complex for everybody and you can keep adding way to
get out of that state and you're not having that explosion. And here we just did that for multiple
heads. But we have many state that we need to represent and in Mercurial we track various kind of
data and so when you start to combine a lot of different concept together you want them to be simple
and combine well together instead of having gigantic table.
[05:12:47] So instead of having a big fence of case you want a nice butterfly flying around. And
in Mercurial everything is CRDT like the commit object but the tag the way we deal with immutability
the branches both the long-terms and the draft one the branch life cycle like everything we have is
still CRDT compatible with small way to represent state especially rewriting in Mercurial is a CRDT
and I'm going to hand the stage to Raphaël for that Y so sometimes we talk about history rewriting
as being history editing.
Distributed history rewriting
[05:13:28] In practice it is not editing it is rewriting because in both Git and Mercurial the
hash is derived from the content and its metadata. So what you do when you do an amend for example
what happens is that you actually destroy the previous commit and then you create a new one. What
destroying means depends on on on exactly what your model does and we're going to discuss this
because you cannot throw away data in a CRDT. So how do you do it? Simple case you have a linear
branch you have the last commit.
[05:14:00] You just want to get rid of the last one because that's happens. In Git you what you
would do is you would just move the branch pointer. However, and everybody in this room knows what
I'm going to say. If at any point this has been observed at anywhere else then you're going to have
a problem because the commit can reappear on fetch. The branch may have been otherwise updated. So
you can you know force push with whatever depending on on what Git came came up with to work around
that particular problem.
[05:14:36] Or you can create workflows workflows that make it so that you don't fall into that
problem. Let's say you don't push other pe you don't touch other people's heads. You don't you keep
everything very segmented. But we say that there's a better way. Another clone may have the commit.
Another branch may have the commit. So even if you try to like move it back down, another branch
might keep this commit alive. And so it's that's not a way of propagating a deletion. But it's not
just about propagating deletion. There's more complicated workflows.
[05:15:07] What happens if your two green commits are based on things that have been rewritten
since? This is something that can happen in many different ways in both human workflows and agentic
workflows. You have concurrency problems. What happens if it's been rewritten in two places at once?
You also have immutability problems which is in Mercurial we have phases. We have like commits that
are immutable and others that are not. And so if you if there's something you cannot rewrite or
shouldn't rewrite, then there's a problem, but it's something that can just happen because it is a
distributed version control system.
[05:15:46] So how does Mercurial handle that? We do that through obsolescence markers. So every
single red arrow include excluding this one, sorry. Every single red arrow in the previous
schematics are obsolescence markers. Their extra information that we add on history rewriting that
is stored separately from the commit graph. It's not inside the commit. It's something that is
separate. It is a CRDT. Big surprise. And that means that it can be propagated. This information is
not local to your clone or it is something that can be propagated to everybody else. And so the
overarching umbrella feature is called changeset evolution.
[05:16:32] It was starting out started out in 2011 by Pierre-Yves here. And so it's basically a
story of why and how the commit is obsolete. It's history of your history, if you will. It stores
metadata about successors of a given commit. Split and fold information, which is something that a
simple change ID cannot keep track of. The who, the when, the what. Basically, it's rich information
that allows you to go into your history and figure out what happened and also enables great
automatic resolution. So, let's get to an example.
[05:17:12] It's going to be maybe a little bit more of a complex example, but I think it's worth
it to explain what happens. It's up. [Laughter.] Sorry.
[05:17:25] Can I please one sec? Maybe it's a resolution problem. I can probably figure it out
somehow. Yeah. Nope. Well, I mean, you can imagine it maybe. So, no problem. We have three commits
inside a draft branch. It's what we call the topic. It is those three of them I've sent them for
review or I've given them to an agent, whatever. I given them to someone else. And let's say S1
needed to be amended because there's a typo in it. And then it needed to be moved to the default
like to the main branch because it's okay.
[05:18:04] But S2 and S3 are not okay. And so there's a partial move of what you're doing. What
that gives you in Mercurial is a trail of obsolescence markers that shows you, hey, currently
[dependent commit labels unclear] are what we call orphaned. They're based on top of something that
was rewritten. And here's the chain that you need to follow to make sense of that. And so if you're
a user, you can see that in a GUI or if in a in a graph thing, but you can also ask Mercurial to
explain what's going on.
[05:18:39] You can So we have an obslog command. We've had it forever. Maybe it r it feels
familiar to some of you. And it Git it says that someone else, not you, the victor rewrote the
history, which apparently happens a lot of times. And it tells you that this commit was amended
first into this hash and by this person at that time and then it was rebased by the same person at
another time. And so you don't even have the commit in the middle. You don't care about it. You just
follow the follow the chain.
[05:19:10] And so you can ask about the patch like what exactly was changed. It is indeed an
amend of the fix. And this works through the rebase. And so then you can just ask Mercurial to
automatically fix this problem. This is an easy problem to fix because it's basically okay well my
base was was like moved so I can need I can just rebase automatically. This is one command in
Mercurial that does it for you. We do that we do this all the time. We use it. It's actually great
to use and every time I cannot use that feature in other software I get mad.
[05:19:44] You we have commit message diffing. I guess that's cool. The thing I want to bring
home is what it enables in more in other cases in on the left. What the obslog gives you is the
difference that is exactly the difference that is contained in the patch regardless of all of the
rebase stuff that went around it. I know that it this is so this is another change but I know that
this is exactly what the change did and on the right this is what a normal diff would give you on
that range and so you know what part of that is the relevant part this is what obslog gives you
and if you happen to have a rebase conflict sorry in the rebase well you can just see how the
conflict was was done and this is was done by somebody else not you an agent someone wherever this
is fully distributed mutable history which is a feature unique to Mercurial.
[05:20:41] There are as we are aware there are no other version control systems that can do
that. We even used it yeah to create those slides. It uses just a small number of concepts built on
top of the existing Mercurial stuff and it has been pretty future proof in the sense that every time
we've added new use cases it always just fit that model. Meaning that some distributed version
control systems are more distributed than others. And it especially makes very much sense today with
all of the agentic stuff that is going on that everything is being rewritten all the time by a lot
of different things at once in a fully distributed way.
[05:21:21] So enough about modeling and everything. Let's talk performance. So we can go into
full details about things but Mercurial and Git are very similar at the basic data schema stuff like
we have commit that point to something that all the file directory structure like trees and that
point to file content and all that is identified by the hash of its content like very much like it.
We do have some differences and especially we Mercurial tend to track a bit more data like the
rename of file explicitly and history of file and a bunch of things but like we only have 30 minutes
and we already spent a lot on modules.
Flat manifests and storage
[05:22:06] So I'm going to focus on one small thing that is not visible to the user the
difference with how ml store the file directory structure. So in Git you have trees. So a commit
point to a tree is a tree point to file something. I think you already have like three explanation
of that already. So I'm not going to go into details. But in Mercurial we don't have many trees. We
just have one manifest. And the one manifest is a full flat list of every content associated to
every file path not a small subset of a directory.
[05:22:40] At first glance it looks suboptimal, right? Because you don't have the path-prefix
compression, you're missing some opportunity to do diff shortcuts, you have more data to handle
overall and a bunch of thing like that. So it seems like the Git model is probably best for that.
However, there is like pro and cons for both. But there is definitely issue with the manifest model
and because there is definitely issue with the manifest model we got problems with it quite early in
the project and so we have been improving the way we handle manifest for over 10 years now and the
result is actually not bad.
[05:23:17] So here is a comparison of the disk space used to store tree in Git and manifest in
Mercurial for various public repo and despite the handicap of the manifest on the worst case we are
about as good as Git and in some case we are quite much better let's zoom to the Homebrew-cask case
the Homebrew-cask case in Git the tree take most of the space used by the repo but Mercurial is like
one to seven more like seven times more efficient to store the manifest. The commit are weird. I
know why in Mercurial and we have a plan but I don't know why Git is also bad with the commits.
[05:24:01] So we should look at that at some point. And so in some case the manifest despite its
disadvantage is much more efficient and it matters for much larger repository like for private repos
repository we deal with like 20 million revision 26 million revision the tree like approach people
tried some stuff takes about two terabyte to store but the manifest is only 1.3%. So with a
tree-like approach it's impractical to imagine you're going to do like a full clone normal clone on
on a dev box like the two terabyte it's like server territory but with the manifest you can
definitely use this kind of like with the manifest smaller you can definitely have a full copy of
the full repo on developer machine and do normal workflow with it.
[05:24:53] You the full size of the manifest if you look at all of them like uncompressed and
packed is is 1.6 petabytes and so the resulting compression it's 60,000. So it's a nice number. Yep.
Yep. But it's also kind of a self-inflicted problem. Like we are very good at solving a problem that
we created ourself. So what about real life file? Like real file usually that are small that have
that many revision but actually not all files. There exist a lot of files not the majority but that
are big with small update and like the same kind of pattern as the manifest.
[05:25:38] So let's have a look at the NixOS/nixpkgs. Repo. So for the manifest we are not
much better but for the files we use half of the size. We save half of a gigabyte just by storing
files in Mercurial. And so if we look a bit at the file with the most change, we see that we have
files that are not that big but have a lot of revisions where Mercurial is much better and save
hundreds of megabytes on like we are up to 40 times smaller.
[05:26:15] But like it really depends of the file. So how do we do it? We don't really have time
to go in the details. We have a talk at FOSDEM that talk about some of that if you want to but the
short thing is we use delta encoding which is kind of the same thing as Git packfiles is doing but
we do it differently and the two short version is you don't need to run any maintenance like it's
done online when you do a commit it's computed once and then that's it. It's the way it's built mean
that you can reuse the data for exchange. You don't need to have an overhead when you do exchange
and we didn't have that compression by compromising on the read performance like the read
performance are still good and we can actually reuse the structure we do for the compaction to do
very fast diff.
[05:27:02] So for 100-megabyte manifest we have computing in the range of 1 millisecond to 1
microsecond for the for real life use case which is quite good to diff that many things. And all the
number here are not like the final number because in Mercurial we keep improving things by
iteration. Everywhere we do something we learn about it we do again we change our format all the
time which our protocol which are algorithm or implementation thing like that and we know we have a
we know place where like we know we have a lot of headroom to improve a lot of things for example we
saw that for our Nixpkgs the size was about the same for tree and manifest and we started a small
experiment of our new approach to storing the manifest and it's already improving compared to what
we had already it's not super nice because I would like that to be at the last part but still like
there is room to improve and I'm going to give
[05:28:04] Back the stage two.
Scaling beyond a central server
[05:28:05] Okay, thanks. We could have talked about so many different things we glossed over
many many details and there's a whole bunch of things that we can discuss if you're interested. We
felt that focusing on content storage was a good example of how you know content storage is kind of
a fundamental part of version control like there's no way around it. So it would be good good
analogy and also because nowadays it's not really about how fast your horse can go compared to the
other horse. It's about does your horse tip over.
[05:28:33] It's can your devbox handle 30 plus million commits. Do your projects need to
rearchitect around the limitations of your VCS. For example, the Nixpkgs is things they have started
splitting up their their files because their forge couldn't handle it. Speaking of can can a forge
survive the agent agentic stampede? Like everybody's been talking about how like everything is
crumbling down and how like maybe it doesn't even handle distributed work correctly. It's more
centralized. And so we think that Mercurial storage works quite well. We have ideas of how to make
it much better still.
[05:29:14] And maybe some of those ideas can be discussed together. We can even maybe think of
creating a good Git server based on Mercurial storage for not that much work. Couple thoughts for
after larger scale stuff. Most large companies have some something similar to this. Everything goes
through a central server. It is quite centralized. However central centralized systems are going to
be a bottleneck at some point. I know that you know some large companies are already quite
struggling with their centralized systems because it's carrying everything and so between Atlas's
hands you have about 200 millisecond roundtrip in fiber and that's purely with no network overhead
if you are truly distributed.
[05:30:01] So there's no multi-master international servers. If you want to talk to the CI
directly you have to go through the main server. All of that creates a lot of over overhead and is
always going to be more costly than having a direct call. And so you can imagine as as complicated
as you want a graph, I just just for laughs I just put a bunch of arrows everywhere. It's it this is
this seems complicated but in the sense if every single place here uses the same model is just all a
CRDT and it's all the one implementation. And it's the same on the client, same on the server, same
on the small repo and a large repo, then it's not actually a problem.
[05:30:44] It's just you can just do it and then you figure out your network topology and etc.
And it's all thanks to it being a CRDT. Like commits that never reach your main server are easy to
scale. So that's kind of the crux of it. That's basically all all the time we have. Thanks. I hope
you learned something. Maybe some of that is interesting to you. In general, it's good to see that
the VCS interest is growing in the community and see so many people, care about this space because
it didn't used to be the case a few years ago.
[05:31:18] There is still room for fully free and open source actually scalable version control.
So, let's keep that going. Everybody seems to be using Rust as a common language. So, maybe we can,
exchange ideas and implementation for some stuff. Let's talk about it. Thank you so much.
[05:31:38][Applause.]
Q&A
[05:31:53] Speechless.
[05:31:56] Wow.
[05:31:58] Thank you very much, guys. Oh, wait. We do have one question. Oh, wait. Do we have
three? Okay, guys. You made me stand up.
[05:32:04] Sorry, Patrick. [Laughter.] People being shy. This is a simple question. When will
you come back and educate us more?
[05:32:12] Sure. [Laughter.]
[05:32:13] Well, there's a there's a tomorrow like we are all doing put stuff on the board.
Wait, we didn't really talk about this, but tomorrow we're doing unconference style. So if obviously
you can you know talk to everybody that you see on stage all the time but if there's like think
about a really cool topic around Mercurial that you want to talk about I'm sure they would love to
talk about it in detail. Actually this is partially how this happened is that you guys did a
breakout two years ago in Berlin and it was one of my favorite breakouts.
[05:32:42] So so yeah let's put stuff on the board tomorrow as well.
[05:32:47] Thanks.
[05:32:48] Yeah. You mentioned that you solved this self-inflicted problem of the manifest and
you get like 60,000 times compression, but could you say just a few words about how that happens?
[05:33:02] Yes. So we do delta encoding, the same as packfiles. So sometime we do a full we
store the full version of the text and most of the time we store deltas again a full version or
another version encoded as delta and then we have a bunch of heuristics to build good trees that are
not too long but also doesn't take too much space and I think the crux of it is that the manifest is
highly highly repetitive and has most generally a couple of changes or even if they're large
changes, they're not large compared to the entire manifest.
[05:33:43] That's one of the reasons why it compresses so well. Good one. Yeah, one last thing
on that topic is I mentioned that Mercurial is tracking file history. We also track the manifest
history. So, we can leverage the manifest history to know big are going to be good candidate to
start searching for a good delta base. Yeah. So, I was just curious about the obsolescence markers
like what does it mean semantically if there ends up being multiple obsolescence markers of the same
commit. What does that mean?
[05:34:20] So if if you have two if you have multiple changesets that claim to be the successor
of something you have kind of a succession crisis. But so you have I do explain that quickly like
you have a divergence in addition. So the it's the same as if you had two head on your branch like
you have you need to do something like you could merge them only merging the edit or you could
choose one that replace the other or like it just it's encodes a problem that happens because of
workflow and at some point you need to solve it as workflow.
[05:34:51] What you said earlier it's a you problem not a Mercurial problem. It's like you've
done multiple things at once. We give you tools to figure out what you want to do but it's something
that just happens. But it's roughly analogous then to having multiple heads.
[05:35:03] Yeah. Yes, it's very analogous to that with a bit more subtlety and stuff. But and
there we distinguish between having two successor that that compete and having a split where someone
is explicitly cut into multiple ones.
[05:35:16] Okay. Thank you.
[05:35:18] Thanks.
[05:35:22] I'm going to stand up so that other people can be like, "No, I actually have a
question." Okay. Thank you very much.
[05:35:33] All right, our next speaker: you know him, you love him. Steiny. For those of you who
don't know, Patrick Steinhardt is our next speaker. He's been working on core Git via GitLab for a
long time now, and before that, libgit2. He's going to be talking about, I believe, pluggable
object backends. But the most important thing for you to understand is that he's to be called
Steiny. He prefers that as a nickname. So if you see him in the hallway, if you have a beer with him
later or whatever, just be sure to say, "Hey, Steiny."
[05:36:10] Okay. Thank you very much. And a big round of applause for Steiny.
A load-bearing question mark
[05:36:14] Thank you. Thank you. [Applause.] I mean, at least it's not Steiny the reftable guy
today, so I'm already appreciative of that. So thanks. So what I want to talk about today is how we
at GitLab are rebuilding Git from the ground up and that is like somewhat of a boring talk because
everyone is currently basically doing the exact same thing because of agents. So I want to add one
more character to that slide which is this question mark and I mean to say it in Claude terms this
question mark is load-bearing for this talk.
[05:36:47] So I mean you already introduced myself. I don't really have to reintroduce I guess.
I started diving into Linux in 2002 when I was 11 years old. I then became an open source
contributor in 2011. Am a Git and libgit2 contributor since 2015 and nowadays I'm an engineer at
GitLab which I joined in 2020 where I'm the tech lead of the Git team. So a couple months ago,
GitLab announced that we are rebuilding Git as part of our engineering efforts to address the
current scalability limits and challenges that we're facing. Of course, that announcement caused a
couple of questions out there and like many people were skeptical.
[05:37:27] They were asking like, are you really rebuilding Git? How much of it? Is it a fork?
Are you rewriting it in Rust? And are really the next company that is basically betting their
infrastructure on a vibe-coded Git back end. But to answer the questions, it's no, not a lot. No,
maybe some of it. And no. So let me expand on that a little bit. No, we are not rebuilding Git. In
fact, that would not really made a lot of sense for us. For the last couple years, we always had an
upstream first approach. We have a full team dedicated to upstream Git development and have a very
community-centric approach.
[05:38:07] So we rely on Git being a healthy project and we really benefit from his community.
So we will keep on investing into Git itself. So what we're actually doing is that we're rebuilding
Git storage foundations. So the layer that stores your references and your objects. And even more
importantly, we're doing this inside Git itself and together with the Git community. But first, why
would anyone want to mess with a storage layer in the first place? There's no doubt that Git is
extremely successful out there. 94% of all the developers use it. There is hundreds of millions of
repositories out there and an uncountable number of scripts and tools that depend on it.
[05:38:48] But that success is also exactly why the storage layer is under pressure nowadays. It
was designed in 2005 for the world of 2005. It was envisioned as a distributed version control
system where users synchronize their changes with one another directly. But today most of the users
only synchronized their changes with a central server and thousands of concurrent users are hitting
those servers at the same point in time. The Linux kernel was also considered to be a rather big
repository back then. But I mean nowadays when you have the Chromium repository that is almost 100
GB in size, well it's kind of tiny in comparison.
[05:39:29] CI systems were kind of the exception back then and nowadays I have customers where a
single commit will kick off thousands of jobs that all clone the repository. Also monorepository is
a term that nobody has really heard of back then and nowadays everyone is talking about them even
though you also have to admit that nobody really agrees what a monorepo even is. But there's another
elephant in the room here and I mean we've been talking about that multiple times already in today
and yesterday and that is really the most important shift that we have seen in our industry in a
very long time.
[05:40:07] I'm not really here to preach artificial intelligence to you but there's definitely
trends that we cannot really deny there as an example like these are 100 different developers that
you have from the Stack Overflow developer survey 2025. Half of them use AI tools every single day.
17 of them use it weekly. 13 use it monthly or less. Four plan to start using it. And these 15 here,
they don't use it at all. And they also don't plan to. And that is fine. Like I mean, there's good
reasons why you wouldn't want to use AI.
[05:40:41] But regardless of where you stand, this has an impact on us. And remember, this
survey is for 2025. In 2026, this has changed significantly. Again, a third of all the code in
organizations is already AI generated nowadays and we see customers whose codebase grows up to five
times in a single year. The point is simply this repositories are growing at an unprecedented scale
and are also growing bigger than ever before because of it. And this of course has an impact on our
hosting infrastructure which is under lots of pressure this year. Which brings us to the core
observation.
[05:41:21] Git storage layer is becoming the bottleneck. The data model, the protocol, and even
the user interface is mostly fine. Or at least Claude doesn't care about the shitty user experience
of Git. But storing data in a local file system alone is simply not working anymore. And we're not
really alone with this observation. This is something that I would claim all the hosters are
struggling with right now. And this is why we are rebuilding Git storage layers. There is two
essential parts to the storage layers. References and objects. I mean this is simplifying things a
little bit.
References and reftable
[05:41:57] There's of course also other stuff but at least for the sake of this presentation we
can pretend that there is nothing else. The first part that I want to get into is references. Let's
first do a quick recap of the status quo. For 20 years, Git has stored references with a files back
end. So every branch or tag is a small file in the .git/refs hierarchy where the name of the
reference is literally the path of the file itself. And the content of that file is the object ID
that the reference points to.
[05:42:28] And because thousands of small text files are inefficient, Git periodically packs
them into a single sorted text file called packed-refs. It's basically very trivial and also very
easy to debug and it has served us quite well for the last two decades. But it has three fundamental
problems. Problem number one is that file systems are simply weird. Refs are encoded as file paths.
So they inherit all the file system quirks out there. On Windows, you cannot have a branch that is
called NUL or CON. On a case insensitive file system, you cannot have two branches that differ
only in casing.
[05:43:04] And on macOS Unicode ref names may silently change their encoding and all of these
quirks are ex or directly exposed to the user. The second problem is that the design is very
inefficient at scale. Deleting a single packed ref file means that you have to rewrite the entire
file which works well enough if you have a small repository but once you have or once you're talking
about scale it becomes stupidly inefficient. In our one of our worst repositories, we have many
millions of references and deleting a single reference in there requires us to rewrite more than two
gigabytes of data.
[05:43:47] And problem number three is that concurrency is basically an afterthought. There is
no way for you to get a consistent snapshot of all the references as a concurrent writer could
change the state at any point in time. And there's also no way to atomically update your references
if you want to write multiple references at the same time. So Shawn Pearce has been designing an
alternative format for storing references called reftables. Reftables have been my own personal
obsession for a couple years back then when I have been implementing support for them in Git and
they're a new way to handle that load.
[05:44:20] With the reftable back end references live in a set of binary tables in the
.git/reftable directory. The tables.list file essentially tracks the currently set of active
tables. So every time you write new references, a new table gets created and the tables.list file
gets atomically swapped into place by using a rename. The tables themselves are sorted block-based
binary files with prefix compression for the ref names and once written a table is immutable. So you
really only create new references by updating the stack with a new table. Also reftables can have a
tombstone so that you can basically mark the reference as deleted.
[05:44:59] You don't have to rewrite 2 GB of data only to delete a single reference. The result
is that readers always get a consistent snapshot. Writers can update any number of references
atomically and deleting references now constant time operation. I could talk about that format for
quite a while if I wanted to, but I mean today today I really don't want to because here's the
thing, the format is not the point to land reftable. Git had to do something much more fundamental.
We had to force a real reference backend abstraction into the Git codebase. So lots of code paths
that previously accessed references directly for example by just writing the file in .git/refs had
to be routed through a single well-defined interface instead.
[05:45:48] The reftable backend is simply the first alternative implementation of that
interface. It proved that the whole storage layer could be swapped out without breaking everything
built on top of it. But references are only the smaller half of the storage layer. The much bigger
half is objects. The object database is where Git stores the actual contents of your commit graph.
So your commits, trees, blobs, and annotated tags. The newly written objects typically start out as
a loose object which is simply a compressed file per object addressed by its hash in the local file
system. And for efficiency, Git regularly packs them into packfiles.
Making object storage pluggable
[05:46:27] So these contain many objects at once. Use delta compression for efficient storage
and have indices so that you can fast or do fast lookups of your objects. And again, the pro problem
here is not really the formats themselves. They're excellent for what they were designed for
originally, but the use cases around them are evolving. Same as with references originally, the
problem here is that the formats were again hardcoded across the entire codebase. So, hundreds of
call sites simply assumed an object is a file that either lives as a loose object or as an entry in
a packed file.
[05:47:01] And you can't really change the implementation and plug in an alternative backend
without having a proper abstraction layer. So this here is a simplified view of Git's object
database design. At the top you can basically see the consumers that want to access your objects.
And at the bottom you can see the hard two hardcoded formats that we have with loose objects and
packfiles as well as a couple of auxiliary data structures that we use to speed up object accesses.
And running alongside all of that on the right side is the repository which is basically a global
variable that exists all across the Git codebase.
[05:47:40] And you cannot really put an abstraction layer in between those two different layers
when everything can basically reach around you by just going via the repository. So the first step
for us to introduce pluggable object databases was making the object database subsystems
self-contained and not use the repository anymore one by one. So the multi-pack index basically shed
the repository in Git version 2.48 packfiles followed in 2.49 loose objects in 2.51 and commit
graphs in 2.52. It's been rather unglamorous work to remove that global state from all those
subsystems. But it also allowed us to get a little bit more familiar with how they actually look
like and what we need to change.
[05:48:24] And with that the implicit global state in our object database is gone now and we can
thus start to draw boundaries. So the most important step that we then did is in 2.51 we started to
introduce a proper object database abstraction as a starting point for our efforts. From here on
this is basically the single entry point for you to reach out into the object database to access the
objects and to rewrite the objects. We then started to extend that abstraction layer piece by piece.
For example, we introduced proper interfaces to stream objects into your object database or out of
your object database in Git 2.52.
[05:49:06] In 2.53, we then introduced the packed object database source which basically started
to properly encapsulate all the accesses to your packfiles so that you can easily read or write
packed objects. Previously this state was also like managed rather ad hoc by just a list of
packfiles that was owned by the object database itself. And in 2.54 we then finally introduced the
biggest chunk of this whole infrastructure by introducing something called the odb_source. This
data structure is essentially a virtual function pointer table that gives you function pointers to
read objects, write objects, enumerate objects and all the other operations you have.
[05:49:49] At the same time, we also converted the existing sources that we already had and made
them proper odb_source implementations themselves. So the file source is now the entry point that
manages both your loose objects and the packed objects and is properly encapsulated as a virtual
function pointer table. In Git 2.55 we extended this infrastructure with a fourth source that stores
your in-memory objects that you can read but that you can that are but that are never written to
disk itself. That release also brought the foundations for pluggable transactions which allows you
to write many objects at once into a repository.
[05:50:28] This mechanism is for example used when you fetch or when you receive a push in Git
2.56 which will be released at the end of this month. We also made repository maintenance pluggable
as well as consistency checks for git fsck. But there is still one big shift outstanding that we
haven't done yet because your object database can in fact not only exist of a single source but it
can exist of multiple or consist of multiple different sources by using alternates. These are
currently handled on the object database level. So that basically means that the object database
does not manage one source but it manages multiple different sources.
[05:51:09] But that really created a bunch of issues for us. So in Git 2.57 so the next release
we'll also make alternates and implementation detail of the files back end itself and once that is
done we can start to introduce a new repository extension and allow you to implement your own
backends. What's remarkable is that this work has been ongoing for the last two years already and we
have changed lots of the foundational stuff in Git itself but in the best case no user even noticed
that anything has changed underneath them. So again starting with 2.57 the entire storage layer of
Git will be pluggable.
SQLite and S3 demonstrations
[05:51:49] Now the question is what will this unlock once storage is an interface backends
become essentially an implementation detail and that opens the door to a lot of different new ideas.
Storage essentially becomes a design space that you now can start to innovate in. You could, for
example, put your references or objects into a database like SQLite or Postgres if you wanted to.
You could put repositories into object storage and inherit durability, replication, and elasticity
from your storage layer. You could introduce entirely new on-disk formats, for example, content
defined chunking to make storing large binary files way more efficient than it currently is.
[05:52:30] You could even build entirely new semantics if you wanted to, like consistent
snapshots across references and objects, fully transactional writes, back ends that are aware of
replication. I'm not really saying that these ideas are necessarily good ideas. But by making the
storage layer pluggable in the first place, we enable those new ideas. So let's make it a little bit
more concrete like how does such a new back end look like? A back end is essentially just a virtual
function pointer table that you have to implement. So you design the concept, you implement the
individual function pointers and from there on Git knows to speak your back end.
[05:53:08] So implement these two different table like these tables ones for storage for objects
and ones for references and then Git knows to run your storage without changes to any of the Git
commands. I want to give you two little demos. The first one is going to be a SQLite back end where
both references and objects are stored in a single SQLite database. And again, this idea is kind of
dumb in the first place. Like it doesn't scale. But it's something that I would definitely recommend
not to ever use in production systems, but it's an easy example that everyone can easily understand
where the design is really trivial.
[05:53:47] You have a single file that contains your object database and your reference
database, using two different tables, and that's it. Let's see this in action. You can create a new
repository by passing two different flags. In this case, --ref-storage-format=sqlite and
--object-storage-format=sqlite. Create the new repository. You go into it and create your commits
just like you wanted to. It exists. Boom. But if you look at the local disk now, you will not see
any loose objects. You will not see any references, only stubs. But what you can see instead is the
.git/git.db file, which is your single SQLite database.
[05:54:29] And if you can even query that if you wanted to by just selecting a references from
it or your objects. As I said, a single SQLite file on your local disc does not really solve your
problem though. What we want to have to handle the loads from AI agents and the agentic workloads
nowadays is to be able to scale horizontally. And that typically means that your data has to live
somewhere distributed. The obvious answer to that is object storage where references and objects are
stored directly in an S3 compatible object store. On paper, this sounds like kind of perfect.
[05:55:11] Object storage is durable. It's replicated. It's basically infinitely scalable and
somebody else operates it for you. So Git stores objects, object storage stores objects. So it's
like a match made in heaven. The naive design maps Git's model one-to-one onto the bucket. So every
Git object becomes one object storage entity keyed by the hash and every reference is one object
keyed by the ref name. It's again very simple, and it works, which I can show you with another demo.
This time I will first go outside the other repository and spin up a SeaweedFS server to have
S3-compatible storage backing it. Now I can again say git init, this time with S3 as a backend.
You can see that I'm also passing the S3_ENDPOINT variable, which is basically just the URL.
[05:56:07] So we create it go into the repo again commit and have just normal thing. If you now
have a look though you see that there is actually no on disk state at all not even a database file.
So the nice thing about that is that in theory you could just delete that repository, reinitialize
it, go into it again and the data still exists because now your state is fully distributed. It does
not exist on the local disk anymore. And that's exactly what we want. So the design works, but it
does not scale. Object storage gives you milliseconds of latency per request while Git's access
patterns typically assume microseconds.
[05:56:54] A repository has millions of references. So a clone now becomes a million of
sequential requests which is going to be completely unusable. We also lose everything the packfile
format gives us. There is no delta compression. So every object is stored in full. References are
causing issues too for us. Listing refs means listing lots of different keys and atomic
multi-reference transactions are basically impossible because object storage lacks the primitives
for us to implement those atomically. And finally, cost. Object storage typically bills per request.
And a system built around many small reads is basically the most inefficient way that you could use
object storage in the first place.
[05:57:37] So object storage is the right place for Git data. It's durable, replicated, elastic,
exactly what we need for horizontal scalability. But storing individual Git objects in there just
does not make a lot of sense for us. Which brings me to the final part, what we are actually
building on top of that at GitLab. Git basically already does a very good job of storing lots of
references and objects efficiently by using reftables and packfiles. So why don't we just store
those in object storage? And that's exactly what we do. But we add two more important pieces to it.
Manifests, generations, and local caches
[05:58:11] Manifests which describe the currently active set of packfiles and reftables and
generations which is the sequence of manifests as they are activated in that repository and they're
the only mutable part of the system. Everything else is basically immutable and content addressable.
Above that we have our Gitaly nodes which provide the APIs to access your Git repositories. Those
nodes are now essentially stateless and act only as a cache as object storage is the single source
of truth for the repositories. A Gitaly node pulls the files named by the manifest onto the local
disk, tells Git to use that manifest by using a custom backend and publishes new artifacts back to
object storage.
[05:58:51] And on top of that, we have a simple routing layer. Since no node owns a repository
anymore, any node can serve any repository. This new design has the consequence that the fleet
becomes elastic. When we receive more requests, we can easily add another Gitaly node and have it
serve repositories almost immediately. And when nodes are idle, then we can remove them again from
the fleet. So this is the idea, but it makes a big promise. Any node can serve any repository many
concurrent requests at a time in a consistent way. So let's zoom into a single Gitaly node.
[05:59:29] Many RPCs run concurrently against that same local copy of a repository. What do we
want? First, each request should see exactly the state that is currently published in object storage
and that state should stay consistent for the duration of the RPC so that we have no dirty reads and
also when a write finishes its result should be published atomically without impacting any of the
other concurrent readers or writers. With a current Git back end, we don't really get any of that
natively. So let's assume we have two concurrent writing requests. RPC1 reads the current state of
the repo and sees the pack and the reftable.
[06:00:06] RPC1 writes his repository like its objects and those objects directly land in the
shared object database. There are those visible to everybody else immediately even though the RPC
has not concluded yet. Quarantine directories solve one part of that problem but not really. RPC2
now reads the current state and already sees the uncommitted objects from RPC1. So we're getting a
dirty read. RPC2 writes its objects as well and we now have basically an intermix of both different
states. The two writers are not isolated from one another. We update the references but the
repository is already changed underneath RPC1.
[06:00:48] So again the write is not atomic and RPC2 writes as references too. So we have dirty
reads, interleaved writes, and on top of that, it's hard for us to figure out which RPC wrote
exactly what data. You can work around these limitations. But this is getting complex real fast and
doesn't really result in a well-working system. And we know because we tried. Gitaly tried to solve
this by using snapshots. Every Git repository or every Git command basically got its own repository
by hard-linking everything into place. That gives you full isolation. But even with hard links, it
was prohibitively expensive to create those snapshots and it simply didn't scale.
Multiversion concurrency control
[06:01:30] The root of the problem was basically that we were working around Git and not with
Git. So again, what we want is snapshot isolation, atomic publishing, and well-defined change sets.
If that sounds familiar, it's because databases already solve this exact problem decades ago. And
they call this multiversion concurrency control. We basically had the same note as on the previous
slide here, but Git now runs with the custom MVCC back end built on top of the pluggable object
databases that we have been talking about earlier. Git itself still works exclusively against the
local disk. It never talks to object storage at all.
[06:02:09] The trick here is the manifest. It's basically just a small document listing which
refs and packed files make up the repo right now. And since all the artifacts are immutable, that
one document gives us multiversion concurrency control essentially for free. So readers can be
pinned to a certain manifest and they will never see changes performed by any concurrent writer and
writers can update the state atomically by writing a new manifest and activating it. So let's again
replay the same sequence. RPC1 resolves the manifest exactly once and gets a fully consistent
snapshot which is the currently published state in object storage.
[06:02:46] It writes it objects. RPC2 resolves the manifest 2, but it does not see the objects
from the other RPC yet because it's still operating on the manifest version one. RPC2 writes its
objects as well. And RPC1 now writes its references without publishing any state yet. RPC2 does the
same. So we've written all the new artifacts, but as nothing is committed, it's just not visible to
anybody else. RPC1 now writes the new manifest that makes its new data reachable. Gitaly uploads the
diff between the old manifest and the new manifest to object storage and then activates it.
[06:03:25] And once the new RPC starts, it will now see version two. RPC2 is still operating on
the original manifest. So publishing its state now will fail because of the concurrent update. To
fix that, it performs a merge of its changes and then tries to publish again. So the reads are fully
consistent, writes are atomic. So with this back end, we're now able to realize a new architecture
that is horizontally scalable and where object storage is the single source of truth for us. But I'm
deliberately keeping this at the architecture level because the details are a talk of their own
because there's one more thing that I think that is way more important than this architecture.
One shared upstream foundation
[06:04:08] This design is really not breathtaking. A lot of other companies are doing exactly
the same thing. The difference is that we are building this together with the Git community and not
around it. Every piece of the foundations that we are building and that I showed you today is
upstream Git. And if there's appetite for it, we would even upstream the MVCC back end itself into
Git. And to me, this distinction is quite important because the interfaces are upstream. Anyone can
now build backends. The SQLite demo from earlier is basically not privileged code or anything like
that, but is very easy to build for everyone out there.
[06:04:42] There you can also implement your own idea. Also, the ecosystem stays whole. There is
one Git with potentially many different backends instead of five different companies all forking Git
and doing their own stuff. Innovation becomes possible. Whether you want to introduce better storage
for large binary files, a database backend or a back end we haven't thought of. Everyone benefits
from the same shared foundation and improvements to that foundation benefit everyone. But most
importantly, by innovating in the Git project and evolving its foundations, we also ensure that the
project remains healthy and can rise to new demands.
[06:05:23] Honestly, scratching our own itch would have been way faster compared to building all
of that. But moving the whole ecosystem as a whole is worth the extra investment in my opinion. So,
are we rebuilding Git? No, we're not. We just made Git storage layer rebuildable and we did that for
everyone. So if you have an itch on your own, the interfaces are upstream. The mailing list is open
and I'd love for you to bring your own use cases to that mailing list as well. And that's it
basically. So if you want to chat about pluggable backends, reftables, object storage or whatnot,
please come just like hit me on the hallway track and I'm very happy to talk about.
[06:06:02] Thank you. [Applause.] questions. Hey, thank you. Great talk and thank you for the
leadership in the Git project on this as well and open sourcing everything like great job. Bit of an
obvious question. I'm not going to lie, but you probably skimmed it for a reason. But you were
talking about the RPC1 and RPC2 and writing yeah here here good when they let's say lock the
manifest but what happens if there's conflicts and they're writing to the same packs or writing to
the same refs and stuff. So I mean for objects as per has already been saying earlier objects are
basically CRDTs so you can just merge them there is no real conflict there but for references what
we're basically doing is that you perform a three-way merge against the old manifest the new
manifest and the base manifest that is currently published upstream and by that you're able to like
resolve the conflicts and if there of course is an actual conflict in your references then you will
basically have to fail the RPC because there were two concurrent writers trying to update the same
reference But if you're updating different sets of references, then you're good.
Q&A
[06:07:20] So worst worst case scenario, one will fail. The receiver fail on the push.
[06:07:26] Like if two if two users are pu pushing to the same branch at the same point in time,
there's going to be a conflict.
[06:07:36] Hi. So this is really interesting and the you know pluggable backends is really neat
but you mentioned like one Git but practically it seems like we have multiple Git implementations
going on in the background like there's Git proper and there's also libgit and rust there's like
gix and all sorts of other stuff right so I you know this clearly this works well on a server
because you're you know you're just receiving objects from the Git protocol. But like if this starts
to become a big thing on the client, right, and you introduce maybe a new storage backends on the
client side, how do you coordinate across the practically many implementations of Git that are
expecting to read you know the traditional format?
[06:08:22] It's very hard and I mean historically Git has also been very conservative about
introducing breaking changes. So we already are basically doing that exercise right now with the
pluggable reference databases where we now have an second implementation of how to store references
in your repository already by using reftables and some of the things that we did at GitLab for
example is that we also implement reftable support in libgit2. I was also trying to get that into
gitoxide, but my Rust Fu is not really that good. I have to admit I'm more of a C person.
[06:08:57] So, that is unfortunately something that we haven't done yet. But I'm also talking
with Byron about that problem like at least at least we're talking, let's put it that way. But we're
also just trying to not like break everyone by giving like heads up notices so that everyone is
aware. But I mean I also get your point that in the best case of course it wouldn't even be
necessary to have like many different implementations of Git but I mean that is to blame on the Git
pro like the blame there is on the Git project itself because we never really published a proper
library.
[06:09:29]libgit2 was originally designed to become that library in Git but we never pulled
it back and that is a little bit of a shame. So that makes it way harder for us nowadays to actually
introduce change. But the way that we do this is also by having at least repository extensions. So
that you can still choose to pick the old back end if you wanted to. So that is still an option
going forward if your specific implementation of Git does not support the new backends.
[06:09:58] With some of this work, is there any consideration towards licensing? Because I think
one of the major blockers for relying on Git code for a lot of companies is the fact that it's GPL
[06:10:11] Rather than something more permissive like MIT. The reftable back end or the reftable
library for example that exists in the Git codebase itself is MIT licensed so that is something that
you can just like use for your own project as well I mean for the object backends going forward it
will of course like need to be like determined we also do have the idea that eventually we want to
have like a well-defined interface so that you can just load a shared object for example into your
Git process and in the best case like that would then mean that you can like have your own license
for that shared object But like even better would be if that shared object could also be shared
across Git libgit2 maybe even gitoxide.
[06:10:51] That would be awesome if it would work be working but that is still future music.
[06:10:56] Thank you. Yeah, I think mine mine is a little related to that which is you have to I
mean for all of these other backends that you're doing you're recompiling Git with those backends,
right? Because it has to be loaded. So is there a plan of being able to I mean I don't even know how
you do this in a way that is performant but like to have a second binary or whatever that you own. I
like I just it does kind of move towards forking Git to have like unless you get every single sort
of pluggable backend upstream, right?
[06:11:27] Yeah. Like I mean that is exactly what I just mentioned that my plan around that is
to have a back end that is simply just like giving you the flexibility to load whatever logic you
want into Git itself without having to fork it because I mean that would also be painful for
ourselves if we actually had to fork just to have our own back end and Git itself if we were not
able to upstream it. Whether that is going to be a shared object, whether it's going to be just like
a server interface or something like that with a well-defined protocol, I mean that's to be
determined.
[06:11:58] But we definitely do want to introduce the ability eventually that to just run your
own external logic without having to recompile Git.
[06:12:16] Okay,
[06:12:16] One more question over there, I think. Sorry, [Laughter.]
[06:12:22] It's not a question. It's not a question. It's more like an observation like JJ
inspired Git to be more pluggable.
[06:12:29] Yeah.
[06:12:29] And now we're hitting the sameish problems at at around the same time. Yeah.
[06:12:34] Because this discussion is very active right now.
[06:12:37] So yeah, we'll see how both projects will move forward on this.
[06:12:43] See which one will be better off.
[06:12:45] I agree. And I mean in the best case I would think that just both can exist next to
one another. But I also have to admit that JJ itself really finally moved the Git project into the
direction of modernizing a little bit because like the Git project has been stale for way too long
and that is something that I'm really happy about that by JJ becoming more popular we start to
innovate again in the Git project. Not only with like this kind of stuff here but also for example
like the Git history command or other stuff that we have been doing recently and you've been quite
inspiring in that case.
[06:13:23] Okay, big round of applause. Thank you Steiny. [Applause.]
[06:56:38] We got some people trickling in. So, I'm going to make good use of the dad jokes
channel of our Discord. So, what happens when you cross a rabbit with a water hose? You get hare
spray. So, I was rearranging my spice drawer only to have one of them sort of explode all over me.
I've got thyme on my hands. Oh boy, that one's not good.
[06:57:09] Sorry. How do you measure the speed of a swimmer? With a Speedo-meter. Okay, that one
doesn't work. Because they're wearing Speedos.
[06:57:24] Yeah, it's better when you explain it. Okay. Why did the tomato blush? Because it saw
a salad dressing. Why can't you work twelve jobs? Dozen't work. All righty, enough of that. Let's
introduce the co-founder of Tangled who's going to be telling us about why the sort of future of
forges needs to be hackable. Please welcome to the stage Anirudh. Thank you for being here.
[06:58:00] Yeah, thanks for having me.
Hackable, malleable software
[06:58:07] All right, awesome. Hey everyone. I am Anirudh and I'm here to talk about why the Git
Forge must be hackable. But before we get into it, obligatory intro. Here's a picture of me. The
distinct difference being I now have a beard. So I am the co-founder and CEO of Tangled. I
previously worked at, various dev tooling companies doing infra and distributed systems work. And a
bit about Tangled. It's a pretty new company. We founded it, co-founder is my brother Ash. We
co-founded it together in well, I guess obviously in 2025.
[06:58:45] We're now a globally remote team of seven. So, what's the whole deal with
hackability, right? So, what's our bet on hackability? So, oh, actually, give me a second here.
Well, looks like I cannot pull my notes up for whatever reason. So I'm just going to have to wing
it. Anyway, so the era of personal software is here, right? So like agents have essentially made it
so that building custom tools is now super easy and super cheap and you can sort of as long as you
have open interfaces and formats, you can hack anything into into existence. You can just will it
into existence really.
[06:59:34] And also self-hosting and I guess owning your infrastructure is is really as easy
it's ever as it's ever been. Again thanks to agents because you can quite literally just have Claude
take care of your entire VPS or you know deal with your Terraform or NixOS configs or what have you
and it you kind of don't have to think about it. Or if you don't want to deal with the VPS, you
could potentially just use something like Cloudflare Workers or Vercel and still have some semblance
of ownership in the sense that you can still control this the service to a degree, right?
[07:00:08] And lastly, we believe that you know our software must outlive us. So anything that
we build at Tangled, we want to ensure that it exists beyond the company. And this is a core premise
or tenet or ethos of how we build software and malleable software and open formats are the way to
enable this. But before we get into the I guess the nitty-gritty of the Tangled architecture itself
and how it's hackable, let's talk a bit about the protocol that I guess builds the whole it I guess
forms the whole foundation for what we've built.
Identity and the AT Protocol
[07:00:49] So we use something called AT Protocol or the Authenticated Transfer Protocol. And in
AT Protocol you have the notion of PDSs: personal data servers where
all your signed data and signed structured records live. These are then collected. So
PDSs are super lightweight. You can run them on anything you can think of. And they're super
self-hostable, but most cases you can probably just sign up with a service that's offering accounts
and you will get an account on a PDS.
[07:01:25] And you can also migrate between PDSs. So you don't you're not really, stuck to a PDS
unlike in other forms of federation like in Mastodon for example where your identity and account is
is sort of tied to the server that you choose and you're you kind of have to there's more weight to
that decision. In AT Protocol land it's not quite so so you can hop around freely between PDSs you
can also adversarially migrate between PDSs should the current host I don't know go go go rogue or
down even and then you have the concept of relays right so relays are these horizontally scalable
services that sort of hoover in the data that lives in PDSs and create a firehose so you don't
really have to trust the relay because all data in AT Protocol is self-verifying and hence
authenticated transfer.
[07:02:22] So thanks to that property of the protocol you can sort of just consume this firehose
and build any kind of index on top of that. So you can sort of choose what you can sort of pick and
choose right so in Tangled's case we have org.tangled.feed.starorg.tangled.actor.profile a
whole bunch of different org.tangled records in your pds and in for example Bluesky which is
another AT Protocol app would have app.bsky or whatever. And you as a consumer of the firehose can
pick and choose what you want to build your index off essentially.
[07:03:00] And in AT Protocol terms, this is an app view. It originates from the fact that this
is a materialized view over all of this data. Indexers are are are sort of free game. You can sort
of optimize and build custom services alongside this to support your user experience as much as you
wish. And this is something that we do. And on top the indexer lives the actual client. So typically
the indexer presents itself as an API service of some kind and the end user client is perhaps like a
like a web app potentially or an Android app or an iOS app that will you know just directly query
your API and present a nice interface for the end user.
[07:03:50] And importantly the client performs writes directly to the PDS. So this is what
closes the whole loop. So the client is authenticated via the PDS via OAuth and is able to write
relevant records back to the PDS. And then the whole thing kicks off again. So that's sort of a
somewhat high level overview of the AT Protocol. And then comes the Tangled bits. And I guess this
is where it gets very interesting. Because we do a lot of AT Protocol stuff, but we've also done a
lot of custom stuff to and then hacked it into not hacked it, but like worked it into AT Protocol.
Knots, spindles, and Bobbin
[07:04:31] So at the core of Tangled is what we call knots. So knots are your Git remote. So
knots are these super lightweight headless servers that you can run again on any infrastructure that
you can that you control. This could be a Raspberry Pi, a VPS whatever. And these store your
importantly your Git repositories but they also store issues pull requests collaborators ACLs and
also resolve RBAC and they store them as COBs or collaborative objects so there's a there's a lot of
prior art here so the Radicle folks are here and they were the people who sort of created this
notion of COBs and we have borrowed that and well rebuilt it for our use case the so prior to this
we were actually storing issues and pull requests and collaborator info in your PDS as if they were
AT Protocol records.
[07:05:32] The reason we decided to or opted to move away from this model is largely due to the
fact that AT Protocol is at least in its current shape is pretty single user in that if you create
an issue you are now the owner of that issue and you can choose to delete that and it's completely
gone from the repository which sounds fine in theory but in practice you might actually want the
maintainer might actually want to maybe preserve that data because say the issue is actually
something worth preserving or the pull request is something worth working on or keeping keeping
around.
[07:06:11] And in effect these start sounding a lot like contributions in fact to your work. You
can't just you know will you can't you can't just like revoke a commit that you've made to an open
source project right because it's a contribution. It exists in the commit graph and you can't just
change your mind and be like, "Oh, I want it now. I want it back." So, we figured that issues and
pull requests are very similar in that sense and ought to be preserved. Of course, you can
potentially disown them but they still ought to remain in control of I guess the repository which is
the owner of it.
[07:06:49] And therefore they're now stored as COBs and stored as Git data in the repository.
And then we do some very fun stuff at the knot level itself where we produce or I guess like project
all of this stuff into an AT Protocol repository which is another term that AT Protocol uses
confusingly but into a into a Merkle search tree or an MST. So all of this stuff is then like you
know pushed into AT Protocol land and then the knot starts behaving much like a pds. Therefore,
we're able to reuse components of AT Protocol like the relay to then you know hoover in knot data as
well.
[07:07:28] So this is sort of how knots work. We've the current knot implementation is in Rust
and it is a fully custom Git server that does not touch the Git binary server side and gives us a
lot of flexibility around how we do stuff. But yeah, we can talk a bit more about that soon. Yeah,
of course, then you have the relay and then you have spindles, which is our term for the CI runner.
U we won't talk too much about spindles here because well, it's not really relevant right now.
[07:08:04] But it handles CI and handles webhooks. And then comes Bobbin, which is a super
interesting piece of kit. I highly suggest that you find Lewis here and talk to him about Bobbin
because I think he's done an incredible job here. So Bobbin is our API service and how this works is
it is entirely diskless. It stores no state on disk and it runs within 100 megs. If I remember
correctly this was again in the speaker notes which I don't have but it runs within 100 megs and
essentially stores the entire state of Tangled all of the Tangled records and Tangled data and
identity cache in memory.
[07:08:47] And it's able to fully hydrate this on a fresh start in under 30 seconds. So this is
a very unique property and a unique way to design an API service because now we can effectively spin
up as many of these for as little you know cost as as you know as little cost right because it takes
so little compute to run and we can effectively in our new architecture deploy as many Bobbin
instances around the world as you can imagine and run them as close to the user as possible because
then that gives everyone the best experience.
[07:09:22] So that's Bobbin and then we have some supporting services like our repository mirror
service code search index and then emails etc. So these and notifications etc. So that's all of this
sort of combined forms the app view and then we have a Svelte client that you know sits in sits on
top of all of this and you know writes Tangled data back to your PDS. There's a lot more here. I've
decided to simplify this for the sake of the talk but that's the high-level overview of the Tangled
architecture then so at the core of Tangled really are interfaces and this is where I guess the
hackability bits start coming in right so we have our knot distribution and for now it stores Git
repositories on disk and then it exposes something called an XRPC API which is just AT Protocol
terminology for HTTP.
Open interfaces and a Code Storage demonstration
[07:10:28] And as long as your knot satisfies the exact same API surface, you are a knot. As far
as Tangled is concerned, your storage could be anywhere, right? There are there have been many ideas
presented here already. But we'll I'll show you one in our demo next. And then of course there are
spindles which again follows the same pattern where our spindle points to a custom microVM-based
scheduler that gives you NixOS microVMs but your spindle could potentially point anywhere Buildkite
even GitHub actions if that's your thing. So having talked about that let's talk actually you know
what let me just show you what I've built right so here is my demo so this is a repository on
Tangled looks pretty benign as it were but it gets a bit interesting because remember I said you can
point your your storage to anywhere I've decided to use Pierre Computer Company's Code Storage
product which is I think a very cool piece of kit and so I built a knot that runs on Cloudflare
Workers as using their container platform and stores repositories on Code Storage.
[07:11:49] So this is the exact same repository but it's store it's stored on Code Storage and I
think this is neat because you can now essentially have a fully agentic knot as it were but still
federated completely on Tangled I can start it fork it make pull requests to it whatever right just
to sort of take it a bit further I have pre-warmed my agent with a prompt so this is supposed to be
using the what's it called? Code Storage's commit pack API and also Git push and create a whole new
repository without me doing anything.
[07:12:32] So while this is I guess doing its thing, I will take the liberty to perhaps show you
some of the new stuff we're working on. Maybe a sneak preview. So this is our new profile view which
I think is very swanky and cool. So to be to sort of explain a bit further, we're actually working
on a whole new revamp of the Tangled platform. The architectural stuff is what I showcased in the
slide, but this is also the visual stuff. So this is our whole new profile view. I think it's very
cool.
Stacked review and interchangeable CI
[07:13:04] But it gets even more fun when it comes to the pull request view. So Andre showcased
a screenshot. I figured I'd take it one step further and show you an actual demo of what a stacked
PR looks like on Tangled. So here's a pretty standard pull request. It's looks like a stack, I
guess. But it's cool because you can now drag this selector and then you get a very nice range diff
of the actual change set. You can just hit base and then you get the entire thing.
[07:13:40] Or you can for example choose a revision. So you can submit the same change set and
number of times to address reviews as you see here. So this is V1 reviews and then V2 was submitted
and then now you have V3. So if you want to interdiff between all of them or like few of them, you
can hit this and then voila, you're now in the interdiff view. And this is I think it's pretty
awesome. So then you can like pick individual changes to see what's actually changed. Well, not
nothing changed here.
[07:14:14] Some stuff changed here. So on and so forth. You can hit this and actually look at
the JJ change ids. Because this is all built on top of Jujutsu. But you can also do commit based
review if you don't do use Jujutsu because you can just hit a particular commit and then you know
comment on that particular change. But anyway, that aside, let's maybe check back on our agent and
see if it's done something. It's doing something. Well, [Laughter.] let me head back to my
repositories. Oh, well, we have a repository.
[07:14:48] And we have data. So, I guess what it's way it's just waiting on CI to run. But the
neat thing about this now is all of this is actually hosted on Code Storage. So if we go back here
and then go to repositories, we should see hello Git merge. Yes. And none of this was done using
Tangled in the sense that we have implemented no changes to our code to allow this. This is all done
via Code Storage and I guess just the open interfaces. And well while CI is running I can also well
CI actually just completed.
[07:15:23] Great. So CI is again running on an entirely separate service called spindle right.
It is currently running on our flagship one, but you could theoretically point this anywhere. And an
example of this is Mitchell H. Mitchell Hashimoto of Ghostty and HashiCorp fame has built a pretty
neat little tool called tack, which allows you to stitch any CI into Tangled. So in Mitchell's
case, he is essentially pointing his builds to Buildkite. You could send this anywhere as I said. So
yeah, that's sort of like a like a high-level overview of what you can do with I guess open
interfaces and just hackability in general.
[07:16:06] Anyway, back to it. So you can go further, right? You can even build your own forge.
So thanks to Bobbin being so light and so easy to run, you can essentially just run an instance of
Bobbin and point build your own client on top of it. And this could for example just be your own
single user forge right that for example if you are running cgit or something similar you could
instead just have your agent hack together a custom front end for you so you can have a personal I
guess readonly Git server if that's what you like but at the same time you can federate so you can
still get vanity stuff like stars and it is still under your identity etc.
Custom forges and agents
[07:16:50] So, that's awesome. You can go even further and then have your agents work alongside
you on Tangled. So, AT Protocol has a very fun story for agents, right? So, you can essentially just
have your agent write records to your PDS like I demonstrated where the where my agent just wrote a
repository to my PDS and everything just magically happened. U, it could either talk to the API or
just consume the firehose directly. It's kind of all up to you. And then you have this, it all shows
up on Tangled. Dolly is just looking up from there in, I don't know, it looks very angelic.
[Laughter.] Anyway, so yeah, in conclusion, malleable software wins.
[07:17:38] U, you can build your own tools. You just need, interfaces to do so. And when we have
these interfaces, you can essentially just hack anything into existence. And we have a pragmatic
approach or I guess a practical approach to decentralization. You can pick and choose. You can run
all of it, parts of it, some of it, or just none of it and just host everything with us. That's
totally fine. And then yeah, it's all built for humans and agents. So, open protocols and formats
are great for humans and great for agents.
[07:18:09] And then this whole architecture that Tangled and AT Protocol has arrived at is is
hyperscalable. So that's Tangled. Yeah. Happy to take any questions or
Q&A
[07:18:38] I think you said you were writing the records into your PDS for the repo. Would that
go into a knot eventually or did I miss sort of what the new architecture is like or
[07:18:46] Yeah. So I kind of skipped certain details there for the sake of brevity. So yes you
will have a record in your PDS but at the same time you will have to also make a call to the knot to
then create that repository there. But yeah
[07:19:01] Okay so the PDS is just like a pointer or something like that or
[07:19:04] Yes. So in our new model we're actually calling them repo declaration records. You
should find Lewis later and ask him about it but
[07:19:12] Sounds good. Yeah. I was just reading a blog post about like static sites, like Git
repos that were static sites yesterday.
[07:19:28] Yeah.
[07:19:29] And I was wondering like could you make a knot that's a static site
[07:19:33] 100%. So yeah, you can essentially, run a knot and then just slap a front end on top
of it because you have APIs to do so or you in fact just have the Git repos right there. So yeah,
very much so. That's a that's a great article by the way. This is less of a question because I don't
really know how to state this as a question, but [Laughter.] Tangled has has seemed to be to me to
be some of the like the leanest most chunkable architecture for a forge I've ever seen.
[07:20:11] And I really like it. I just wanted to say I really like it. It's good.
[07:20:17] Happy to hear. Happy to hear. Any more compliments are always welcome. [Laughter.]
since the knot server is now nowadays in Rust do you have any plans to integrate with JJ directly
[07:20:40] Potentially yeah this is again a question that you should ask Lewis we don't
currently have any plans to do so in the near term but yeah it's again all hackable any more
questions?
[07:21:05] No. All right. Thank you very much. Let's give a round of applause.
[07:21:15] Fascinating to hear what Tangled is and what it's knot. All right, let's welcome on
stage Toon. He's a GitLab developer and I hear an Emacs aficionado. So, let's give him a warm round
of applause.
[07:21:39] Oh, more dad jokes. Okay.
[07:21:43] Oh, no more joke. Oh, I see how it is. Oh, dear.
A view of work in flight
[07:21:47] Okay. Do we have sound? Yes. Hello. Welcome to my talk. It's titled or whatever I
titled it's can we design a git smartlog. At the moment Git doesn't have a smartlog. So, can we
add one? That's basically the question we're trying to ask or to answer here. So, first little
introduction. I'm Toon. I'm a software engineer at GitLab and you can find my socials on my website,
iotcl.com. A little fun fact about me. I like Git graphs. Like I've been talking to some folks as
well, like the ERSC folks about nice how nice Git graphs can look.
[07:22:27] And so I can enjoy when a graph looks really nice. So at some point I think it was
Git merge in Los Angeles. We were talking about like can you put something weird into Git? And at
some point I decided to put my resume in Git. So you can clone my resume and just hit git log and
you get whatever is my resume. It's a little bit outdated at the moment, but it was just a fun
experience to build this. But that's enough about me. Let's first have a look at what we got.
[07:22:58] So I guess most people know this command git log --graph --oneline. And can I maybe
ask like who ever uses this? Maybe hands. Are we still excited about raising hands? Okay, not that
much. That's what I expected. Do you use an alias if you use it or do you raw-dog type it every
time? Okay. And when you use it, did you ever regret it? I mean, if you look at git.git and do the
command on this on this repository, we see there are some lines and you can follow what's going on.
But if we go deeper, there are more lines and suddenly there are even more lines.
[07:23:41] So you can see like this escalated pretty quickly. Let's have a look like where
git log comes from. So --graph is is 18 years old. It was introduced in 2008. Oh I forgot to
look up who made it. But yeah I don't have in my notes so sorry to the person who created it first.
Anyhow, over time it accumulated about 100 commits which is I would say not that much but as you can
see like it has not been really actively developed over time. So maybe it's time that we give it a
little bit of a modernization I would say to do this let's have a little bit of a look to what else
exists in the ecosystem.
[07:24:24] And we have our friends at Jujutsu or JJ. I'm not sure if you supposed to be say
Jujutsu like I hear no one else saying it. So it's on the slide anyway. Meta has created Sapling and
we also have the friendly folks of GitButler who have something you would call a git smartlog.
They all have some kind of notion like what is your work in flight and they most of the time show it
without using any arguments. So no weird --graph or --oneline or whatever it is.
[07:24:56] So like the first example is JJ and this is like sample repository I created for this
and it shows commits that are relevant to me and it also with the tilde removes some revisions that
are basically in this history not relevant for me. Then we also have Sapling and Sapling has also
nice graph and it also has these nice annotations like the merge request or the pull request number
and approved state for those pull requests. So that's nice that you get a little bit more metadata
in your log next to the bare commits. By the way this is the super smartlog which is arguably a nice
name but you can use it in Sapling.
[07:25:39] And then we have the nice command-line tool but which also shows what's work in
progress. So what are your uncommitted changes and also shows your change ids. So there's are
various things that they do and don't do and but there are some things that they do agree on. So
most of them hide things that are not relevant to you. Especially if you look at JJ in the beginning
right with the tilde it just cuts down the graph where you don't you're not interested in what's
there also they make the default view useful like I said no arguments you just see what you need to
see without typing any weird option or argument there also few things as I pointed out before that
they do not agree on like what is relevant to me that's a little bit also an arbitrary question what
do I really care about what is relevant to me, what is relevant to my repository, I don't know.
[07:26:36] And the uncommitted changes is also something they do differently. We know JJ doesn't
have uncommitted changes, you would argue. But GitButler shows them at the top. So they do things
differently. And then they also might show some extra metadata like Sapling does with the pull
request and the pull request status. So now we can start asking ourselves a question like what can
Git do to fill this gap compared to the other tools and so as I shown git log shows the history of
the repository but it doesn't know much about state talking about the metadata it doesn't show
anything about the metadata and then on the other hand you also have git status which has
knowledge about the state for example the uncommitted files that's gets something that git status
knows but git log doesn't and yeah maybe you would say like smartlog needs both so the first in
the beginning I showed you the graph of git.git Git and we showed it's like it's really really
wide and it's get dirty pretty quick.
Making the graph narrower
[07:27:46] So can we make it less wide? As we know Git is using a directed acyclic graph. So
every commit is edge complete in this graph. So every commit draws a line back to the commit the
parent commit where it started from. One way you could work around this is by using the
--first-parent option in git log. And this option would when it encounters a commit with
multiple parents, it just omits the second and third and how many other parents and just only
follows the line to the first parent. So this simplifies the graph a lot because you only see
basically the commits that are in the master branch and not in the branches that are merged.
[07:28:33] And while this is nice, maybe depending on what you like to see, it's also removes a
lot of data from the graph because you don't see no more which commits were actually merged in. This
is an option that exists today by the way in contraries to some things I will show later but okay
that's not that important. So then you have this other thing. It's the --graph-lane-limit. That's
something was added that was added in 2.55. So the last release of Git and it was added by Pablo. I
think he's in the crowd as part of the Google Summer of Code.
[07:29:09] And what it does it limits the graph to the number of lanes you to specify. So when
the graph gets wider than those lanes, it writes a tilde to indicate there's stuff missing here, but
at least your your your graph is not getting as wide and your commits your commit titles are not
pushed all the way to the right. It's nice, but is it really what we want? Let's have a closer look
to what the problem maybe is and then we can decide whether we can design a better solution. In this
example, we have at the top the long-lived branch that is being merged into master into main.
[07:29:52] But that branch was started like all the way at the bottom. So it's really a
long-lived branch as it says it's created very long ago and after it was created there are two other
branches like branch B and branch C which were created and merged. And this is one of the reasons
that's causing the graph to draw these lines because they draw all the way from the commit where it
started to basically where it's merged. And then we can ask the question like do we really care
about everything? Like do we really know need to know where the merge branch was created from?
[07:30:33] When a mer when a branch is merged it contains the history of both branches the
parent like the parent branch where the branch was created from and the source branch which commits
are merged in. So all the changes that were like basically where the branch was started from are
indicated by the merge commit anyway that they are in that branch. So I was thinking about a new
option. So this is by the way the first option that I'm suggesting or proposing and to be clear
nothing of this is yet in Git or ready to be in Git just some ideas I'm trying to present and then
we if I hear there's some traction for them then we can start to upstream them and make them maybe
part of the Git official release.
[07:31:20] So this first option I'm suggesting is --untangle and in this graph so this is
basically the same graph as the first one again for the mer branches we just cut the line at the
bottom of the branch when there are no more commit in the branch that are are not in the target
branch. So as we see like the utmost branch the JKT-551 it merges two commits but we cannot see
where that branch was created from. And I would say that's not really interesting anyway because we
just need to know what was merged in not when it was created to be merged in.
[07:31:55] And this already makes the graph a lot less wide. And also we still have all the
commits something that we didn't have with the --first-parent. So we can compare it by side by
side. So these is basically the same histories presented in a different way. And yeah, I would say
this is a little bit more readable. Okay, that's one aspect. So we make the graph less wide, but I
was only showing the history of the main branch and that's not really what I want to see when I see
a smartlog. As we shown in the beginning with the other examples the they show what work am I
working on so what is relevant to me and that's not only the main branch so how can I see my work at
the moment for git log there is an option --all which basically takes all the local branches and
shows them in the graph so everything that has a local branch or is locally checked out or has been
checked out is shown in this graph with --all this is pretty okay.
Finding relevant work
[07:33:01] But can we do better or what is wrong with this? There are two problems I would say
with this view is like what commits are mine or what branches are mine because it shows all branches
that were that exist locally but it doesn't mean that all those branches are created by me. I might
have checked out other people's work or there might be maintenance branches I don't really care
about or like stable branches. So those are not filtered out. And then another thing with the graph,
if there's a branch that hasn't been touched like a month or so, it would get buried all the way
down in the graph because the history is old and I have to scroll down all the way to find that
branch.
[07:33:44] So can we address these two problems? Let's first have a look at the branches. So
which branches are mine? And this is another suggestion I'm making here is oh sorry. So this is the
output of Git branch for my example I'm using. So basically this is the same thing as what --all
uses to determine which branches to use in to display in the graph and it's all the branches that
exist locally as I mentioned. So what are the branches I am working on here? I'm suggesting an
option, --mine. So this is a graph where only the branches that are mine are being created.
[07:34:27] So they are shown in the graph and it simplifies my view quite a bit. But as I
mentioned like what is mine anyway? Are those branches with commits I created or are those branches
starting with my name as a prefix and a branch name or anything that's not merged yet? For my
example here I picked the first one. So I'm checking out I'm checking for every branch whether the
top commit is authored by the current user which is in the Git config and if it matches then it's
being shown when you use that option --mine.
[07:35:06] Then the other option then the other problem we were having was where are my old
branches and that's basically a different problem that --untangle it where it cuts down the width.
We basically want to cut down the height. So when you have an old branch your history might be
thousands of commits long and you won't be able to find your commits. So we need to make it smarter
and we need to make smartlog decide what to remove. So can we collapse it vertically? And here is
what I'm suggesting to introduce --fold-others.
[07:35:48] So it's pretty pretty similar to what JJ does only here is like showing ellipses for
the commits. It removes and it swings down the commit graph quite a lot. At the bottom we see there
are like 23,000 commits by others. You could also argue whether that interesting or not. And that's
why I'm also thinking about this option I call --down-to-upstream. Like I want to emphasize a
little bit that these option names are open for debate. There's nothing nothing upstreamed yet of
course but what's in the name anyway, right? So this is something to eliminate the problem of going
showing the history all the way down to the root commit because we don't know we want don't want to
see the first commit in Git or the project you're working on anyway.
[07:36:37] So can we stop where the upstream branch begins? And that's where this option comes
in and it's sort of doing the same as this command here. So if you would specify topic branch on the
on the command line to say I want to see all changes in the topic branch, it would add the inverse
of the upstream branch for that branch. So it stops that graph at that point for that branch and it
does it basically for every revision you specify in the command line or when you specify with
--mine because it under the hood does similar things.
[07:37:19] So this is obviously a lot to write out if you would like to use this. That's why
this option I was suggesting and yeah as you can see it would reduce my graph vertically a lot
because it doesn't throw all it doesn't draw all the commits down to root. Cool. So we try to make
the graph less big but can we also make it more interesting? So can we add more data to it? Because
the graph does know nothing about the state at this point. So the first option I'm suggesting here
is --ahead and I can imagine you already have an interesting or like a thought what this could be.
Adding state and metadata
[07:38:03] It basically shows how far your branch is ahead or behind on the remote. For example,
the toon-replay-linearize branch it's had two commits and it's behind three. So there are three
commits in its source or like its target branch and two commits in the feature branch itself. Yeah,
I most forges can show it as well. Cool. Well, another thing and that's I would argue maybe is a
little bit more interesting when you're using agents or at least that's what I'm hearing like I've
impressing more people's are using worktrees nowadays because they tend to do a lot more parallel
work and that's because of agents because you can have multiple agents to build multiple features at
the same time.
[07:38:53] But if you have all these worktrees, do you really know what work is happening
anymore? Because where is your work anyway? And I'm adding here the option --worktrees to show the
checked out worktrees in your log as well. So we have the toon-replay-linearize worktree which is
the current Git repository. But if we go down to the bloom filters branch we see it's checked out in
../demo-worktree-bloom. So this can help you whether you have multiple directories working on
similar work and with option ahead you can see okay it's ahead of the remote so maybe you need to
push some commits cool we saw with Sapling it also has different things about things with the super
super smartlog it finds the merge request or the pull request number and the approved state so this
is something I would not see Git to do it itself because it's very opinionated which what your mode
is and what tools you use and Git usually doesn't care too much about it.
[07:40:06] So I proposing here an --decoration-hook and this is basically in this design at
least an script that you can run to decorate every commit in your graph. And for example I've using
an example here where it prints the CI status of your changes. So it goes to the remote it or to the
API of your remote of your forge and checks whether the CI status is okay and then it shows like a
an red cross or a green tick to say like okay it's passing or it's failing. So there are actually
that's the nice thing about it.
[07:40:45] There are different use cases for this because you can yeah you can show the CI
status but you can also show the review state similar to what Sapling does and also like show the
signed state is the commit signed and is is the signature actually verified or not. So this is
hookable and user defined. Still not decided on like how you would actually configure this. It's
probably easier to do this through the configuration than on the command line, but that's open for
discussion as well. So, if you put all of this together, we create something like a monster like
this.
[07:41:23] So, if you were using an alias for what I showed in the beginning, then you better do
something similar here. So, yeah, nobody is ever going to type this by hand. So, sure, we create
git smartlog. We just add it and then everybody happy. But that's not as easy. Yesterday was
someone was saying like Git is allergic to change. I would a little bit say that's not correct but
they're cautious about change and adding a smartlog like this would be very hard to change in the
future and that's something the Git project doesn't like.
Composable options and defaults
[07:42:01] So this opinionated view is is not very fitting for Git. So Git ships a mechanism. It
gives you the tools but it never enforces workflows on the users. That applies to various things
like merging strategy. Whether you are interested in fast-forward merges or doing squash merges. All
is possible with Git. You're you don't you're not limited to anything. You only get the tools to do
whatever you like. So making git smartlog do this thing for everyone it's probably not going to
work. So what you can do as I shown in the beginning as well create a self-service alias.
[07:42:40] So we can alias a git smartlog in your Git config and then you can use
git smartlog as you would like. This has a few benefits and must have messed up my slides here but
whatever. So this is a few benefits. It's composable as I said like you can def you can choose
yourself whether which options you add to this smartlog command. If you want to see the CI status or
you want to see the commit verified status you can add it yourself and you configure it as you like.
[07:43:20] Another benefit is we can upstream this piece by piece because every option can be
upstreamed to the Git project as a separate thing and the git smartlog as a feature as a whole
doesn't need to be completely finished. It doesn't have to need all the flags before the Git project
can say okay yeah I want to merge this because yeah we can do it independently and also it allows to
add more feature f future options because when new options would be added would be we add it to the
default git log or we would we give the users the choice to add them with an alias that's very
easy to do so that gives us a little bit more flexibility So what I kind of would suggest is that we
can add some documentation about this.
[07:44:11] So this is kind of similar to what there's also m man page 7. So this is what we do
for gitrevisions. We show some documentation and we say okay this exists and this is how you can
use it. And that's probably the most future proof and also one of the ways that the Git project
maybe would be interested to add this. Then if we look at an example once more, there's well
ignoring the change ids, there's one more thing that we didn't really address and that's basically
Unicode. Git has this ASCII graph for ages and it's probably the main reason why people would think
the graph is very ancient.
Unicode rendering
[07:45:00] I would hope that we can soon upstream --unicode: a graph that uses Unicode
box-drawing characters inside Git itself, to make it more modern. To summarize, I've proposed nine
potential new flags that we can maybe or maybe not add to Git. Nothing is upstreamed yet. I'm only
proposing these ideas. I want to share them with you and hear whether there would be interest in
them. There are probably more ideas to explore as well. Cool. Thanks. That was my talk. My slides
are up in GitLab -- also hosted in SHA-256, by the way -- and you can find my contact info at
iotcl.com. [Applause.] Questions?
Q&A
[07:46:04] Oh, thanks for the great talk. I was wondering, so I was I love this idea of having a
documentation page that that documents some like useful like options for git log. I thought that
was really cool. And I was wondering if the Git like project has any mechanisms for kind of because
I know that like there are a lot of like kind of like popular options for Git that people like to
use.
[07:46:25] And I was wondering if the Git comm like Git project has any ways to kind of like
take these ideas from the user community and be like, "Hey, turns out lots of people love to do it
this way. Here's like you some information about that like about things that are popular
[07:46:40] Or like like what you've seen in the past or what you think that could look like.
[07:46:44] I'm not sure what the question is but yes there it would be nice if we would share
more best practices with the community. As I mentioned like Git usually is not that opinionated but
there definitely are some ways that we can help users in the right direction. So that would
definitely be nice. I'm not sure what where that medium should be. Git only lives in the terminal
usually. So that's why I was thinking about a man page and the man pages by the way also end up in
git-scm.com automatically. So that's nice.
[07:47:17] By the way, thanks for all the work you have been doing on the website, Julia. I
really appreciate that. [Applause.] So yeah, I'm open to suggestions. But yes, I'm happy to hear
if if we can how we can share best practices more. One thing I would say is also a little bit
undervalued, there's a monthly Git Rev News newsletter created on by Chris and some other folks. If
you're not aware of that, look it up, Git Rev News. It's a really great resource for articles in the
internet, what's happening in the Git mailing list, and it's really great to stay on top of things.
[07:47:58] Thanks, I just wanted to are you aware of with the JJ log about the how it selects
commits and how it displays them?
[07:48:19] Nope.
[07:48:20] Okay. So in your JJ configuration you can actually it has a default template of how
it how it builds out the representation for the commits. You can actually provide your own there's a
whole templating language that we have a reference for on the documentation website as well as the
revset that is used to select given your working copy what is what is selected to be in your log. So
it is fairly highly customizable.
[07:48:52] Okay, sounds great. I will check it out. Thanks.
[07:48:56] So, sorry, I had to step out real quick. So, tell me if you covered this, but I'm
curious, like one of the things that GitButler does, I think the default revset in Jujutsu, is has
like a an a concept of sort of what upstream is other than upstream for that branch, right? Like if
it if you're doing not branch@{upstream} then you push and then it disappears right
[07:49:22] So like I was wondering if you if it crossed your mind at some point to have some
concept of like what GitHub does which is have like a default branch right like origin/main and to
do the default sort of log differential between origin/main and my branch like --mine right like
not
[07:49:42] Not integrated or something like that right as opposed to just like that. That's one
of the things that I find very useful about the at least our smart smartlog implementation is nice
because I can set a default branch that says origin/main is special. So once something's merged
into it, forget it exists, right? Like don't even show it in branch listing because who cares at
that point and like to try to simplify that that set. But it is something that Git has never really
had like there's no special branch even though 95% of us have an origin/main and that's exactly
what it means, right?
[07:50:12] Is that something that you think would be interesting in a system like this? Like I
think it would be cool to have like a Git SL or whatever that you could do that says I'm going to
have a default revset that you can change the way you can do Jujutsu of like this is what I want to
see. But the default is show me all mine, not on these things or something like that, but very
specifically not on origin/main. Yeah, I skipped over a little bit quickly maybe because yeah,
it's kind of hard to determine what your upstream branch or your target branch is because the way
that I'm doing it now is whether you have configured an upstream branch or not.
[07:50:46] You could also decide whether you want to use merge base although that's also can of
worms that's not as easy as like Elijah showed out like in before noon. So yeah, there there are
different things and maybe like specifying a default branch would simplify things a lot. Yes. So I
think one of the things you talked about briefly with the smartlog or not having a smartlog command.
Yeah, sorry.
[07:51:16] Great.
[07:51:17] Is that you didn't want to like ship you didn't want to ship an opinion that was like
not, hardcoded that, people might get upset if you changed in the future. I wonder I was wondering
if you considered like shipping configuration options instead of or in addition to flags so that you
could kind of maybe ship a default that people might like for new installations or something like
that. And then people could just turn off all those config values. And then you don't have to like
put together an alias and read through an alias. Can just kind of like oh the default's good or you
know if I really don't like all that visual stuff I can just you know set all those options to false
in my Git config in my home directory something like that.
[07:52:03] Yeah then then we maybe are have to look a little bit closer to what JJ does because
maybe it seems like they have some kind of that that configuration in some kind of templating
language. So I don't know what that could look like but it's interesting idea that yes
[07:52:18] Yes it's like Sapling is the same as JJ like they have the It's not I would say it's
not about being opinionated. It's about having a sensible default. You can absolutely customize it
as much as you want. But the nice thing about it is the default out of the box is what most people
want rather than having to spend like a couple of hours setting up a tool that you've just got out
of the box. It's like and this is something that compounds over time as well. Like imagine in 20
years time and you still have the Git experience from 2005 when you run when you run you know yum
install Git you get something from 2005.
[07:52:46] That's not great. I will say as well I wrote the Sapling graph renderer like the
graph renderer the Sapling uses. I think JJ uses the same one. Like is there any appetite you get to
change one because like the graphs with all the especially with the wiggly diagonal lines look
pretty horrible. Like one of the things that saplings graph renderer tries to do is make lines as
vertical as possible like avoid having too many horizontal or diagonal lines. I think that produces
more beautiful but also easier to understand graphs.
[07:53:16] Is there any any appetite to do something like that in the Git graph renderer as
well?
[07:53:22] Yeah, two things. So first was also about opinions. I personally would like to ship a
git log or Smith git smartlog that would do the sensible default and whenever some user then
upgrades their Git they get new features for free. That would be I think would be beneficial for the
user because otherwise they have to read every every time the release notes or the documentation
like what is new, what can I add. So that's annoying. So I would not be against that, but yeah, it's
sometimes a little bit harder to get things like that done in the Git project.
[07:53:54] Then the other thing I was made aware about the graphs being for Sapling and also for
JJ being rendered by renderdag which is a Rust crate.
[07:54:09] I really
[07:54:10] Yeah. [Laughter.] So I was made aware of that and I don't know now we're talking
Rust in Git anyway. So maybe we can reuse it. I definitely want to explore that idea. So I'm will be
thankful if you actually yes do the less tangling of branches because that's really an issue for the
user. So that's part of the problems of this. In my proof of concept for Unicode that I wrote it's
basically reusing the same graph engine. Then the ASCII one with few few small changes but if we can
borrow things for other project that would be great.
[07:54:45]renderdag also has an ASCII mode if you want to have like backwards compatibility
as well. So you have it's not it's actually not that great but [Laughter.] because ASCII is pretty
[07:54:54] I don't want to touch ASCII anyway
[07:54:56] I had to put in because of back when I wrote it there were still like Windows
machines using 8-bit encoding so I had no choice.
[07:55:05] Thanks. So I wanted to clarify like the remote server actually does announce the
default branch to the client as well by using the HEAD revision because that is typically symbolic
reference by that you do know what the remote default branch is and you can use that information and
I would also say that we should feel a little bit more comfortable in the Git project to really
become a little bit more bold and I would say that this should totally be something that we have by
default and not only in docu in documentation going forward because like our documentation is
already huge and nobody's really able to like read through all of it and having one more man page is
probably something that people will just not really notice even exists.
[07:55:44] So I think we should just like try to implement that which will lead to lots of
bikeshedding of course but I mean we can navigate that.
[07:55:57] Thanks.
[07:56:03] Okay. Yeah, I hope you all stay around for the next talk which is also about SHA-256.
So,
[07:56:09] All right. Thank you very much. Let's have a big round of applause. [Applause.]
[07:56:16] Up next, we've got Emily, who is a staff engineer at Google. She leads a team that's
now working on JJ, but in the past has made contributions to the libification of Git and other
notable work. So, let's give her a big round of applause.
[07:56:41] Are we good?
[07:56:44] Yeah.
[07:56:48] Why can't a nose be twelve inches?
[07:56:51] No,
[07:56:52] Because then it would be a foot. [Laughter.]
[07:56:55] That's awful. Wow. So I have really good news before I start is that we don't have to
do any more of those because this is the last talk. Hi everybody. I'm so happy to see everybody
here. Yesterday was great. Today has been great. Thanks everybody for hanging in there for like two
long days of talks for most of you. One long day of talks for some of you. I'm the last thing
between you and the afterparty. I know. So I'm going to try to be really interesting. I promise.
[07:57:24][Laughter.]
[07:57:25] So first thing before I start, if you want, you can follow along with the slides on
my QR code, short URL, whatever. If you're like me and you can only hear Netflix with the subtitles
on, you're in luck because I have a bunch of stuff in the speakers notes. You can use that basically
as closed captions if you want. The talk contains some technical depth. I prefer not to interrupt
it. So, just like the rest of the speakers, just hang on to your questions till the end. Make a note
or if you really want and you don't want to wait for the microphone you can leave a comment on the
slides on Google Docs and I will reply sometime later after the conference.
[07:58:02] So, awesome.
Background
[07:58:05] It is so nice to see so many familiar faces and so many brand new faces at this
conference. I know a lot of you but not all of you. So I'm going to introduce myself for you those
who don't know me yet. I'm Emily. I've been working at Google for just over 10 years. I just had my
10th anniversary like two weeks ago. Yay. [Laughter.] Thank you. Thank you. The vast majority of
that time has actually been spent working on open source code. So, I've been working on open source
for over nine years, starting with the OpenBMC project and the IPMI driver stack for that project.
[07:58:39] That was cool. And I've been working on Git and version control for over eight years
and actually under one year working on JJ formally although I was involved in the setting up the
governance for JJ like two years ago something like this. So I am now the TL of the JJ development
team here at Google. I also really love regular expressions and regex crosswords. So fun fact about
me. So first things first today we're going to talk about the background. So what is SHA-256? What
methods are available for us to move to SHA-256 if we want to adopt it.
[07:59:19] We're going to discuss some ways that the rest of the ecosystem relies on SHA-1 today
to refer to Git repos. And then we're going to at the end cover what it takes to adopt SHA-256 at a
hyperscaler like Google right now and what we're looking forward to in the future with the SHA-256
ecosystem. So, what is this whole SHA-256 thing? We already use SHA-1, but what does it mean? I'm
going to try to go kind of quickly. I think that most people in the audience already kind of know
the basics here, but I do want it on the recording for later, so please bear with me.
What the object hash means
[07:59:54] What does the hashing algorithm mean? Okay, let's do this one more time. Git is based
on a Merkle tree. The ID of each layer of object is based on the content in that object. So blobs
are based on the content of the file. Trees are based on the ID of the blobs inside the tree. And
commits are based on the ID of the trees inside the commit as well as the metadata in contained in
that commit. So these hashes are supposed to be one-to-one. That means that one hash can only be
generated by one set of bytes and vice versa.
[08:00:27] This property is key to Git's distributed nature. So I know that my copy of the Linux
kernel is the same as your copy of the Linux kernel and we both have the same copy as Linus Torvalds
because they all have the same ID at top of tree and that guarantees that all of the other IDs
through history are also the same. However, if I can artificially manipulate the input to that hash
function and cause it to create a specific output ID, that is if I can cause a collision, I can
create an impostor copy of the repository.
[08:00:59] So classically, let's say that I manipulated this blob B2 here to contain a Bitcoin
miner. Git cannot tell that B2 is different because T1 and C1 like so so these also all have the
same hash because B2 is coming out to the same hash. So my entire tree looks the same except I have
this sneaky little bit of code because this promise that the ID specifically identifies the content
is broken. SHA-1 right now is broken. So it is possible to generate the same hash from more than one
block of input. Right now I like to say that it's not usefully broken which means that it's very
computationally expensive to generate this kind of colliding object and it's really difficult to
like work in a payload that's useful like my Bitcoin miner.
[08:01:49] However, we can brute force these collisions and computationally expensive becomes
practically cheaper every day. All you need is a big data center with a lot of GPUs in it. I'm
really glad we haven't been building any of those over the last couple of years. So, enter SHA-256.
This is a new generation of hash algorithm. It is currently unbroken. The output of SHA-256 is
longer, which means that consumers of the Git object ID need to adapt to learn about this longer
hash ID. And the SHA-1 output is not in any way predictive of the SHA-256 output.
[08:02:26] Meaning that you need to calculate the hash from the source to identify with either
the SHA-1 or the SHA-256 algorithm. The you cannot just recalculate the SHA-1 output. Git already
supports SHA-256 repositories thanks to so much hard work from brian m. carlson. Let's clap for
brian. They're so great. What a great person. And support from Thank you. Support from forges is
still on the way. So that means that you can create a local Git repository using SHA-256, but you
can only push it to a handful of forges in order to share it with others.
[08:03:00] This slide is actually a little bit stale. I found out literally this morning that
GitHub has SHA-256 in private preview. Thank you again, brian. So great. We solved the collision
injection crisis. We can adopt SHA-256. That's so easy. That's my six minute talk done. Great. The
naive approach here is to just say like, okay, let's just cut over in the middle of history. Like
next Monday we just start using SHA-256 objects. That should be really easy, right? So in this
diagram and the ones that follow, I'm indicating SHA-1 ids with a numeral and SHA-256 IDs with a
letter and italics.
Why a mixed object graph does not work
[08:03:37] So like CA represents the hash of that commit. So the first problem, do I really want
to mix SHA-1 and SHA-256 hashes into the same object? Besides this being just kind of ugly, we also
actually don't have any strong security guarantees this way. It's still really easy for somebody to
distribute their own version of my repository that contains a malicious injection. And this like CA
ID, it remains untouched because it's got this like SHA-1. It's depending on the SHA-1 and not a
SHA-256. Plus, it kind of gives us a false sense of security.
[08:04:15] We're like, "Oh, yeah, it's SHA-256. I'm great." And like, is this still a problem?
So, this doesn't work. Okay, so next thought, let what if we just like do a cut point? So, like
today, close of business, I'm going to say this is the last SHA-1 ID and then next week I'm going to
come back and we're going to start pushing SHA-256. We don't have like mixes inside a single commit.
And we're just going to like go from here. We also run into issues syncing this across Git's
distributed nature. So like how are we going to accept a push if there's a push based on the
outdated top of tree?
[08:04:53] How are we going to do that across a distributed set of hosts? If like not all of the
hosts have this like CA thing. Let me see. I also had Yeah. Oh, I was going to say too, we also
still have this weird mixed hash commit. Like you can see even in the commit object we still have
too many things going on. It's still kind of ugly. So with all of these approaches we still end up
with like latent injected collisions if you will. So that means that like I can sneak my Bitcoin
miner into history and I just have to convince you to run git bisect on my copy of the repo or you
can check out an old tag or something because I like told you there's a bug there and then I have
you know bitcoins from your machine.
[08:05:33] Thank you very much. Ultimately, the presence of a SHA-1 object anywhere in the tree
reduces the overall hash strength of that tree to the same strength as SHA-1. This is true for
submodules too. A tree that contains a submodule that's hashed as a SHA-1 cannot guarantee the
overall validity of its components. Even if the rest of the tree is in SHA-256, a malicious object
in the submodule can sneak in the SHA-256 hash of the superproject remains unchanged. And in fact,
this is even worse than in those like mixed history modes that I was talking about because if I
point to a compromised remote, then the malicious object can just exist at top of tree and we don't
even have to notice like we don't have to run Git bisect.
[08:06:18] We don't have to do anything. My Bitcoin miner is just there. So therefore, Git does
not and Git will not support mixing SHA-1 and SHA-256 hashes in a given repository in any way ever.
No. Projects need to adopt SHA-256 wholesale. The entire Git history needs to be converted based on
a known correct copy of the repository. There's a few types of projects where this is a problem for
adoption. Let's just hold on to that thought for a second and I'm going to get to them in a moment.
So for a single repository to convert it's maybe like not such a big deal right you like spend a
couple minutes you recompute the repository and then you have enhanced security posture easy great
that's my eleven-minute talk done however we also have the rest of the ecosystem to consider so
because the commit ID unambiguously identifies the complete state of your repo it's used both inside
and outside of Git in all kinds of places so it's used in CI/CD where your hashes indicate whether
you want to do a full build or an incremental build.
Object IDs throughout the ecosystem
[08:07:23] It's used in releases so you can like identify your commit train and or your your
release train entirely by the commit ID. It's used in passive scanning where we can look for bad
objects that are like like known bad objects in your dependency graph anywhere. It's used in
ticketing to tools like you open up a Jira ticket and you say there's a problem with commit one 345.
It's used in cross references. So you can say this like in email or in Slack or whatever since
commit ABCD this bug is solved therefore let's do blah blah blah thing like these kinds of
historical references are everywhere and it's also used in Git itself in signed tags and this is
kind of a an interesting case because the signed tags contain a plain text commit ID that is being
referred to so even if you're like rehashing the whole thing the sign tag is still talking about the
SHA-1 ID this is a little messy as So remember what I said earlier about the SHA-1 not being
translatable to the SHA-256 by just looking at the output alone.
[08:08:25] So this means that without any way to look up these hashes to like look and say I
have my SHA-1 ID what SHA-256 does that correspond to or vice versa all of these external references
from the last slide are going to break and we don't have any way to tell what code we were talking
about way back when. And by the way, I also mentioned that the format of the SHA-256 output is
differently sized from the format of the SHA-1. And so that means that even if you're not really
worried about continuity, your ecosystem still needs to be updated to accept this longer
64-character SHA-256 hash.
[08:08:58] And so simply truncating the SHA-256 hash to 40 hexadecimal characters, like maybe
that would kind of work, but you don't have any guarantee of uniqueness. So you're probably just
going to be confused in the long run. Like you can't tell which type it is and then you're going to
run into a collision and yeah don't do it that way please. So okay this is complicated. What does
Google do about it? Right? Google is like big big we have a lot of moving parts right. So the
migration sounds really painful. And for us it becomes a problem when our users begin expecting that
SHA-256 is going to just work on our infrastructure.
Adoption at Google
[08:09:37] And we expect this to happen in three stages. So the first stage is when we have some
early adopter. So, we have somebody who like came to Git Merge today and was like, "Oh, look at all
these SHA-256 things. I'm going to make one. I'm going to try it. This is so cool." for better or
worse, this isn't really a thing at Google. So, Google has very strict rules about what types of
code storage are acceptable for developers to use. So, if you're just like some random hacker and
you're saying like, "Look at my SHA-256 repo I made," like you can't really treat that as like
Google source.
[08:10:07] Sorry, it's not really going to happen. The second step is when Git 3.0 lands and
this is one that we're much more worried about. So, Git 3.0 since it's a major release, it's going
to contain some breaking breaking changes. One of those breaking changes is to the default hashing
algorithm that we use for new repositories, meaning that repositories we create with git init
whenever Git 3.0 comes out, which we thought was maybe going to be this month. I think it's still
going to be quite soon. So this means that we have a lot more developers who are making new repos
and then trying to check them into our hosting infrastructure and getting an error message and
saying like hey I don't know what happened I just ran git init and now my stuff doesn't work like
please fix it P1, P0 you know and the third step is when we start wanting to rely on some open
source project as a source dependency for something else we're already bu building at Google excuse
me but that project that we want to rely on is hashed with SHA-256 and at that point we need some
way to ingest that project and keep track of the version of that SHA-256 project because that code
is something that we want to consume and contribute back to from within Google.
[08:11:18] So this becomes a much bigger problem finally this is like way way way later this
becomes a much bigger problem when SHA-1 is actually truly broken. So like true death is what I
would say. If when that happens, we can no longer depend on the content of any repository that we're
pulling from outside of Google if it's hashed in SHA-1. So when this happens, we need to move
everything really really fast. And it's not just going to be us that's like scrambling here. The
entire industry is going to have to come, have some moment of reckoning here.
[08:11:52] Maybe raise your hand if you remember Log4j Yeah. So, SHA-1 collisions are only going
to excuse me only going to keep getting easier to generate. So, they can be brute forced with the
application of enough computing power. Quantum computing also makes this easier. And like I said,
we've been building so many high-powered data centers to support this current AI boom that this
timeline is only going to keep accelerating. So, what can we do to like mitigate this at Google? As
I mentioned before, we don't really believe that this like early adoption thing is going to be a big
risk.
Temporary interoperability bridges
[08:12:31] We can just move straight past that. The Git 3.0 release actually turns out to not
really be the end of the world either. So, Google distributes a system level Git config to two out
of the three major development platforms that our users use. So this is like a system-level
configuration --, /etc/gitconfig I think is where that that goes by default. So if we're not ready
to accept SHA-256 by default, we can change the default back in our system Git config and fudge
things for a while. Yeah, and then when the time comes for us to start importing SHA-256 repos back
into Google, if we're not ready to accept SHA-256 throughout our ecosystem, I have this on another
slide.
[08:13:17] Great, perfect. This makes way way more sense. Okay. Then we can start using Copybara
to kind of fudge stuff and rehash it back into SHA-1 for internal use. So let me talk about Copybara
just for a second if you're not familiar with it. Copybara is a Google developed fully open source
mostly open source tool that is used for importing code and other metadata and for keeping it in
sync between codebases that are managed with different dependencies. It can import between Git,
Mercurial, Piper is our internal Perforce clone thing for our monorepo.
[08:13:52] And it's also really easy to extend to other version control systems and it can also
sync some types of metadata between actual specific forges. So, Copybara very cool. External repos
are typically already imported into Google's monorepo or our internal Git hosting using Copybara. We
already have written an extension to Copybara to work with SHA-256 as a data source. And so we can
bridge the gap for a little while by relying on external SHA-256 repos. But by building our own
infrastructure to convert them back into SHA-1.
[08:14:27] This has a couple of problems. So one is that iterative updates to those dependencies
is really tricky. Once we've already converted it to SHA-256, that hash isn't the same on both sides
of the like the Copybara bridge and we can't really like test this out in scale and see just how bad
it is because many forges aren't supporting SHA-256 yet. This slide is also stale again as of this
morning. Actually, we can make a test repo on GitHub. Thank you, brian. [Laughter.] we are also
missing a way to convert between the like the new SHA-256 ID and the old SHA-1 ID or vice versa if
we're converting it to SHA-1.
[08:15:11] We don't have any kind of like lookup table. So one thing we can do with Copybara is
annotate our imports. So on the like converted SHA-1 commit message, we can add the SHA-256 ID that
it came from. That's okay. It's not great. It's a little difficult for us to like search against for
example commit ID based vulnerability scanning and it's not super easy to use when you're looking up
like a cross reference from a ticket or from a chat message or whatever. So I guess that means that
we're like stuck making more and more elaborate ways to like stick with SHA-1 and stay living in the
past forever, right?
A gradual transition with two server views
[08:15:52] Like it's a little dire. But there's a concept again brian to the rescue there's a
concept that is not expected to land with Git 3.0 but that is coming and that makes this all a lot
simpler. So brian is working on a feature to enable server-side interoperability between these two
hash algorithms and that means that a server will keep a source of truth tree hashed in SHA-256 the
stronger hash and it will also maintain a SHA-1 rehash of that same tree and it will be able to
serve either one to a client that's connecting to that server and from the client side that means
that I can talk to the same repo URL and I can say give me SHA-1 or I can say give me SHA-256 and it
they both just kind of work.
[08:16:37] This interoperability layer enables us to do a gradual transition of our huge complex
repositories that have lots of contributors and lots of forks. So, of course, I'm talking about
Android. Android has hundreds of submodules. It has a massive complex commit history. It relies on
close work with a ton of vendors who all are maintaining their own forks. Android receives a
staggering number of commits somewhere in the tree every minute. So it's non-trivial for us to just
like call a flag day and move everybody over. But forge level interop and a model for forge level
interop lets us do a gradual transition.
[08:17:15] So in stage one we have what we have today. Everything's in SHA-1. Everybody's
speaking SHA-1. Everybody just pretends that SHA-256 isn't there. In stage two, we turn on server
level interop. This might take a while because I think we also have to turn it on for the forks that
our vendors are using probably more places than we think. The server source of truth at this point
is in SHA-256, but all of our clients or approximately all of our clients are still using SHA-1. In
stage three, our clients and tools start to slowly learn about SHA-256.
[08:17:51] So we'll have some portion of the clients cloning as SHA-256, some other portion
still cloning as SHA-1. And everything kind of just works with the server interop. And then in stage
four, finally, we have cleaned up the long tail of all of our tooling. Everybody is just cloning as
SHA-256. We can look in the logs and see nobody's using SHA-1. We're done. That means we can turn
off interop. Yay, we have SHA-256. So this process probably takes years. However once we pass stage
two we actually start to gain a lot of safety from the true breakage of SHA-1 because our SHA our
source of truth is available in SHA-256.
[08:18:33] This means that we can notice more easily if a SHA-1 repo that we were depending on
has been changed after the fact because when we go to convert that SHA-1 repo to a SHA-256 repo in
order to store it, we will notice that the content is different than it used to be. It will produce
the same SHA-1 hash, but it will produce different SHA-256 hashes. Or if we're comparing notes with
another mirror that also has SHA-256 interop turned on, we can notice that the SHA-256 IDs do not
match anymore.
[08:19:06] Double like yeah, this just gives us a little extra safety. So we that means we can
actually take our time transition our tools and vendors even in the event of like a real useful
SHA-1 breakage. That's about what I had. Here's the link to the slides once more. And we have a
couple minutes for questions if anybody wants to ask them. Mark, [Applause.] so my question is
about submodules. So the submodule pointer presumably in the new to SHA-256 repo is the SHA-256 of
the commit in the submodule source. Is that right?
Q&A
[08:19:46] Yeah.
[08:19:47] Yeah. So what happens if in your commit history you have a submodule pointer to a
SHA-1 hash to a repo that no longer exists and therefore there is no known SHA-256 for that commit.
I think that you end up having to rehash it and put the rehashed version into an archive and point
to that in your rehashed superproject which is so you have to like actually delete the submodule
pointer because there's no there is no possibility of rehashing if the repo is completely gone like
it's on like I think you have to do some recursive thing there.
[08:20:18] Yeah. Thanks for the talk. So after you cut over to SHA-256, are like old cross
references that reference SHA-1 just going to be totally broken forever or
[08:20:39] Yeah. So this is part of the infrastructure that I think is still kind of missing. I
so what seems very handy to me and brian can correct me if this is not part of the interop story
already is to have during this interop stage some kind of like lookup table and then to eventually
archive that lookup table. So like maybe this year I can ask the remote like I have a SHA-1 tell me
which SHA-256 it is or vice versa maybe five years from now I do that like from Gitiles or something
right I'm not sure of curiosity because you said that like SHA-1 is exploitable technically.
[08:21:29] Are there any like known cases of anyone actually exploit or like using the using a
hash collision to inject possibly malicious code or false information or something like that.
[08:21:49] Yeah. So maliciously no not to my knowledge. So Git itself actually doesn't use
specifically pure SHA-1. It uses SHA-1 with collision detection which means that the handful of
known SHA-1 collisions are like hardcoded in SHA1DC. And so any so far known collisions we are a
little bit immune against for a while. To my knowledge, nobody has managed to stuff in a Bitcoin
miner yet. [Laughter.]
[08:22:19] All right. Thanks. You talked about the having mapping between SHA-1 and SHA-256 that
you can ask the remote. But what if the remote is malicious and is using the SHA-1 to 256 mapping to
give you a wrong the wrong 256 content? Like what do you have any plan for that like a more unified
way to decide which part of the mapping you can trust or not? That is such an interesting question.
I haven't thought about that at all. [Laughter.] yeah, I mean I think that this is maybe a place
where get being traditionally a distributed version control kind of helps and where relying on a
single centralized source of truth kind of comes to hurt you, right?
[08:23:08] Because when one person who's not you is allowed to tell you what's true or not,
you're always going to have some trust issues, right? But if you have lots of mirrors that are
independently calculating their own hash based on the patches, then I think it's a little bit harder
for that to kind of sneak in. It is also the case that Git makes it really easy for you to
recalculate at clone time or at fetch time and verify your repo. So like in your example like you
have malicious SHA-1, you have a lie for a SHA-256 repo.
[08:23:43] I clone your lie of a SHA-256 repo. I can run, like I can recalculate the whole tree
and then it'll turn out to be something else because the client side is recalculated.
[08:23:56] But the hash would be correct in that case. Like the SHA-1 would be a collision and
the SHA-256 would be the correct content
[08:24:05] For the commit you to get, but it's not
[08:24:08] You're saying to only lie in the case where you're cloning the SHA-1.
[08:24:12] So So you know a SHA-1 that you want to update to. But because of the collision there
is two SHA-256 that it could match and if you ask a good clone is going to give you the good SHA-256
but if you ask the malicious remote is going to give you the wrong SHA-256 that still hash properly
when you check the repo but that contain
[08:24:38] Yeah the wrong content. Yeah. And I think that's when I would want to rely on having
a lot of mirrors and a lot of like the community of the distributed thing to tell me like five out
of five out of six of my mirrors I'll say that the correct SHA-256 hash is AAA. Your mirror says
the correct SHA-256 hash is BBB. I'm going to look into what's the difference here.
[08:25:02] Okay. So like the lookup would ask multiple people to get to make sure that they
don't ask the bad actors. I'm sorry I didn't catch what you're saying.
[08:25:10] So, so you mean having asking multiple people to reduce the number of people you need
to compromise to serve the bad content?
[08:25:21] Yes, I think you're say okay. I think you're saying to have lots of opinions so that
you have to compromise more people in order to compromise the whole repo. Yeah, I think I agree with
that. There's I think a couple
[08:25:38] From an like end user perspective if you like if I look at a commit ID like a
truncated one can I tell if it's a SHA-1 ID or SHA-256 ID?
[08:25:47] This is the one thing I didn't include the talk and I don't have off the top of my
head I think it is a little hard to tell. brian can correct me because I think they're still just
using like 16 16 characters in the alphabet, right? So they should look kind of the same from the
truncated.
[08:26:02] Correct. That you can't tell whether a truncated object ID is one or the other. We
could have prefixed it but we didn't do that. So
[08:26:10] Yeah, thanks brian.
[08:26:14] Possibly related dumb question.
[08:26:18] Is this all being Yeah. Yeah. Yeah. Is this whole thing being done in a way that like
when SHA-256 is broken or at risk or whatever, like it's not going to be a technical huge slog, but
will still be an ecosystem huge slog because that part seems unaddressable like is the
infrastructure wise like in Git?
[08:26:36] I really hope brian still has the microphone.
[08:26:39][Laughter.]
[08:26:40] So yes, one of the things we did in the SHA-1, SHA-256 transition is we updated a lot
of the Git test suite to use things like references that are like three commits beforehand instead
of hard-coded object IDs. So the extent is basically just add the new hash algorithm and then update
the Git test suite and then it's done. And so the Git test suite is in a much better place because
we have the infrastructure to support multiple hash algorithms. So it's much easier to add support
for that. It's still a little bit of work, but I expect like you could put probably one engineer on
this for like a month and have the problem solved.
[08:27:18] Awesome. I think this has to be the last one so that we can all go have wine. So,
who's got the mic now?
[08:27:26] Okay.
[08:27:26] Is there a performance regression for adopting SHA-256?
[08:27:30] SHA-256 is more expensive to compute than SHA-1.
[08:27:32] Do you know how much?
[08:27:34] I do not have the profiling off the top of my head. Actually, SHA-256 is faster than
SHA1DC because we're using SHA1DC by default in many different places and SHA1DC has extra
computations on top and these make SHA-1 slower compared to SHA-256. But if you don't use the SHA1DC
and plain SHA-1 then it's a regression in performance. But to add to your question like how like how
hard it is to add a new hash at least in Git I did that exercise on April Fools' Day and proposed to
switch back to MD5 and that was basically like it took me two hours and then like almost all the
tests in Git were already passing.
[08:28:13] So it's really like brian did very good work there. Yeah like the ecosystem still has
to move of course but like in Git it's very easy. I think that's the harder part is the ecosystem
and I think that's true with the SHA-256 transition as well. The ecosystem is just going to be very
difficult. Okay. Somebody from GitButler want to tell everybody that they can go drink.
[Laughter.][Applause.]
Closing
[08:29:47] I would like to do one final big round of applause for all of our speakers who came
here and gave amazing talks. I would really like to thank Caleb for helping out at the last minute,
coming in.
[08:30:03] And so, a round of applause for Caleb for doing a great job. I know we still have
another day of this, but I do want to do one more round of applause for all of our sponsors. There's
a lot of people in the audience here sponsoring all of this stuff, including the party you guys are
about to go to. Thank you. I mean, it's so funny. I've never had a problem remembering all of the
sponsors [laughter] of a Git Merge conference, because usually it's two.
[08:30:30] So again, thank you, everybody. Please talk to the Tangled guys, talk to the OpenAI
guys, talk to everybody that's here. Everybody's working on such interesting [unclear] in this space
right now. It's really an exciting time to be doing Git stuff. So, thank you, all of our sponsors.
[08:31:15] The GitButler people were like, "Please mother-duck them," so you get as many of them
to actually get to the place as possible.
[08:31:25] So, we're going to take one minute to gather all the stuff. I'm just going to head
over there. If I can get at least a good number of you to pretend to follow me for a while, then
most of the rest of you that haven't been paying attention will probably follow those people, and
we'll probably pick up a handful of people on the way. That's also fine. So again, thank you,
everybody. I'll see you over there.