GitHub issue: NixOS/nix#7730
Nix flake lock files can become very big: they include locks not only for the immediate inputs of a flake, but also for all transitive inputs. It makes update semantics weird: transitive locks can get out of sync with what's in the lock files of the inputs.
In addition, there are fundamental design issues with the current lock file format and semantics with respect to overrides and "follows".
The root cause is that the current lock file stores the result of override/follows resolution, but not its provenance. A follows edge on a node doesn't record whether it came from an override in an ancestor flake, or from the input's own flake.nix. This forces lockFlake() to do unnecessary refetches of inputs that are pinned and unchanged (the mustRefetch logic in src/libflake/flake.cc).
Example: suppose the top-level flake has an input foo:
{
inputs.foo.url = "github:foo/bar";
}and foo's own flake.nix declares a follows for one of its inputs:
{
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs";
nixpkgs-stable.follows = "nixpkgs";
};
}The top-level lock file then contains a node for foo with a follows edge for nixpkgs-stable:
{
"nodes": {
"foo": {
"inputs": {
"nixpkgs": "nixpkgs",
"nixpkgs-stable": ["foo", "nixpkgs"]
},
"locked": { "...": "..." },
"original": { "...": "..." }
},
"...": "..."
}
}This is indistinguishable from the lock file produced by a top-level override inputs.foo.inputs.nixpkgs-stable.follows = "...". That matters because when such an override is removed from the top-level flake.nix, the old lock file entries cannot be reused: the input that was overridden away (here foo's real nixpkgs-stable) was never locked, so foo's flake.nix must be refetched to rediscover its declaration and lock it (NixOS/nix#5289).
Since lockFlake() cannot tell the two cases apart, it conservatively refetches in both: whenever it encounters a follows edge that has no corresponding override in the flake.nix files parsed during the current run, it refetches the input — even though in the example above foo is pinned to a revision and nothing has changed, so refetching is guaranteed to reproduce the exact same locks. This happens on every evaluation of the top-level flake, and it cascades: the refetched input's own inputs are then subjected to the same check, so a chain of dependencies that each self-declare a follows gets refetched all the way down. (The trustLock mechanism limits the check to inputs of flakes whose flake.nix was actually parsed in the current run, but the root's direct inputs always are.)
The new lock file format will have version 8 (current version is 7).
Unlike the old lock file format, which stores a graph of all transitive inputs, the new lock file format will only store the immediate inputs of a flake.
Thus, for a flake.nix like this:
{
inputs = {
foo.url = "github:foo/bar";
bar.url = "github:baz/qux";
};
}the lock file will look like this:
{
"version": 8,
"locks": {
"foo": {
"original": {
"type": "github",
"owner": "foo",
"repo": "bar"
},
"locked": {
"type": "github",
"owner": "foo",
"repo": "bar",
"rev": "abc123",
"narHash": "sha256-..."
}
},
"bar": {
"original": {
"type": "github",
"owner": "baz",
"repo": "qux"
},
"locked": {
"type": "github",
"owner": "baz",
"repo": "qux",
"rev": "def456",
"narHash": "sha256-..."
}
}
}
}Note the absence of the root node.
The foo and bar inputs may have their own inputs, but these are not stored in this lock file; they are locked by the flake.lock files of foo and bar, and will be lazily fetched when needed.
Transitive lock files, if they exist, must be complete. I.e. they must have a lock for every input. It is a fatal error at evaluation time if we encounter a transitive lock file that is missing a lock for an input that is actually used. This ensures the "hermetic evaluation" property of flakes: if a flake is locked, all of its transitive inputs are locked, either by the top-level flake.lock or by a transitive flake.lock. We never fetch an unlocked input at evaluation time.
"Follows" are not stored in the lock file at all. For example, given a flake.nix like this:
{
inputs = {
foo.url = "github:foo/bar";
nixpkgs.follows = "foo/nixpkgs";
};
}no lock is recorded for nixpkgs in the lock file. The "follows" is resolved at evaluation time, i.e. the nixpkgs input will lazily fetch the nixpkgs input from the foo input when it is needed. Note that this requires (lazily) fetching the lock file of the foo input.
Non-follows overrides are stored in the lock file (since they need to be resolved to a locked input). For example, given a flake.nix like this:
{
inputs = {
foo.url = "github:foo/bar";
foo.inputs.nixpkgs.url = "github:my-org/my-nixpkgs";
};
}the lock file looks like this:
{
"version": 8,
"locks": {
"foo": {
"original": {
"type": "github",
"owner": "foo",
"repo": "bar"
},
"locked": {
"type": "github",
"owner": "foo",
"repo": "bar",
"rev": "abc123",
"narHash": "sha256-..."
}
},
"foo/nixpkgs": {
"original": {
"type": "github",
"owner": "my-org",
"repo": "my-nixpkgs"
},
"locked": {
"type": "github",
"owner": "my-org",
"repo": "my-nixpkgs",
"rev": "ghi789",
"narHash": "sha256-..."
}
}
}
}That is, the keys in locks are slash-separated input attribute paths. locks contains both the top-level inputs and any overrides.
Overrides can be arbitrarily deep. For example, if you specify
inputs.foo.inputs.bar.inputs.nixpkgs.url = "github:my-org/my-nixpkgs";the lock file will record the override under foo like this:
{
"version": 8,
"locks": {
"foo": {
"original": { ... },
"locked": { ... }
},
"foo/bar/nixpkgs": {
"original": { ... },
"locked": { ... }
}
}
}(Note that there is no lock for foo/bar, since it's presumably locked by foo's lock file.)
Note that due to the laziness of the locking algorithm, such overrides may not be validated, e.g. foo/bar may not actually have a nixpkgs input. That's fine for correctness, but worth noting that a user may not get an error message if they mis-spell the override path. (We can validate overrides as soon as we fetch foo/bar, either during locking or during evaluation, and issue a warning.)
In case of conflicting overrides (e.g. inputs.foo.inputs.bar.inputs.nixpkgs.url at top-level, and inputs.bar.inputs.nixpkgs.url in foo), the outer-most override wins (in this case, the top-level override).
The new lock file format avoids the "follows" problem by construction: follows are not stored in the lock file at all, and overrides are stored together with the input that declares them, so their provenance is explicit and "did an override change?" can be answered locally without fetching anything.
The overall semantics of "follows" remains the same: "follows" are relative to the declaring flake (where the declaring flake for "follows" overrides given on the command-line is the top-level flake).
"Follows" can resolve to other "follows", so resolution continues until we hit a non-follows input (while detecting cycles).
It is allowed for an input not to have a lock file. In that case, the referring lock file must lock the inputs of that input.
For example, if the top-level flake.nix looks like this:
{
inputs = {
foo.url = "github:foo/bar";
};
}and foo has no lock file, but does have a flake.nix like this:
{
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs";
};
}then the top-level lock file will look like this:
{
"version": 8,
"locks": {
"foo": {
"original": { ... },
"locked": { ... },
"locks": {
"nixpkgs": {
"original": { ... },
"locked": { ... }
}
}
}
}
}Note: if we update the foo input, and it now has a lock file in the new revision, we delete the locks attribute for foo.
Overrides take precedence. E.g. in
{
"version": 8,
"locks": {
"foo": {
"original": { ... },
"locked": { ... },
"locks": {
"nixpkgs": {
"original": { ... },
"locked": { ... },
"locks": {
"bar": ... A ...
}
}
}
},
"foo/nixpkgs/bar": ... B ...
}
}we use B as the lock for foo/nixpkgs/bar, not A. It might be tempting to omit A from the lock file; however, if we remove the override, we need to fall back to A without having to refetch/relock foo/nixpkgs.
Relative path inputs (inputs.foo.url = "./subflake") are not recorded in the lock file. This means that updates to a flake.nix/flake.lock of a relative path input will propagate automatically to the referring flake. This does not violate the hermetic evaluation property of flakes, since they're part of the same tree (i.e. a change to the Git revision of the subflake would also change the Git revision of the referring flake).
nix flake metadata will now only show the immediate inputs of a flake, not all transitive inputs. Apart from avoiding an expensive fetch of all transitive inputs, this is also psychologically less intimidating for users (by default, they don't see inputs they may not need to care about). A new flag --transitive will be added to show all transitive inputs.
Unchanged behaviour: --override-input can override any input, including transitive inputs.
Nix will continue to support format 7.
If the user runs nix flake update on a format 7 lock file, it will be updated using format 7 semantics. Switching to format 8 requires recreating the lock file (--recreate-lock-file).
For new lock files, initially we will continue to default to version 7. We will switch to version 8 at some point in the future. The user can manually select the format with --lock-file-format [7|8]. (This is a setting that can be added to nix.conf.)
Version 8 lock files can depend on version 7 lock files, but not the other way around. (In principle, if desired, we could extend the version 7 lock algorithm to support version 8 lock files, generating a fully version 7 compatible lock file.)
Version 7 may be deprecated in the future. We will probably never drop support for evaluating version 7 flakes, but we may drop support for creating/updating version 7 lock files.
Step 1 (done, on branch lockfile-ng): refactor the current implementation so that everything outside of the version 7 implementation is independent of the lock file format. LockedFlake is now an abstract base class, and the version 7 implementation (the node graph, the lock algorithm, everything) is entirely private to lockfile-v7.cc; its header only exposes two free functions returning a std::unique_ptr<LockedFlake>. The version 8 implementation should mirror this (lockfile-v8.{hh,cc} with parseLockFileV8() and lockFlakeV8()).
The abstract interface that the version 8 implementation needs to provide (see src/libflake/include/nix/flake/flake.hh for the definitive version):
struct LockedFlake
{
Flake flake;
/* Map each input of the input denoted by `prefix` to either
std::nullopt (for a regular input) or the input attribute path
of the immediate target of a "follows" input. Backs the
`listFlakeInputs` primop. */
virtual std::map<FlakeId, std::optional<InputAttrPath>> getInputTargets(const InputAttrPath & prefix) const = 0;
struct InputInfo
{
FlakeRef lockedRef;
bool isFlake = true;
bool buildTime = false;
/* For relative path inputs: the flake (relative to the
top-level flake) that *declares* the path. */
std::optional<InputAttrPath> parentInputAttrPath;
};
/* Look up an input, resolving "follows" indirections. */
virtual std::optional<InputInfo> findInput(const InputAttrPath & path) const = 0;
/* Return the source path of an input (or of the top-level flake,
for the empty path), fetching it if necessary. The result is
backed by `EvalState::rootFS` (i.e. a store path, possibly a
virtual one with the input's accessor mounted on it if lazy
trees are enabled) and includes the flake's subdirectory.
Fetched inputs are cached, so this can be called concurrently
(e.g. by `nix flake prefetch-inputs`). */
virtual SourcePath getSourcePath(EvalState & state, const InputAttrPath & inputAttrPath) const = 0;
/* Walk all transitive inputs in depth-first order, reporting
either an InputInfo or the target of a "follows". The callback
controls recursion. Used by `nix flake metadata`, `nix flake
archive` and `nix flake prefetch-inputs`. */
virtual void visit(VisitCallback callback) const = 0;
/* Return an unlocked or non-final input, if any. Used by the
generic `getFingerprint()` and lock file writing. */
virtual std::optional<FlakeRef> isUnlocked(const fetchers::Settings & fetchSettings) const = 0;
/* Human-readable diff relative to an older LockedFlake. If the
old lock file has a different version, the implementation
diffs against an empty lock file. */
virtual std::string diff(const LockedFlake & oldLockFile) const = 0;
virtual nlohmann::json toJSON() const = 0;
};The version 7 header (lockfile-v7.hh) only declares:
/* Parse a version 5-7 lock file (`json` must be null if the lock
file doesn't exist). */
std::unique_ptr<LockedFlake> parseLockFileV7(
const fetchers::Settings & fetchSettings, Flake flake, const nlohmann::json & json, std::string_view path);
/* Compute a version 7 lock file, reusing entries from `oldLockFile`
(which must have been produced by `parseLockFileV7()`) where
possible. Does not write the new lock file. */
std::unique_ptr<LockedFlake> lockFlakeV7(
const Settings & settings,
EvalState & state,
const LockFlags & lockFlags,
Flake flake,
const LockedFlake & oldLockFile);The version 8 lock file representation (internal to lockfile-v8.cc):
struct LockFileV8
{
struct Lock
{
FlakeRef originalRef, lockedRef;
std::unique_ptr<LockFileV8> locks; // only present if the input has no lock file
// Note: no need for isFlake, buildTime, parentInputAttrPath from v7 - we get that info from each flake.nix.
/* The source path of this input, if it has been fetched
(cf. `LockedNode::sourcePath` in the v7 implementation);
set during locking and cached by `getSourcePath()`. */
mutable Sync<std::optional<SourcePath>> sourcePath;
};
std::map<NonEmptyInputAttrPath, Lock> locks;
};Done for version 7: the locking logic has been moved out of the free function lockFlake() into lockFlakeV7(). The free function now only does version-independent work, expressed in terms of the abstract LockedFlake interface: reading the old lock file into JSON, constructing a LockedFlake from it (via parseLockFileV7() — this is where we will dispatch on the version field of the JSON), change detection (by comparing the toJSON() serializations of the old and new locked flakes, which also does the right thing across version migrations), printing the diff (via diff()), and writing/committing the new lock file (via to_string() and isUnlocked()).
lockFlakeV8() will implement the new lock algorithm. It works as follows (pseudo-code):
std::unique_ptr<LockedFlake> lockFlakeV8(..., Flake flake, const LockedFlake & oldLockedFlake /* cast to LockedFlakeV8 internally */)
{
LockedFlakeV8 newLockedFlake;
auto createLock = [](const FlakeInput & input, Lock * oldLock) -> std::optional<Lock> {
auto ref = input.ref.value_or("flake:<inputname>");
if (the input is a follows) {
throw an error if input.ref is set
return std::nullopt;
} else if (the input is a relative path) {
check whether the path exists (relative to the flake.nix file)
fail if it does not have a flake.lock (in the future, for convenience, we may want to automatically create/update the relative lock file)
// otherwise do nothing; relative paths are followed at eval time
return std::nullopt;
} else if (!oldLock OR ref != oldLock->originalRef OR its inputAttrPath is in lockFlags.inputUpdates)
{
Lock lock;
lock.originalRef = ref;
lock.lockedRef = ... fetch ref ..;
if (the input does not have flake=false) {
check that the input has a flake.nix
if (the input does not have a lock file) {
lock.locks = recursively calculate a lock file, including it into the new node as `locks`
}
}
return lock;
} else {
return *oldLock;
}
};
apply any overrides from lockFlags.inputOverrides to the inputs (including recursively applying overrides to transitive inputs)
for (every input in flake.nix) {
auto lock = createLock(input, &oldLockedFlake.lockFile.locks[input.name] or nullptr if not found);
if (lock)
newLockedFlake.lockFile.locks.insert(input.name, std::move(lock));
for (every override in input.overrides, recursively) {
if (override.second.follows && !override.second.overrides.empty())
throw an error
if (override.second.ref)
newLockedFlake.lockFile.locks.insert(
overrideAttrPath,
createLock(override.second, &oldLockedFlake.lockFile.locks[overrideAttrPath] or nullptr if not found));
}
}
return std::make_unique<LockedFlakeV8>(std::move(newLockedFlake));
}Done, and better than originally planned: instead of separate call-flake-v{7,8}.nix files, there is a single call-flake.nix that is entirely independent of the lock file format — the version 8 implementation only needs to implement the LockedFlake interface, and evaluation works without further changes.
callFlake() passes three arguments to call-flake.nix: an ExternalValue wrapping the LockedFlake object, and two primops that take the locked flake and an input attribute path (a list of strings, where the empty list denotes the top-level flake):
-
listFlakeInputs lockedFlake inputAttrPath: returns the inputs of an input as an attrset mapping input names to either null (for a regular input) or the input attribute path of the target of a "follows" input (backed byLockedFlake::getInputTargets()). This only consults the lock data, so it doesn't fetch anything. -
fetchFlakeInput lockedFlake inputAttrPath: fetches an input (viaLockedFlake::getSourcePath()) and returns itssourceInfoattributes and flake subdirectory; or, for build-time inputs, the locked input attributes without fetching, so thatcall-flake.nixcan construct thebuiltin:fetch-treederivation.
These primops are registered with internal = true, so they're not exposed to the user under builtins; callFlake() gets their values from EvalState::internalPrimOps.
call-flake.nix lazily constructs a tree of inputs keyed by input attribute path. A regular input is constructed in place; a "follows" input is resolved by walking the edges of that tree from the top-level flake. This preserves evaluation sharing (every distinct input is constructed and evaluated only once, as with the node-keyed sharing of the old scheme), and chains of "follows" resolve automatically since the walk goes through edges that may themselves be "follows". Constructing the edges never fetches anything, preserving laziness.
Fingerprinting for the evaluation cache is largely unchanged. LockedFlake::getFingerprint() is now implemented generically in terms of the abstract interface (isUnlocked() and the toJSON() serialization), so nothing is needed for version 8.
We need to verify that if flake.lock cannot be written, the evaluation cache is disabled (or ensure that the contents of the unwritten flake.lock is hashed into the fingerprint).
In some cases this is a win (less spammy diffs), in other cases it is a loss (you don't see that a transitive input changed). We may want to add a nix flake subcommand to diff the lock files of two flakes.
In some cases, the new lock file format may require fetching more inputs than the old format. With the old format, inputs.nixpkgs.follows = "foo/nixpkgs" would not require fetching foo if nixpkgs is not used, but with the new format it does. However, this is rare in practice, since you typically only reuse transitive inputs of inputs that you actually use.