Frontear/benchmarks.md

Benchmarking Nix (and its derivatives)

The configuration used for testing was my own, and can be found at Frontear/dotfiles. To test in much the same way, the following command was used to run 10 isolated benchmarks:

# This was run within the root of my repo.
# To obtain working versions of the other nix* derivatives, use `nix shell` so that there is no overhead in processing the store.

for i in $(seq 1 10); do
  time nix eval '.#nixosConfigurations.LAPTOP-3DT4F02.config.system.build.toplevel.drvPath' --no-eval-cache
done

Results

Results have been compiled into a nice table. Raw data can still be viewed further down.

Nix Version	Average Time
Nix 2.18.4	35.682s
Nix 2.24.0	33.213s
Nix Super 2.22.0	32.779s
Lix 2.91.0	29.408s

Raw Data

Below you can find the raw data from the command run above, for each of the nix derivatives.

With nix (Nix) 2.18.4:

nix eval   37.74s user 4.67s system 89% cpu 47.416 total
nix eval   35.75s user 4.50s system 89% cpu 45.119 total
nix eval   35.32s user 4.51s system 89% cpu 44.600 total
nix eval   35.37s user 4.46s system 89% cpu 44.319 total
nix eval   35.41s user 4.51s system 89% cpu 44.474 total
nix eval   35.47s user 4.38s system 90% cpu 44.031 total
nix eval   35.43s user 4.47s system 89% cpu 44.513 total
nix eval   35.36s user 4.47s system 89% cpu 44.443 total
nix eval   35.46s user 4.32s system 90% cpu 44.196 total
nix eval   35.51s user 4.39s system 89% cpu 44.338 total

With nix (Nix) 2.24.0pre20240711_0363dbf:

nix eval   33.84s user 4.61s system 89% cpu 42.840 total
nix eval   33.21s user 4.51s system 90% cpu 41.885 total
nix eval   33.35s user 4.37s system 90% cpu 41.796 total
nix eval   33.13s user 4.49s system 90% cpu 41.683 total
nix eval   33.13s user 4.44s system 90% cpu 41.652 total
nix eval   33.00s user 4.50s system 90% cpu 41.579 total
nix eval   33.09s user 4.50s system 90% cpu 41.588 total
nix eval   33.14s user 4.44s system 90% cpu 41.714 total
nix eval   33.19s user 4.37s system 90% cpu 41.636 total
nix eval   33.05s user 4.49s system 90% cpu 41.630 total

With nix (Nix Super) 2.22.0pre20240705_1b14f2e:

nix eval   33.52s user 4.42s system 89% cpu 42.318 total
nix eval   32.79s user 4.39s system 89% cpu 41.381 total
nix eval   32.78s user 4.50s system 90% cpu 41.266 total
nix eval   32.56s user 4.40s system 90% cpu 40.847 total
nix eval   32.72s user 4.45s system 90% cpu 41.063 total
nix eval   32.74s user 4.33s system 90% cpu 40.958 total
nix eval   32.84s user 4.46s system 90% cpu 41.307 total
nix eval   32.69s user 4.27s system 90% cpu 40.844 total
nix eval   32.62s user 4.36s system 90% cpu 40.863 total
nix eval   32.53s user 4.39s system 90% cpu 40.864 total

With nix (Lix, like Nix) 2.91.0-devpre20240711_a5d431a:

nix eval   29.68s user 4.31s system 89% cpu 37.961 total
nix eval   29.20s user 4.37s system 88% cpu 37.727 total
nix eval   29.16s user 4.49s system 88% cpu 37.838 total
nix eval   29.67s user 4.38s system 89% cpu 38.070 total
nix eval   29.33s user 4.41s system 88% cpu 37.920 total
nix eval   29.36s user 4.39s system 88% cpu 37.942 total
nix eval   29.32s user 4.35s system 89% cpu 37.730 total
nix eval   29.21s user 4.42s system 89% cpu 37.746 total
nix eval   29.13s user 4.50s system 89% cpu 37.771 total
nix eval   30.02s user 4.49s system 89% cpu 38.478 total

	import re

	EXPR = r"nix eval\s+(.+)s\s+user"

	# Copy raw data here
	DATA = """
	"""

	if __name__ == "__main__":
	matches = re.finditer(EXPR, DATA, re.MULTILINE)
	average = 0.0

	for i, e in enumerate(matches, start=1):
	average += float(e.group(1))

	print(average / 10.0)