matrix - Nix NixOS - Chat Export - 2024-10-30T04-12-17.208Z.txt

Tue, Oct 29, 2024, 16:26:26 - kait: hello, i need a slightly customised LLVM for my work and the llvmPackages Nix code is a bit confusing. what are the suggested ways to override LLVM with cmake flags (e.g. LLVM\_TARGETS\_TO\_BUILD) and to point it to a particular commit of the llvm repo (i.e. override the src)?
Tue, Oct 29, 2024, 16:27:53 - kait: i would like to do this in the "correct" way to get a consistent llvmPackages
Tue, Oct 29, 2024, 16:29:46 - ElvishJerricco: kait: IIRC the llvm nix code is pretty complicated, and I doubt it's designed with that use case in mind (Tristan Ross or emily can probably correct me if I'm wrong on that). It might be easiest to start with editing the expressions in nixpkgs directly to get started
Tue, Oct 29, 2024, 16:30:41 - emily: I think it does actually let you add new versions
Tue, Oct 29, 2024, 16:30:56 - emily: if you look at `llvmVersions` and how it's used in `pkgs/development/compilers/llvm/default.nix `
Tue, Oct 29, 2024, 16:30:58 - ElvishJerricco: oh, that must be new since the last time I tried to do that... which was a long time ago I guess
Tue, Oct 29, 2024, 16:32:04 - emily: for overriding flags, I have no idea… if you just need a specific target you might be able to get away with just using Nixpkgs' cross machinery and setting `targetPlatform`
Tue, Oct 29, 2024, 16:32:34 - emily: I think we just build, all the targets though?
Tue, Oct 29, 2024, 16:32:43 - emily: so if all the targets is cool for you then you can maybe just kinda let taht happen
Tue, Oct 29, 2024, 16:33:14 - kait: <@emilazy:matrix.org "for overriding flags, I have no ..."> maybe a bad example, i actually need to set LLVM_ENABLE_EH and LLVM_ENABLE_RTTI :)
Tue, Oct 29, 2024, 16:33:21 - emily: right
Tue, Oct 29, 2024, 16:33:23 - ElvishJerricco: (when wasm was still new, we didn't build that target, and I think that's because it wasn't in LLVM's default for "lots of targets please")
Tue, Oct 29, 2024, 16:33:28 - emily: is that on all the derivations or just LLVM proper?
Tue, Oct 29, 2024, 16:34:02 - kait: <@emilazy:matrix.org "if you look at `llvmVersions` an..."> yea it looks well-factored in the code, i'm just not sure on how to reach into that from outside (i would like to override without changing nixpkgs if possible)
Tue, Oct 29, 2024, 16:34:23 - emily: you should be able to `.overrideAttrs (prevAttrs: { cmakeFlags = prevAttrs.cmakeFlags or [ ] ++ [ blah ]; })` but of course that's just one derivation, idk how you reconstruct a consistent package set from that
Tue, Oct 29, 2024, 16:34:26 - kait: <@emilazy:matrix.org "is that on all the derivations o..."> just LLVM itself, and i'll use this llvm version only where i need it
Tue, Oct 29, 2024, 16:34:32 - emily: <@rina/:matrix.org "yea it looks well-factored in th..."> it's a parameter
Tue, Oct 29, 2024, 16:35:15 - emily: `llvmPackagesSet` is private I guess. but you can `callPackage` it
Tue, Oct 29, 2024, 16:35:40 - emily: `(callPackages (pkgs.path + "/…/llvm") { llvmVersions = … })."myVersion"`
Tue, Oct 29, 2024, 16:35:48 - kait: yeah, i can't find how to get to that point from the nixpkgs package set
Tue, Oct 29, 2024, 16:36:21 - emily: for what it's worth though it may genuinely be easier to just patch Nixpkgs. but I'm pretty sure you could construct something out-of-tree that works
Tue, Oct 29, 2024, 16:36:53 - emily: I'm not actually an expert here so probably someone on the LLVM team could give you something that works perfectly in 30 seconds
Tue, Oct 29, 2024, 16:37:14 - kait: patching nixpkgs is very painful  with updates and everything ;-;
Tue, Oct 29, 2024, 16:37:29 - kait: (i wish it was easier)
Tue, Oct 29, 2024, 16:37:40 - kait: <@emilazy:matrix.org "`(callPackages (pkgs.path + "/…/..."> this sounds okay
Tue, Oct 29, 2024, 16:38:14 - emily: it's not so bad if you rebase frequently. but yeah it requires some maintenance work
Tue, Oct 29, 2024, 16:38:18 - emily: (so does overriding derivations though, but)
Tue, Oct 29, 2024, 16:42:37 - emily: you can also just like, copy the entire LLVM derivation directory
Tue, Oct 29, 2024, 16:42:43 - emily: and do violence to it
Tue, Oct 29, 2024, 16:43:06 - emily: oh hey
Tue, Oct 29, 2024, 16:43:08 - emily: `, devExtraCmakeFlags ? []`
Tue, Oct 29, 2024, 16:43:10 - kait: one could ;-;
Tue, Oct 29, 2024, 16:44:26 - kait: <@emilazy:matrix.org "`, devExtraCmakeFlags ? []`"> fascinating
Tue, Oct 29, 2024, 16:45:55 - kait: this is very new :D
Tue, Oct 29, 2024, 16:46:06 - kait: but the pr has good discussion about my exact use case
Tue, Oct 29, 2024, 16:46:12 - kait: https://github.com/NixOS/nixpkgs/pull/342040
Tue, Oct 29, 2024, 16:46:21 - emily: ok I think you just want to `callPackages (pkgs.path + "/development/compilers/llvm/common/default.nix")`, pass `gitRelease` and `devExtraCmakeFlags`
Tue, Oct 29, 2024, 16:46:45 - emily: or yeah maybe you can even just `llvmPackages.override` with a new version and flags
Tue, Oct 29, 2024, 16:47:01 - emily: my condolences for your upcoming build times
Tue, Oct 29, 2024, 16:47:15 - kait: i'm used to it :")
Tue, Oct 29, 2024, 16:47:34 - kait: llvm is /only/ a half-hour build
Tue, Oct 29, 2024, 16:47:46 - kait: but it is really painful while working out the nix code aha
Tue, Oct 29, 2024, 16:48:47 - emily: <@rina/:matrix.org "llvm is /only/ a half-hour build"> maybe on your machine
Tue, Oct 29, 2024, 16:49:00 - emily: probably doesn't help when you run like 5 of them at once tbh
Tue, Oct 29, 2024, 16:49:30 - emily: just be glad you don't have to bootstrap an entire stdenv
Tue, Oct 29, 2024, 16:50:34 - ElvishJerricco: hey my pastime is rebuilding a hundred times more things than should be necessary on traditional distros
Tue, Oct 29, 2024, 16:51:05 - ˈt͡sɛːzaɐ̯: Traditional distros involve rebuilding?
Tue, Oct 29, 2024, 16:51:59 - ElvishJerricco: *something something* ca derivations *something something* it doesn't work that way *something something*
Tue, Oct 29, 2024, 16:52:47 - emily: it works that way if (links the me and amjoseph thread again)
Tue, Oct 29, 2024, 16:53:14 - ElvishJerricco: I think I forgot to read that whole thread :P
Tue, Oct 29, 2024, 16:53:22 - ElvishJerricco: (if you don't mind sharing it again)
Tue, Oct 29, 2024, 16:53:25 - emily: sucks to be you. missing out on timeless wisdom
Tue, Oct 29, 2024, 16:53:52 - emily: https://discourse.nixos.org/t/why-does-the-nixos-infrastructure-have-to-be-hosted-in-a-centralized-way/46789/32?u=emily onward
Tue, Oct 29, 2024, 16:53:53 - ElvishJerricco: oh damn, my wisdom
Tue, Oct 29, 2024, 16:56:55 - emily: fwiw it doesn't help as much with stdenv stuff (at least my version)
Tue, Oct 29, 2024, 16:57:01 - emily: but something like OpenSSL it does
Tue, Oct 29, 2024, 16:58:32 - kait: does the cache do any deduplication of identical or similar files?
Tue, Oct 29, 2024, 16:58:40 - emily: though the ninja validations thing for tests would speed up stdenv build meaningfully I think
Tue, Oct 29, 2024, 16:58:57 - ElvishJerricco: `auto-optimise-store` deduplicates files when builds are done
Tue, Oct 29, 2024, 16:59:00 - emily: <@rina/:matrix.org "does the cache do any deduplicat..."> I think it's NARs in S3, so at most deduplication would be on the entire NAR level I think.
Tue, Oct 29, 2024, 16:59:05 - ElvishJerricco: but the cache can't know what file you're going to build
Tue, Oct 29, 2024, 16:59:14 - emily: oh, or do you mean the store?
Tue, Oct 29, 2024, 16:59:26 - emily: if you mean the store then there's `auto-optimise-store` (do not use on Darwin) and `nix store optimise`
Tue, Oct 29, 2024, 16:59:46 - ElvishJerricco: > do not use on Darwin
Tue, Oct 29, 2024, 16:59:48 - ElvishJerricco: wut
Tue, Oct 29, 2024, 17:00:22 - emily: there is a race condition or whatever and it reliably fails builds and sometimes maybe corrupts the store
Tue, Oct 29, 2024, 17:00:33 - emily: nix-darwin has `nix.optimise` to run a scheduled thing that does not have these issues
Tue, Oct 29, 2024, 17:00:39 - ElvishJerricco: I will... disable that, then...
Tue, Oct 29, 2024, 17:00:46 - ElvishJerricco: is this fixable or is it fundamental?
Tue, Oct 29, 2024, 17:00:57 - emily: it's some stupid bug probably
Tue, Oct 29, 2024, 17:01:07 - emily: that nobody is fixing because Nix
Tue, Oct 29, 2024, 17:01:17 - kait: i did mean the binary cache, but this is good to know
Tue, Oct 29, 2024, 17:01:52 - ElvishJerricco: kait: yea the binary cache only knows what exact nar you want. The implementation of the cache can be smart about deduplication (see: attic)
Tue, Oct 29, 2024, 17:02:05 - emily: I think the answer to "does the binary cache do this smart thing that would bring costs down" is "no" for all such things
Tue, Oct 29, 2024, 17:02:13 - emily: but hopefully that will change in the future
Tue, Oct 29, 2024, 17:02:27 - ElvishJerricco: <@emilazy:matrix.org "I think the answer to "does the ..."> *(((((attic)))))*
Tue, Oct 29, 2024, 17:02:30 - kait: it would probably need very speial code on the binary cache side
Tue, Oct 29, 2024, 17:02:38 - emily: I'm talking about cache.nixos.org
Tue, Oct 29, 2024, 17:02:43 - ElvishJerricco: ah yea
Tue, Oct 29, 2024, 17:03:14 - emily: <@emilazy:matrix.org "but hopefully that will change i..."> (because also all such things have been thought of by now given the Situation with the cache)
Tue, Oct 29, 2024, 17:03:31 - emily: (but not having to pay for years did not exactly set incentives to do those smart things)
Tue, Oct 29, 2024, 17:03:49 - Leonard Menzel joined the room
Tue, Oct 29, 2024, 17:03:52 - ElvishJerricco: wasn't attic considered for solving the cache.nixos.org problems, but the issue is mainly the orders of magnitude larger compute infrastructure it would require?
Tue, Oct 29, 2024, 17:04:03 - emily: dunno. probably
Tue, Oct 29, 2024, 17:04:09 - emily: by the "all such things have been considered" rubric
Tue, Oct 29, 2024, 17:04:16 - ElvishJerricco: yea
Tue, Oct 29, 2024, 17:05:02 - kait: these optimisation problems sound super interesting, but also really difficult
Tue, Oct 29, 2024, 17:05:07 - kait: i hope they get worked on eventually
Tue, Oct 29, 2024, 17:05:35 - ElvishJerricco: I think nixos would have to be a few orders of magnitude bigger to really solve them
Tue, Oct 29, 2024, 17:05:51 - ElvishJerricco: because the issue is worldwide infrastructure
Tue, Oct 29, 2024, 17:05:56 - ElvishJerricco: we can optimise a lot already
Tue, Oct 29, 2024, 17:06:11 - ElvishJerricco: but making it huge like it needs to be is hard unless you use commodity stuff like S3
Tue, Oct 29, 2024, 17:06:43 - ElvishJerricco: and the commodity stuff is what prevents optimisation
Tue, Oct 29, 2024, 17:09:05 - emily: there's plenty of smart stuff you can do and some of it has been worked on
Tue, Oct 29, 2024, 17:09:13 - emily: there was just no pressure to do it before
Tue, Oct 29, 2024, 17:09:26 - emily: and less pressure again now that AWS is footing the bill
Tue, Oct 29, 2024, 17:09:57 - ElvishJerricco: you don't think a lot of the needs are application specific?
Tue, Oct 29, 2024, 17:10:08 - ElvishJerricco: because that's what requires custom solutions, which are hard to scale
Tue, Oct 29, 2024, 17:10:46 - emily: not sure what you mean. I mean that there are viable ideas for reducing storage costs for the NixOS cache
Tue, Oct 29, 2024, 17:11:16 - ElvishJerricco: yea but like dedup for instance requires an application that knows how to do the dedup onto the storage and undo it for distribution
Tue, Oct 29, 2024, 17:11:36 - emily: I don't think some amount of deduplication is fiendishly difficult. we already have a CDN in front of things
Tue, Oct 29, 2024, 17:11:54 - emily: it was just enough work and no reason to do it that it never happened before
Tue, Oct 29, 2024, 17:11:55 - ElvishJerricco: well IIRC attic does smarter NAR-specific dedup