RFC: Nix backend #9089
Comments
I'm not sure I quite understood this. Do you mean temporally interleave such builds, or something like "build A->B->C where B is local and A and C are remote"? A major issue to me seems to be: how would we specify the system dependencies? We're going to need GHC, various C libraries and so on. Probably we want to get these from nixpkgs somehow, but the user will likely have an opinion on which nixpkgs, and it's not trivial to map from the system library names we use to nixpkgs derivations (see e.g. https://github.com/input-output-hk/haskell.nix/blob/master/lib/system-nixpkgs-map.nix in So maybe the user is going to need to tell us some of this stuff. The easiest way for them to do that would probably be for them to write some nix code that we can import to get things... but then we can't have a store-only implementation. An alternative model would be to do something more like Here's a sketch of an alternative plan. We can think of the process of building a local package a bit like this:
Instead of making it possible to replace the "build a remote package" part, we could make it possible to replace the "turn a Plan into a Package DB" part. Then we can out-source that to, say, The advantage of this is that it sidesteps the problems above. The Plan contains Cabal's view on what to build. It doesn't have to say how you get a particular GHC or how you get a system library with a particular name, that's the job of the tool that consumes the Plan. So the UI for this might look a bit like providing an |
|
cc @angerman |
Yes temporally. If a package in the dependency tree is local, then all its reverse dependencies (A in your example) will be local as well.
There is a simple solution to this problem: require the system dependencies to be already present in the store. This way cabal only controls Haskell dependencies, while system dependencies can be controlled externally. For example, when checking for the compiler, cabal can error out if GHC isn't in the store already, while if it is cabal can add the store path of its deriver to inputDrvs. Of course the only way of providing those dependencies right now is through the Nix language, but it isn't a requirement, and more importantly it can be done externally. Nix the language can provide the environment, cabal can build inside that environment using the nix store. This is the opposite of what haskell.nix does. It's more similar to an ordinary If system dependencies are stable there is also the sandbox-paths escape hatch of course... I'll add this to Challenges |
I'm not sure. Anything that allow us to throw away (large?) parts of haskell.nix is welcome.
That's what we currently do with our devx repo. E.g. and then using Works very well for interactive development. It does of course only work with the pre-set dependencies, and has no way of discovering dependencies needed by projects in any dynamic form. Having something that would do the same, but be able to declaratively figure out which dependencies the build plan needs would be nice. In any case a mapping from commonly referred to libraries to the names that nix calls them would be needed, but that can equally well live in nixpkgs itself. There is a bit of a chicken and egg issue if you try to use cabal's planning for this. To compute the plan, you may need some pkgconfig dependencies (and some package might just auto reconfigure if the pkg-config pkg isn't available; whether that's the intention of the actor or not). Or the build plan might fail because some build dependencies can't be found (I think something with postgresql is quite infamous for doing this 😬). Of course if we require people to have all of these in store (and environment) prior to running any cabal-install command, that will certainly work well! haskell.nix doesn't really try to solve any of those issues. It fundamentally only tries to solve this questions:
|
|
I have played with these ideas at lot @fgaz so if you want to start hacking some PoC togheter I am in. FWIW nix already support importing derivation graphs ( Others have already commented on the issue of the surrounding build environment. Re: multiple backends. This is the part I wish we could start with. As you must be aware:
I don't want to sound discouraging; the opposite! I encourage you, and anybody interested, to give it a go. In the end cabal-install is a library now so you can start your own cabal2nix solution. I assure you it will be warmly welcomed by all the other ones 😂 💜 |
|
Thanks for the feedback!
What I meant to say is this:
Yes! The separation between planning an building is there, but it gets complex fast.
This should be fine, nix derivation can be "package"-level, like the current cabal store paths, and everything can be source-based since we let nix do the caching.
Environment-specific information can be a problem and we'll have to use it carefully. IO is fine as long as the build plan doesn't depend on the store, and as far as I can tell it doesn't until the plan improvement phase. Ideally we'd skip plan improvement as it will be done by Nix.
If this is somehow made pluggable (backpack maybe?) other backends may actually be possible... |
I think a useful sub-goal is to make sure Cabal/cabal-install can always "be told" rather than "autodetect". An interesting thing to compare is @mpickering's multi-repl work, where As a bonus, the environment detection logic can be repurposed as testing after-the-fact logic. E.g., we can just "asssume" we'll have some C library eventually and As a final note, all this means that the right way to get stuff from the outside world (e.g. Nixpkgs) is not the build results but the derivations themselves. This allows maximal eager planning and lazy building. |
|
Also, I hope this code can co-exist with haskell.nix in a very healthy matter. The core work of separating planning and building within Cabal/cabal-install is good for both. The details of whether want to control everything and output our own derivations (this, an internal project), or know more about Nixpkgs / integration and output nix language expressions (haskell.nix, an external project) should be not too hard to abstract over for all the code that doesn't care. |
I'm not sure how this will work. What does it mean to say that "a system dependency is already present in the store"? To find something in the store you need to know the derivation that built it, which is the same thing as being able to build it! You have to say specifically what GHC derivation you want. There isn't a way to just say "get me a GHC from the store"! That's why I was saying we might need the user to tell us which specific GHC derivation they want. Otherwise there really isn't a way to find it, I think.
Yeah, I was really mostly bringing up |
|
Another relevant ticket is #6885 |
|
I am watching a github action build dozends of depenencies that I am sure have been built thousand times before by someone. So I'll fill the time waiting by saying that a nix backend would be great… :-) |
|
@nomeata there are a few caching solution for haskell and cabal on GitHub actions, are you using any of those? I am usually ok with just caching the store. That said, I had a look at implementing a "remote cache" for cabal in the style of bazel. This would be also very similar to how GHA cache works. It requires some rework of the store configuration (it's not just a path anymore) but it's doable if there are any takers. |
|
I was considering mentioning that that sounded less like a Nix backend than a Cachix backend. |
|
@angerman @michaelpj @Ericson2314 If the upshot here is that implementing this would allow big chunks of Haskell.nix functionality to move into cabal itself, I'd be very interested in lending a hand (I may be able to have Anduril fund this). |
Motivation
The current Nix-style store implementation works well but has a number of limitations, such as
Nix can solve many of those problems, but Nix as is requires total buy-in to be useful, which means using the Nix expression language and command line, and a cabal user may not be interested in those.
cabal-install 3 is uniquely positioned to transparently take advantage of the store part of Nix without having to sacrifice its own solver, cli, or ability to perform incremental builds on local packages.
Proposal
I propose to:
If Nix RFC 134 is implemented, this will be based on a stable interface independent from Nix-the-language.
Challenges
The main challenge is designing the interface of cabal-install backends. While the similarity of Nix and the Nix-style store helps, there are a few obstacles to consider, for example:
The interface has to be flexible enough to accommodate for these differences.
Future work
If this turns out to work well and Nix RFC92 is implemented, cabal-install can be used to generate build plans directly in nixpkgs without recursive Nix, closing the loop and demonstrating how system package managers and language-specific package managers can integrate even when dependency trees become complex.
Prior work
#3882
haskell.nix
The text was updated successfully, but these errors were encountered: