Skip to content

Support Nix depexts with opam env#5982

Merged
kit-ty-kate merged 12 commits intoocaml:masterfrom
RyanGibb:nixos-depexts
Apr 16, 2025
Merged

Support Nix depexts with opam env#5982
kit-ty-kate merged 12 commits intoocaml:masterfrom
RyanGibb:nixos-depexts

Conversation

@RyanGibb
Copy link
Copy Markdown
Contributor

@RyanGibb RyanGibb commented May 30, 2024

Nix doesn't install packages in the traditional sense -- instead it puts them in a store and makes them available through environment variables. I.e., nix-shell -p gcc will drop us into a shell with gcc in the $PATH.

We can set appropriate environment variables with Opam to make system dependencies (depexts) available with Nix.
Similar to how nix-shell works under the hood, we create a Nix derivation such as

{ pkgs ? import <nixpkgs> {} }:
with pkgs;
let
  packages = [ gmp ];
  inputs = with buildPackages; packages ++ [ pkg-config ];
in
stdenv.mkDerivation {
  name = "opam-nix-env";
  nativeBuildInputs = inputs;

  phases = [ "buildPhase" ];

  buildPhase = ''
vars=("NIX_CC" "NIX_CC_FLAGS" "NIX_CFLAGS_COMPILE" "NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu" "NIX_LDFLAGS" "PKG_CONFIG_PATH")
for var in "''${vars[@]}"; do
  escaped="$(echo "''${!var}" | sed -e 's/^$/@/' -e 's/ /\\ /g')"
  echo "$var    =       $escaped        Nix" >> $out
done
echo "PATH      =       $PATH   Nix"
  '';

  preferLocalBuild = true;
}

Which we can build to output a file with environment variables that make depexts available, in Opam's environment variable format. This file is a Nix store root, so it's dependencies won't be garbage collected by Nix until the file is removed.

This approach came from conversations with @dra27 and is distinct from previous approaches in that it supports providing development environment, which imperative installations with Nix don't #5332 (comment); and doesn't require the user to manage the environment outside of Opam, which would lead to a different workflow #5942.

Initial experiments were done using a package to set such environment variables https://github.com/RyanGibb/nix.opam.
However, in order to work with generic depexts (as opposed to just conf packages), to avoid cyclic dependencies (if conf packages depend on nix.opam but nix.opam depends on conf packages), and due to opam package sandboxing (nix derivation fetching would require opam sandboxing to be disabled or the nix store to be 'primed'), it's better implemented as a depext mechanism.

This has been tested and successfully creates an environment to provide packages conf-gmp and conf-libseccomp, as well as depexts directly.

@RyanGibb
Copy link
Copy Markdown
Contributor Author

Rebased on master

@RyanGibb RyanGibb force-pushed the nixos-depexts branch 3 times, most recently from 9e67415 to f348b9f Compare June 28, 2024 21:29
@RyanGibb
Copy link
Copy Markdown
Contributor Author

RyanGibb commented Jul 6, 2024

Looks like I'm running into NixOS/nix#10587 with the Docker image

@RyanGibb
Copy link
Copy Markdown
Contributor Author

Rebased on master -- and could do with another depext CI run if it needs re-approval!

@RyanGibb RyanGibb force-pushed the nixos-depexts branch 2 times, most recently from ff88713 to 2184b56 Compare July 18, 2024 15:49
@RyanGibb
Copy link
Copy Markdown
Contributor Author

Okay! The CI is failing only due to ocaml/opam-repository#26261 (I've got it working locally with my branch of opam-repository)

@kit-ty-kate kit-ty-kate added this to the 2.4.0~alpha1 milestone Sep 25, 2024
@RyanGibb
Copy link
Copy Markdown
Contributor Author

Anyone wanting to play around with this on a NixOS system can try:

$ nix shell github:RyanGibb/nixos/#legacyPackages.x86_64-linux.nixpkgs.opam
$ opam --version
2.3.0

Or add it to your own overlay with:

{
  nixpkgs.overlays = [
    (final: prev: {
      opam = prev.opam.overrideAttrs (_: {
       src = final.fetchurl {
          url = "http://ryan.freumh.org/software/opam-full-2.3.0-nixos-depexts.tar.gz";
          sha256 = "sha256-mRxxZtWFgQ8v1szVq5g5+qVqa+OffoG1aHzGUiMMvT0=";
        };
        version = "2.3.0";
      });
    })
  ];
}

This URL is just the https://github.com/RyanGibb/opam/tree/nixos-depexts-2.3.0 branch with vendored dependencies since I couldn't get them working in Nix,

@RyanGibb
Copy link
Copy Markdown
Contributor Author

RyanGibb commented Feb 3, 2025

David made an observation in today's Opam dev meeting that this PR will not remove depexts from the Nix derivation until the next install. This might effect reproducibility in some circumstances, but this isn't a huge deal as if Nix users want full reproducibility they can use opam2nix.

@kit-ty-kate kit-ty-kate requested a review from rjbou February 11, 2025 16:58
Copy link
Copy Markdown
Member

@kit-ty-kate kit-ty-kate left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

while tracking down NixOS/nixpkgs#388471 i tried this branch and it worked really well in a nixos docker container. I was able to build opam-publish which requires openssl and gmp without any issues at all, like magic.

Comment on lines +988 to +991
let stateless_install ?(env=OpamVariable.Map.empty) () =
match family ~env () with
| exception Failure _ -> false (* no depexts *)
| Nix -> false
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't the function name be reversed? Nix is a stateless install, as well as no depext, but everything else should be stateful. So either the boolean needs to be flipped or the function needs to be renamed to stateful_install

rjbou and others added 11 commits April 16, 2025 01:18
Co-authored-by: Ryan Gibb <ryan@freumh.org>
…entiate newly requested and required ones

Co-authored-by: Ryan Gibb <ryan@freumh.org>
Co-authored-by: Kate <kit-ty-kate@outlook.com>
…onised already installed systems dependencies with switch state

Co-authored-by: Raja Boujbel <raja.boujbel@ocamlpro.com>
Co-authored-by: Kate <kit-ty-kate@outlook.com>
… write internally in the switch

Co-authored-by: Raja Boujbel <raja.boujbel@ocamlpro.com>
At each install, a Nix derivation is created and stored in the switch
containing all system packages needed: newly "installed" ones and
already "required" ones.
An environment files is also generated ans used to complete 'opam env'.

Co-authored-by: Raja Boujbel <raja.boujbel@ocamlpro.com>
Co-authored-by: Kate <kit-ty-kate@outlook.com>
Co-authored-by: Kate <kit-ty-kate@outlook.com>
Co-authored-by: Raja Boujbel <raja.boujbel@ocamlpro.com>
@rjbou rjbou requested a review from kit-ty-kate April 15, 2025 23:19
@kit-ty-kate
Copy link
Copy Markdown
Member

Thanks a lot!

@kit-ty-kate kit-ty-kate merged commit 1e24003 into ocaml:master Apr 16, 2025
55 checks passed
@RyanGibb
Copy link
Copy Markdown
Contributor Author

Wow, awesome that this got merged at long last! Thank you both for all the work you put in :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants