WIP: Nix package. See notes. by hyperrealgopher · Pull Request #27 · someodd/burrow

hyperrealgopher · 2021-06-16T02:44:10Z

Begin the Nix packaging process.

I am new to Nix packaging.

I was having problems with getting table-layouts to work so I removed
the dependency entirely. I recall there being issues with this
dependency in the past and I was using it to justify text, but I have my
own function for doing that. So to temporarily save me the headache, in
order to get the package to build, I have removed that dependency.

I was following this guide to get my first release working:
https://maybevoid.com/posts/2019-01-27-getting-started-haskell-nix.html

When I change burrow.cabal I run: nix-shell --pure -p cabal2nix --run "cabal2nix ." > default.nix. When I want to build the release I run
nix-build release.nix.

Also I had to use the envvar NIXPKGS_ALLOW_BROKEN=1 to allow errata
to not halt the process.

hyperrealgopher · 2021-06-16T02:49:43Z

I don't even understand it all fully yet! Is the idea that I now put my ./result/bin/burrow up as a tagged github release for people to download? And that nix just made the compilation/build of the release reproducible?

hyperrealgopher · 2021-06-16T03:01:39Z

~/Projects/burrow feature/nix-packaging
❯ ./result/bin/burrow --source example-gopherhole/ --destination built/ --spacecookie
burrow: data/fonts/basicthorpe.bmf: hGetContents: invalid argument (invalid byte sequence)

hyperrealgopher · 2021-06-16T03:10:35Z

Like the article suggests I should make a makefile.

hyperrealgopher · 2021-06-16T05:41:51Z

Maybe I'll add table-layout back as a dependency if I can figure out why ir results in an error about QuickCheck.

Begin the Nix packaging process. I am new to Nix packaging. I was having problems with getting table-layouts to work so I removed the dependency entirely. I recall there being issues with this dependency in the past and I was using it to justify text, but I have my own function for doing that. So to temporarily save me the headache, in order to get the package to build, I have removed that dependency. I was following this guide to get my first release working: https://maybevoid.com/posts/2019-01-27-getting-started-haskell-nix.html When I change `burrow.cabal` I run: `nix-shell --pure -p cabal2nix --run "cabal2nix ." > default.nix`. When I want to build the release I run `nix-build release.nix`. Also I had to use the envvar `NIXPKGS_ALLOW_BROKEN=1` to allow `errata` to not halt the process.

hyperrealgopher · 2021-06-16T06:06:58Z

This helped me solve the UTF8 problem: NixOS/nixpkgs#64603

hyperrealgopher · 2021-06-16T06:31:37Z

Maybe I should figure this out: https://github.com/nh2/static-haskell-nix

sternenseemann · 2021-06-16T09:39:52Z

~/Projects/burrow feature/nix-packaging
❯ ./result/bin/burrow --source example-gopherhole/ --destination built/ --spacecookie
burrow: data/fonts/basicthorpe.bmf: hGetContents: invalid argument (invalid byte sequence)

See the Cabal manual on how to acces data files in a portable way (that will also work with nix regardless of $PWD).

sternenseemann · 2021-06-16T09:42:31Z

I don't even understand it all fully yet! Is the idea that I now put my ./result/bin/burrow up as a tagged github release for people to download? And that nix just made the compilation/build of the release reproducible?

Not with standard nixpkgs as the binaries are not fully linked statically, i. e. it'd still expect to find libc, gmp and a few system dependencies installed normally in /nix/store which makes it impractical for distributing.

Using pkgsStatic may save you the trouble of switching to nh2's project, however it currently still has trouble with TemplateHaskell (so it may be a problem if a dependency of yours uses that).

You can still use standard nixpkgs haskellPackages for CI and development environments though.

sternenseemann · 2021-06-16T09:43:59Z

+  inherit (nixpkgs) pkgs;
+  inherit (pkgs) haskellPackages;


nixpkgs.pkgs == nixpkgs, it just recursively re-exposes itself.

Suggested change

inherit (nixpkgs) pkgs;

inherit (pkgs) haskellPackages;

inherit (nixpkgs) haskellPackages;

So inherit just gives us access to a bunch of attributes belonging to some object nixpkgs here? And the added line you suggested will put those attributes into haskellPackages so you access them via haskellpackages.x where x is an attribute?

Oh and also nixpkgs are like a set of specific "stores" which specify some specific set of builds for various software/dependencies? Kind of like which snapshot you want in Stack? And instead of using nixpkgs I can use pkgsStatic if I'm not using TemplateHaskell and that'll make it so these builds are suitable for distribution (being statically-linked) which is pretty much my goal with Nix, here?

sternenseemann · 2021-06-16T09:44:43Z

@@ -0,0 +1,4 @@
+let
+  pkgs = import <nixpkgs> { };


Consider doing this via a function argument as well, like in shell.nix — this allows overriding the version of nixpkgs used from the outside.

sternenseemann · 2021-06-16T09:45:46Z

+external-shell:
+	nix-shell external.nix


external.nix doesn't exist

sternenseemann · 2021-06-16T09:46:46Z

Maybe I'll add table-layout back as a dependency if I can figure out why ir results in an error about QuickCheck.

What is the error? Also why was the test suite removed?

As pointed out here [1], errata has started working on its own again. [1]: someodd/burrow#27 (comment)

hyperrealgopher · 2021-06-16T19:29:43Z

Thank you so much for taking the time to review this experiment/me learning about Nix!

~/Projects/burrow feature/nix-packaging
❯ ./result/bin/burrow --source example-gopherhole/ --destination built/ --spacecookie
burrow: data/fonts/basicthorpe.bmf: hGetContents: invalid argument (invalid byte sequence)
See the Cabal manual on how to acces data files in a portable way (that will also work with nix regardless of $PWD).

Yeah I did this with waffle. The error in question is actually because of some weird envvar issue I fixed by setting another envvar. I however, didn't want to use it in this case because I actually want the data directory to be relative to the user's project folder/gopherhole they're building (it has the config too). I should probably include those fonts as a default and include the config as a default though, now that I think about it, in case they're not found in the current path! Thanks for getting me to think about that!

I don't even understand it all fully yet! Is the idea that I now put my ./result/bin/burrow up as a tagged github release for people to download? And that nix just made the compilation/build of the release reproducible?

Not with standard nixpkgs as the binaries are not fully linked statically, i. e. it'd still expect to find libc, gmp and a few system dependencies installed normally in /nix/store which makes it impractical for distributing.

Using pkgsStatic may save you the trouble of switching to nh2's project, however it currently still has trouble with TemplateHaskell (so it may be a problem if a dependency of yours uses that).

You can still use standard nixpkgs haskellPackages for CI and development environments though.

I see! Thank you for that information! I am looking through Tweag's guide on statically linked Haskell binaries with Bazel, but I have to admit, at the moment it's all Greek to me.

Maybe I'll add table-layout back as a dependency if I can figure out why ir results in an error about QuickCheck.

What is the error? Also why was the test suite removed?

I was getting an error about QuickCheck:

❯ nix-build release.nix                                          
warning: unable to download 'https://cache.nixos.org/9akxxj4c6n0bnplk831f9mqcbnrd70mh.narinfo': Couldn't resolve host name (6); retrying in 345 ms
these derivations will be built:
  /nix/store/fwfsrvgpmyzb91ylhj3wssyd6c7vl2m5-table-layout-0.9.1.0.drv
  /nix/store/dvhjcdk0frcxycdyyzdmb7qwb14k9388-burrow-0.1.0.0.drv
building '/nix/store/fwfsrvgpmyzb91ylhj3wssyd6c7vl2m5-table-layout-0.9.1.0.drv'...
setupCompilerEnvironmentPhase
Build with /nix/store/qam66cqpir6n9260xzs19mbdx6kd6md0-ghc-8.10.4.
unpacking sources
unpacking source archive /nix/store/g7pxbb982x72r3xhp7lx1q87cj3bhqjq-table-layout-0.9.1.0.tar.gz
source root is table-layout-0.9.1.0
setting SOURCE_DATE_EPOCH to timestamp 1000000000 of file table-layout-0.9.1.0/test-suite/TestSpec.hs
patching sources
compileBuildDriverPhase
setupCompileFlags: -package-db=/tmp/nix-build-table-layout-0.9.1.0.drv-0/setup-package.conf.d -j8 +RTS -A64M -RTS -threaded -rtsopts
[1 of 1] Compiling Main             ( Setup.hs, /tmp/nix-build-table-layout-0.9.1.0.drv-0/Main.o )
Linking Setup ...
configuring
configureFlags: --verbose --prefix=/nix/store/aamy990cibbxq7anywgkg2yabydfrpiv-table-layout-0.9.1.0 --libdir=$prefix/lib/$compiler --libsubdir=$abi/$libname --docdir=/nix/store/nd426k51szgxcdhc0bdniqxr1kqmg914-table-layout-0.9.1.0-doc/share/doc/table-layout-0.9.1.0 --with-gcc=gcc --package-db=/tmp/nix-build-table-layout-0.9.1.0.drv-0/package.conf.d --ghc-options=-j8 +RTS -A64M -RTS --disable-split-objs --enable-library-profiling --profiling-detail=exported-functions --disable-profiling --enable-shared --disable-coverage --enable-static --disable-executable-dynamic --enable-tests --disable-benchmarks --enable-library-vanilla --disable-library-for-ghci --ghc-option=-split-sections --extra-lib-dirs=/nix/store/9m4hy7cy70w6v2rqjmhvd7ympqkj6yxk-ncurses-6.2/lib --extra-lib-dirs=/nix/store/0d71ygfwbmy1xjlbj1v027dfmy9cqavy-libffi-3.3/lib --extra-lib-dirs=/nix/store/lqwgi0in94ll5fwsbh8ligvn8l0vqn7v-gmp-6.2.1/lib
Using Parsec parser
Configuring table-layout-0.9.1.0...
CallStack (from HasCallStack):
  $, called at libraries/Cabal/Cabal/Distribution/Simple/Configure.hs:1024:20 in Cabal-3.2.1.0:Distribution.Simple.Configure
  configureFinalizedPackage, called at libraries/Cabal/Cabal/Distribution/Simple/Configure.hs:477:12 in Cabal-3.2.1.0:Distribution.Simple.Configure
  configure, called at libraries/Cabal/Cabal/Distribution/Simple.hs:625:20 in Cabal-3.2.1.0:Distribution.Simple
  confHook, called at libraries/Cabal/Cabal/Distribution/Simple/UserHooks.hs:65:5 in Cabal-3.2.1.0:Distribution.Simple.UserHooks
  configureAction, called at libraries/Cabal/Cabal/Distribution/Simple.hs:180:19 in Cabal-3.2.1.0:Distribution.Simple
  defaultMainHelper, called at libraries/Cabal/Cabal/Distribution/Simple.hs:116:27 in Cabal-3.2.1.0:Distribution.Simple
  defaultMain, called at Setup.hs:2:8 in main:Main
Setup: Encountered missing or private dependencies:
QuickCheck >=2.8 && <2.14

builder for '/nix/store/fwfsrvgpmyzb91ylhj3wssyd6c7vl2m5-table-layout-0.9.1.0.drv' failed with exit code 1
cannot build derivation '/nix/store/dvhjcdk0frcxycdyyzdmb7qwb14k9388-burrow-0.1.0.0.drv': 1 dependencies couldn't be built
error: build of '/nix/store/dvhjcdk0frcxycdyyzdmb7qwb14k9388-burrow-0.1.0.0.drv' failed

The test sutie was removed because I initially thought it had to do with the QuickCheck error, but I should add it back.

I get this error when doing `make run`: Ready component graph: definite QuickCheck-2.14.2-1rgy6m2txlL6P9xFMqTWv8 depends base-4.14.1.0 depends containers-0.6.2.1 depends deepseq-1.4.4.0 depends random-1.2.0-6zIdUIrmHInBiqbyqpDYsu depends splitmix-0.1.0.3-1YkuweqDYq8DBxer1Np43T depends template-haskell-2.16.0.0 depends transformers-0.5.6.2 Using Cabal-3.2.1.0 compiled by ghc-8.10 Using compiler: ghc-8.10.4 Using install prefix: /nix/store/9vinyvwfyy1zq42di9fnbh0g0h2hzdfd-QuickCheck-2.14.2-x86_64-unknown-linux-musl Executables installed in: /nix/store/9vinyvwfyy1zq42di9fnbh0g0h2hzdfd-QuickCheck-2.14.2-x86_64-unknown-linux-musl/bin Libraries installed in: /nix/store/9vinyvwfyy1zq42di9fnbh0g0h2hzdfd-QuickCheck-2.14.2-x86_64-unknown-linux-musl/lib/ghc-8.10.4/x86_64-linux-ghc-8.10.4/QuickCheck-2.14.2-1rgy6m2txlL6P9xFMqTWv8 Dynamic Libraries installed in: /nix/store/9vinyvwfyy1zq42di9fnbh0g0h2hzdfd-QuickCheck-2.14.2-x86_64-unknown-linux-musl/lib/ghc-8.10.4/x86_64-linux-ghc-8.10.4 Private executables installed in: /nix/store/9vinyvwfyy1zq42di9fnbh0g0h2hzdfd-QuickCheck-2.14.2-x86_64-unknown-linux-musl/libexec/x86_64-linux-ghc-8.10.4/QuickCheck-2.14.2 Data files installed in: /nix/store/9vinyvwfyy1zq42di9fnbh0g0h2hzdfd-QuickCheck-2.14.2-x86_64-unknown-linux-musl/share/x86_64-linux-ghc-8.10.4/QuickCheck-2.14.2 Documentation installed in: /nix/store/vqp5k9ix2zxgmx3b0b6dq4jfkz906k9i-QuickCheck-2.14.2-x86_64-unknown-linux-musl-doc/share/doc/QuickCheck-2.14.2 Configuration files installed in: /nix/store/9vinyvwfyy1zq42di9fnbh0g0h2hzdfd-QuickCheck-2.14.2-x86_64-unknown-linux-musl/etc No alex found Using ar given by user at: /nix/store/041mz911y2gc34ga7ppzj69p2w4qdvv3-x86_64-unknown-linux-musl-binutils-2.35.1/bin/x86_64-unknown-linux-musl-ar No c2hs found No cpphs found No doctest found Using gcc version 10.3.0 given by user at: /nix/store/kkazgn30krwr8y4b4fcz3psnj021wimb-x86_64-unknown-linux-musl-stage-final-gcc-wrapper-10.3.0/bin/x86_64-unknown-linux-musl-cc Using ghc version 8.10.4 given by user at: /nix/store/df3h3klc3rjqi6xb0464g39130vbf17s-x86_64-unknown-linux-musl-ghc-8.10.4/bin/x86_64-unknown-linux-musl-ghc Using ghc-pkg version 8.10.4 given by user at: /nix/store/df3h3klc3rjqi6xb0464g39130vbf17s-x86_64-unknown-linux-musl-ghc-8.10.4/bin/x86_64-unknown-linux-musl-ghc-pkg No ghcjs found No ghcjs-pkg found No greencard found Using haddock version 2.24.0 found on system at: /nix/store/qam66cqpir6n9260xzs19mbdx6kd6md0-ghc-8.10.4/bin/haddock No happy found Using haskell-suite found on system at: haskell-suite-dummy-location Using haskell-suite-pkg found on system at: haskell-suite-pkg-dummy-location No hmake found Using hpc version 0.68 found on system at: /nix/store/qam66cqpir6n9260xzs19mbdx6kd6md0-ghc-8.10.4/bin/hpc Using hsc2hs version 0.68.7 given by user at: /nix/store/df3h3klc3rjqi6xb0464g39130vbf17s-x86_64-unknown-linux-musl-ghc-8.10.4/bin/x86_64-unknown-linux-musl-hsc2hs Using hscolour version 1.24 found on system at: /nix/store/30ah1yzvkbv50qghyx5kv0wcx6lqi6bd-hscolour-1.24.4/bin/HsColour No jhc found Using ld given by user at: /nix/store/kkazgn30krwr8y4b4fcz3psnj021wimb-x86_64-unknown-linux-musl-stage-final-gcc-wrapper-10.3.0/bin/x86_64-unknown-linux-musl-ld No pkg-config found Using runghc version 8.10.4 found on system at: /nix/store/qam66cqpir6n9260xzs19mbdx6kd6md0-ghc-8.10.4/bin/runghc Using strip version 2.35 given by user at: /nix/store/041mz911y2gc34ga7ppzj69p2w4qdvv3-x86_64-unknown-linux-musl-binutils-2.35.1/bin/x86_64-unknown-linux-musl-strip Using tar found on system at: /nix/store/rc34ffh62g42vavbsiw5aididd1dmwl4-gnutar-1.34/bin/tar No uhc found building Preprocessing library for QuickCheck-2.14.2.. Building library for QuickCheck-2.14.2.. [ 1 of 16] Compiling Test.QuickCheck.Exception ( src/Test/QuickCheck/Exception.hs, dist/build/Test/QuickCheck/Exception.o, dist/build/Test/QuickCheck/Exception.dyn_o ) [ 2 of 16] Compiling Test.QuickCheck.Random ( src/Test/QuickCheck/Random.hs, dist/build/Test/QuickCheck/Random.o, dist/build/Test/QuickCheck/Random.dyn_o ) [ 3 of 16] Compiling Test.QuickCheck.Gen ( src/Test/QuickCheck/Gen.hs, dist/build/Test/QuickCheck/Gen.o, dist/build/Test/QuickCheck/Gen.dyn_o ) attempting to use module ‘QuickCheck-2.14.2-1rgy6m2txlL6P9xFMqTWv8:Test.QuickCheck.Random’ (src/Test/QuickCheck/Random.hs) which is not loaded builder for '/nix/store/6wg17fsvhs4w3gjnzrfngz6fa8r3kkaj-QuickCheck-2.14.2-x86_64-unknown-linux-musl.drv' failed with exit code 1 building '/nix/store/krmypsgknq1k3ri8k9cjs4bdm04s0zg3-assoc-1.0.2-x86_64-unknown-linux-musl.drv'... cannot build derivation '/nix/store/9r2r3nk9bkmdybqlgm3mv92q2jb5w4gf-hspec-2.7.10-x86_64-unknown-linux-musl.drv': 1 dependencies couldn't be built cannot build derivation '/nix/store/flicif49fyzh5fa73vs9k6fkvclzy728-fuzzy-dates-0.1.1.2-x86_64-unknown-linux-musl.drv': 1 dependencies couldn't be built cannot build derivation '/nix/store/a0z03psffaa49pb90x1ys1rk2idiqsv8-burrow-0.1.0.0-x86_64-unknown-linux-musl.drv': 1 dependencies couldn't be built error: build of '/nix/store/a0z03psffaa49pb90x1ys1rk2idiqsv8-burrow-0.1.0.0-x86_64-unknown-linux-musl.drv' failed make: *** [Makefile:4: build] Error 100

hyperrealgopher · 2021-06-25T00:35:56Z

Closing this for now since static builds were introduced in 0.1.0.0 using github workflow and docker/alpine.

hyperrealgopher force-pushed the feature/nix-packaging branch 2 times, most recently from 132ccff to ce57085 Compare June 16, 2021 05:38

hyperrealgopher force-pushed the feature/nix-packaging branch from ce57085 to b6d022f Compare June 16, 2021 06:06

sternenseemann reviewed Jun 16, 2021

View reviewed changes

sternenseemann added a commit to NixOS/nixpkgs that referenced this pull request Jun 16, 2021

haskellPackages.errata: no longer mark as broken

82ba334

As pointed out here [1], errata has started working on its own again. [1]: someodd/burrow#27 (comment)

someodd added 2 commits June 16, 2021 12:52

add regen to rebuild default.nix from cabal

bd2414d

hyperrealgopher closed this Jun 25, 2021

	inherit (nixpkgs) pkgs;
	inherit (pkgs) haskellPackages;
	inherit (nixpkgs) haskellPackages;

Conversation

hyperrealgopher commented Jun 16, 2021

Uh oh!

hyperrealgopher commented Jun 16, 2021

Uh oh!

hyperrealgopher commented Jun 16, 2021

Uh oh!

hyperrealgopher commented Jun 16, 2021

Uh oh!

hyperrealgopher commented Jun 16, 2021

Uh oh!

hyperrealgopher commented Jun 16, 2021

Uh oh!

hyperrealgopher commented Jun 16, 2021

Uh oh!

sternenseemann commented Jun 16, 2021

Uh oh!

sternenseemann commented Jun 16, 2021

Uh oh!

sternenseemann Jun 16, 2021

Choose a reason for hiding this comment

Uh oh!

hyperrealgopher Jun 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sternenseemann Jun 16, 2021

Choose a reason for hiding this comment

Uh oh!

sternenseemann Jun 16, 2021

Choose a reason for hiding this comment

Uh oh!

sternenseemann commented Jun 16, 2021

Uh oh!

hyperrealgopher commented Jun 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hyperrealgopher commented Jun 25, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hyperrealgopher Jun 16, 2021 •

edited

Loading

hyperrealgopher commented Jun 16, 2021 •

edited

Loading