haskellPackages: update stackage and hackage by sternenseemann · Pull Request #125977 · NixOS/nixpkgs

sternenseemann · 2021-06-06T16:45:06Z

This Merge

This PR is the regular merge of the haskell-updates branch into master.

This branch is being continually built and tested by hydra at https://hydra.nixos.org/jobset/nixpkgs/haskell-updates.

I will aim to merge this PR by 2021-06-11. If I can merge it earlier, there might be successor PRs in that time window. As part of our rotation @maralorn will continue these merges from 2021-06-12 to 2021-06-26.

haskellPackages Workflow Summary

Our workflow is currently described in
pkgs/development/haskell-modules/HACKING.md.

The short version is this:

We regularly update the Stackage and Hackage pins on haskell-updates (normally at the beginning of a merge window).
The community fixes builds of Haskell packages on that branch.
We aim at at least one merge of haskell-updates into master every two weeks.
We only do the merge if the mergeable job is succeeding on hydra.
If a maintained package is still broken at the time of merge, we will only merge if the maintainer has been pinged 7 days in advance. (If you care about a Haskell package, become a maintainer!)

This is the follow-up to #125429. ~~This PR aims to update our default GHC version to 8.10.5.~~ GHC 8.10.5 has been tested as part of this PR. The conclusion is that it's not worth updating for now: #125977 (comment) New PR for GHC 8.10.5 is #126195.

If I get to it, I have the following additional plans to work towards improvements of mostly our GHC derivation, but they are not a hard requirement for merging this PR:

Add GHC 9.2 (Ghc 9.2 #124999)
Improve GHC on Darwin (Fix issues with GHC outside of stdenv on Darwin #125367) and ~~get tests.writers.x86_64-darwin to work again~~ (faill because of rust as well)
Bonus points: Refactor GHC nix expressions
- binary
- source-built
Add a way to specify test flags in a clean way in generic-builder.nix (haskell-generic-builder: allow passing flags to the test suite(s) #126364)
Clean up some failing x86_64-darwin and aarch64-linux builds by updating unsupported-platforms

If we don't pass buildTarget to ./Setup copy and buildTarget is not empty it will try installing targets that don't exist and thus fail.

Make sure they are all prefixed with haskellPackages: except for update-hackage.sh which changes the top-level attribute all-cabal-hashes.

sternenseemann · 2021-06-06T22:04:54Z

GHC 8.10.5 failure on my mac, seems like a (lack of) sandboxing issue:

ln -s ghc-8.10.5 "/nix/store/qjixylxsm5rv0zn7zvzz2pfjpifwcq25-ghc-8.10.5/bin/ghc"
/nix/store/yzvj23zkg314xjywc3dmzdlqchkqq4m0-coreutils-8.32/bin/install -c -m 755 -d "/nix/store/qjixylxsm5rv0zn7zvzz2pfjpifwcq25-ghc-8.10.5/share/man"
/nix/store/yzvj23zkg314xjywc3dmzdlqchkqq4m0-coreutils-8.32/bin/install -c -m 755 -d "/nix/store/qjixylxsm5rv0zn7zvzz2pfjpifwcq25-ghc-8.10.5/share/man/man1"
/nix/store/yzvj23zkg314xjywc3dmzdlqchkqq4m0-coreutils-8.32/bin/install -c -m 644  docs/users_guide/build-man/ghc.1 "/nix/store/qjixylxsm5rv0zn7zvzz2pfjpifwcq25-ghc-8.10.5/share/man/man1"
if [ -e "/usr/bin/xattr" ]; then "/usr/bin/xattr" -c -r .; fi
Traceback (most recent call last):
  File "/usr/bin/xattr-2.7", line 7, in <module>
    from pkg_resources import load_entry_point
  File "/nix/store/h9832pvm0airim7nfc5n7xvqqb7hzl24-python3.8-setuptools-54.2.0/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1367
    raise SyntaxError(e) from e
                            ^
SyntaxError: invalid syntax
make[1]: *** [ghc.mk:833: install_darwin] Error 1
make: *** [Makefile:128: install] Error 2
builder for '/nix/store/rqjwrp4j5zy8r8wn47l3mqkpfc1105c4-ghc-8.10.5.drv' failed with exit code 2
cannot build derivation '/nix/store/an3yqv76k9r9b2yr4syav49q886krfbz-hello-1.0.0.2.drv': 1 dependencies couldn't be built
error: build of '/nix/store/an3yqv76k9r9b2yr4syav49q886krfbz-hello-1.0.0.2.drv' failed

I'll look into making GHC take xattr from the right location.

pkgs/top-level/haskell-packages.nix

sternenseemann · 2021-06-08T09:58:37Z

The doctest failures seem to be related to this GHC issue. We may not be able to update GHC until 8.10.6. A somewhat accurate impression of the state of haskellPackages can be found here — build failures aren't too many it seems so far, but I'm concerned about the degraded quality. Also some of the packages that transitively fail are important to us (like termonad, …).

This commit has been generated by maintainers/scripts/haskell/update-hackage.sh

This commit has been generated by maintainers/scripts/haskell/regenerate-hackage-packages.sh

This reverts commit 683d06d. upstream resolved the issue we were experiencing.

sternenseemann · 2021-06-08T11:32:34Z

I'm fairly convinced that upgrading to 8.10.5 in this manner doesn't make sense. We could patch the issue causing the regressions, but we haven't set up aarch64-darwin bootstrap yet, so there is no actual benefit for us in upgrading.

I'll rebase the 8.10.5 changes out of this branch and open a separate PR for it. Also be warned that there will be a force push to this branch soon.

GHC calls otool on darwin which is contained in the stdenv.cc.bintools.bintools derivation and thus needs adding to the runtime PATH of GHC. Since this is toolchain specific technically, we check for cctools instead of darwin (although I don't know if GHC or nixpkgs work on macOS without cctools). This fixes usage of GHC in an environment where otool is not available and more specifically in stdenvNoCC which is used by writers.writeHaskell. Resolves #123228.

useLdGold previously just checked for useLLVM which (currently) implies `linker == "lld"`. However more accurate is to check the `linker` of the `targetPlatform` as it actually tells us which bintools package we can expect. `linker == "bfd"` implies that we are using the `binutils` package, so gold is available, so we can use it unless musl is the libc. `linker == "gold"` implies that gold is the default linker already and we should absolutely use it.

ghc: check for targetPlatform.linker to determine if gold is available

tests added breaking constraints which seem safe to lift. Co-authored-by: sterni <sternenseemann@systemli.org>

These attribute names were converted into aliases in the following changes: * 62733b3 * #104776 cabal2nix-unstable has been updated to be aware of these changes in 7a9080d, so these aliases should no longer cause issues when evaluating with `allowAliases = false`.

Stackage Nighly recently upgraded their version of hackage-db from 2.1.0 to 2.1.1. 2.1.1 had a compatibility fix for Cabal 3.4 [1]. However it did not increase the version bound on Cabal nor fails to compile with Cabal 3.2, so Stackage was able to update it. Unfortunately hackage-db with Cabal 3.2 causes observable issues [2] in cabal2nix, so we need to downgrade it for all compilers that still ship a Cabal version < 3.4. Also ideally we should update the constraints for hackage-db 2.1.0 and hackage-db 2.1.1 on hackage. See also [3]. [1]: NixOS/hackage-db#12 [2]: NixOS/cabal2nix#501 [3]: NixOS/hackage-db#14

Upgrade to 0.16 which has GHC 9.0.x support. Interesting since cabal2nix depends on memory.

0.29 supports GHC 9.0.x which is why we upgrade. Interesting because cabal2nix depends on cryptonite.

GHC 9.0.x seems to require that the `Main` module also defines the `main` IO action and does not just import it. This is the case with mono-traversable's test suite which is why we (temporarily) disable it.

sternenseemann · 2021-06-11T19:19:43Z

haskell-updates build report from hydra

evaluation 1677383 of nixpkgs commit f8c8bb0 as of 2021-06-11 19:12 UTC

Build summary

Almost all darwin builds are still queued. @maralorn I think all the failures of your packages are because of flaky tests / killed builds. I'll try restarting a few jobs and see if the problem resolves itself.

Platform	Failed ❌	DependencyFailed ❗	TimedOut ⌛🚫	Unfinished ⏳	Success ✔️
aarch64-linux 📱	32	73	8		6557
x86_64-darwin 🍎	3	18		5329	1285
x86_64-linux 🐧	14	21	1		6674

Maintained packages with failed dependency

Unmaintained packages with build failure

38 job(s)

Unmaintained packages with failed dependency

113 job(s)

Report generated with maintainers/scripts/haskell/hydra-report.hs

Adds support for GHC 9.0.x which we also test by compiling it with all available GHC versions on Hydra.

Test suite doesn't build with GHC 9.0.1 and since upstream is currently not invested in fixing it, we (temporarily) disable it. Upside: we can build hoogle again. Soostone/retry#71

sternenseemann · 2021-06-12T15:14:07Z

Handing over to @maralorn now, the queued builds didn't finish in time sadly. Our CI aggregate jobs all succeed on x86_64-linux (or at least should according to my local testing), so it is just a case of waiting for the queue, checking for extra regressions and marking the remaining failures as broken.

maralorn · 2021-06-12T16:35:55Z

haskell-updates build report from hydra

evaluation 1677793 of nixpkgs commit c0d39d2 as of 2021-06-12 16:30 UTC

Build summary

Platform	Failed ❌	DependencyFailed ❗	TimedOut ⌛🚫	Unfinished ⏳	Success ✔️
aarch64-linux 📱	31	71	3		6561
x86_64-darwin 🍎				5348	1285
x86_64-linux 🐧	11	5		2	6690

Unmaintained packages with build failure

31 job(s)

Unmaintained packages with failed dependency

75 job(s)

Report generated with maintainers/scripts/haskell/hydra-report.hs

maralorn · 2021-06-12T16:37:22Z

Marvelous this build report without any pings. Note: the two unfinished x86 builds are maintained and mergeable which are waiting for the darwin builds.

Linker failure outputs look like they are related to the GClosure stuff, so lets try disabling that flag on arm — originally the upstream cabal file disabled that flag by default if arch != x86_64-linux || != sparc64, so this seems to be actually correct.

This commit has been generated by maintainers/scripts/haskell/mark-broken.sh

sternenseemann added 2 commits June 6, 2021 18:40

haskell-generic-builder: fix buildTarget specification if isLibrary

670d6ac

If we don't pass buildTarget to ./Setup copy and buildTarget is not empty it will try installing targets that don't exist and thus fail.

haskellPackages.discord-haskell: unbreak

683d06d

sternenseemann requested review from cdepillabout and maralorn as code owners June 6, 2021 16:45

github-actions bot added the 6.topic: haskell General-purpose, statically typed, purely functional programming language label Jun 6, 2021

maintainers/scripts/haskell: make commit messages more consistent

be6de30

Make sure they are all prefixed with haskellPackages: except for update-hackage.sh which changes the top-level attribute all-cabal-hashes.

cdepillabout reviewed Jun 8, 2021

View reviewed changes

pkgs/top-level/haskell-packages.nix Outdated Show resolved Hide resolved

sternenseemann added 4 commits June 8, 2021 12:50

all-cabal-hashes: 2021-06-02T14:32:36Z -> 2021-06-06T15:18:11Z

81ea9c8

This commit has been generated by maintainers/scripts/haskell/update-hackage.sh

haskellPackages: regenerate package set based on current config

d7b18cb

This commit has been generated by maintainers/scripts/haskell/regenerate-hackage-packages.sh

haskellPackages.discord-haskell: build example binaries again

0b4e5ef

This reverts commit 683d06d. upstream resolved the issue we were experiencing.

haskellPackages.dhall-{nix,openapi}: pin to support dhall 1.38.*

13c72a9

sternenseemann force-pushed the haskell-updates branch from 7720995 to 13c72a9 Compare June 8, 2021 11:36

sternenseemann linked an issue Jun 8, 2021 that may be closed by this pull request

The haskell writer is broken on darwin #123228

Closed

sternenseemann and others added 7 commits June 8, 2021 22:17

Merge pull request #126205 from sternenseemann/ghc-linker-checks

4f97d78

ghc: check for targetPlatform.linker to determine if gold is available

haskellPackages.wide-word: unbreak

121fdde

tests added breaking constraints which seem safe to lift. Co-authored-by: sterni <sternenseemann@systemli.org>

haskellPackages.parallel-io: remove now unnecessary override

f1692c9

haskellPackages: update list of transitively broken packages

8ba5828

haskellPackages.nvfetcher: move overrideCabal to configuration-nix.nix

233087e

haskellPackages.cabal2nix-unstable: 2021-05-28 -> 2021-06-10

7a9080d

sternenseemann added 6 commits June 10, 2021 14:10

haskell.packages.ghc901.memory: 0.15 -> 0.16

c6ed457

Upgrade to 0.16 which has GHC 9.0.x support. Interesting since cabal2nix depends on memory.

haskell.packages.ghc901.cryptonite: 0.28 -> 0.29

362c8da

0.29 supports GHC 9.0.x which is why we upgrade. Interesting because cabal2nix depends on cryptonite.

haskell.packages.ghc901.mono-traversable: unbreak

f8c8bb0

GHC 9.0.x seems to require that the `Main` module also defines the `main` IO action and does not just import it. This is the case with mono-traversable's test suite which is why we (temporarily) disable it.

Merge remote-tracking branch 'origin/master' into haskell-updates

922a444

sternenseemann added 2 commits June 12, 2021 00:36

haskellPackages.cabal2nix-unstable: 2021-06-10 -> 2021-06-12

71ffc07

Adds support for GHC 9.0.x which we also test by compiling it with all available GHC versions on Hydra.

haskell.packages.ghc901.retry: dont build test suite

c0d39d2

Test suite doesn't build with GHC 9.0.1 and since upstream is currently not invested in fixing it, we (temporarily) disable it. Upside: we can build hoogle again. Soostone/retry#71

sternenseemann and others added 6 commits June 12, 2021 18:37

haskellPackages.hackage-db: Reactivate hydra job

7b92c7f

haskellPackages.cabal2nix-unstable: Bump for alsa-lib alias

37c8bfc

haskellPackages: mark builds failing on hydra as broken

6eaf494

This commit has been generated by maintainers/scripts/haskell/mark-broken.sh

Merge branch 'master' into haskell-updates

105bb9c

haskellPackages.mptcp-pm: Disable on darwin to fix eval error

0c8e2c2

maralorn merged commit 06cc417 into master Jun 12, 2021

maralorn mentioned this pull request Jun 12, 2021

haskellPackages: update stackage and hackage #126693

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

haskellPackages: update stackage and hackage#125977

haskellPackages: update stackage and hackage#125977
maralorn merged 29 commits intomasterfrom
haskell-updates

sternenseemann commented Jun 6, 2021 •

edited

Loading

Uh oh!

sternenseemann commented Jun 6, 2021

Uh oh!

Uh oh!

sternenseemann commented Jun 8, 2021 •

edited

Loading

Uh oh!

sternenseemann commented Jun 8, 2021

Uh oh!

sternenseemann commented Jun 11, 2021 •

edited

Loading

Uh oh!

sternenseemann commented Jun 12, 2021

Uh oh!

maralorn commented Jun 12, 2021

Uh oh!

maralorn commented Jun 12, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

sternenseemann commented Jun 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This Merge

haskellPackages Workflow Summary

Uh oh!

sternenseemann commented Jun 6, 2021

Uh oh!

Uh oh!

sternenseemann commented Jun 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sternenseemann commented Jun 8, 2021

Uh oh!

sternenseemann commented Jun 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

haskell-updates build report from hydra

Build summary

Maintained packages with failed dependency

Unmaintained packages with build failure

Unmaintained packages with failed dependency

Uh oh!

sternenseemann commented Jun 12, 2021

Uh oh!

maralorn commented Jun 12, 2021

haskell-updates build report from hydra

Build summary

Unmaintained packages with build failure

Unmaintained packages with failed dependency

Uh oh!

maralorn commented Jun 12, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sternenseemann commented Jun 6, 2021 •

edited

Loading

sternenseemann commented Jun 8, 2021 •

edited

Loading

sternenseemann commented Jun 11, 2021 •

edited

Loading