[OTHER] New build system not compatible with Nix in many places #59
Labels
No labels
bug
confirmed
contribution welcome
duplicate
enhancement
good first issue
help wanted
important
invalid
other
question
upstream
wontfix
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
celenity/Dove#59
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Please explain your inquiry.
I am opening this issue to start a discussion. I might miss something here or someone more experienced might have an idea.
(Opening in the Dove repo, as this is where I encountered the problem. Should apply to Phoenix as well but Dove build fails first for me. Let me know if I should open an issue for Phoenix and link here.)
I have tried to adjust the build scripts of Dove and Phoenix to work with Nix but failed to do so until now.
Unless I am missing something, I think there is no simple solution to get Dove and Phoenix to build on Nix with the new build system. As I see it, it just isn't vey compatible with the limitations of Nix.
Nix generally doesn't allow to write files or create dirs in the nix-store during build. Instead it can use a tmp dir during buildPhase, then build, then copy outputs during installPhase (someone had a similar problem here).
This leads to build failures in a lot of places when trying to build Phoenix or Dove. Mostly this happens whenever files / dirs are written / created under
DOVE_BUILD.Some are easy to fix, e.g. setting
DOVE_LOG_BUILD=0andPHOENIX_LOG_BUILD=0inflake.nixornix/package.nix. However, there some more occurences that might require more branching in the build scripts, e.g. by checking for the value of our recently introducedDOVE_NIX_FLAKEenv variable and writing files to a/tmpdir. Then we have to also adjust the parts that consume such files to use the/tmpdir. I think this would quite some modifications in the build scripts and therefore break very easily.I thought, maybe we can just copy the whole build dir to
/tmpand just setDOVE_BUILDto that copy, this should allow us to write files and create dirs and I think we can then just adjust the nix installPhase to copy the results from there. However, I am not sure yet if settingDOVE_BUILDwould break other parts of the build script.I will continue to try to fix it but wanted to get this out there, in case anyone has a better idea.
Thanks for the report, let's definitely get this taken care of.
DOVE_BUILD(andPHOENIX_BUILD) currently aren't configurable, so I don't think just changing them would work. I think the first thing we need to do to fix this is make both of those envs configurable. I don't think it being configurable should cause any issues. I think we should also add a check for Nix atenv_helpers(We can probably just move the existing logic for ex.DOVE_NIX_FLAKEfromenv_common, and implement that in Phoenix as well). Once we do that, we can make it so that the default paths forDOVE_BUILD(andPHOENIX_BUILD) for Nix are/tmp(instead of the standard/current location). We can also do the same for the log location, and any other vars as necessary.I'll work on this later today/tomorrow. Making this a priority to fix, apologies for the issues here.
NVM, I forgot to account for the fact that Nix can be used on both Linux and OS X. So it'll be best to keep using a separate dedicated env variable here.
Hey @ddogfoodd, Do you mind trying to install Phoenix from its
devbranch, and confirm that the install works? I made some changes that I believe should help fix this issue. If Phoenix installs as expected, I'll implement changes of the same nature for Dove as well - but first want to confirm that it does work correctly.@celenity Phoenix
devbranch builds fine. Thank you very much!Great, thanks for confirming! Will implement the same changes for Dove.
Should now be fixed (for the
devbranch currently, though I do intend to have a release out within the next few days, so will be fixed on production/pagesthen)! Please LMK if you have any further issues.The build was still broken for me, but I wanted to wait for the new dove release, just to be sure.
#61 seems to fix the build.
Hey @ddogfoodd, if you have a chance and don't mind, can you please test installing Dove on Nix from the
devbranch? I made some changes to how Python is handled that I suspect might cause issues with Nix, so I'd like to test and resolve them before next release.The main change I made was adding a few new environment variables:
DOVE_PIP_EXEC- Location ofpipDOVE_PYTHON- Location ofpythonI did also have to set an instance in the build script to call Python from
${DOVE_PIP_DIR}/bin/python- for some reason, usingDOVE_PYTHONdidn't seem to work there.My guess is we'll need to configure these variables for Nix, but I'm not sure how we should configure them and what we should set the values to. Please LMK if you have any thoughts.
also cc: @jbgi Curious if you have any thoughts.
Fails with
/build/4y8mxs0y70lwlhpasm3a727mc228z4v2-source/scripts/build-dove.sh: line 80: /tmp/dove/build/pyenv/bin/python: No such file or directoryAFAIK for nix we probably want to skip pip and instead add requirements to the
nativeBuildInputs(here in the flake).nativeBuildInputsis for things required during build time rather than run time, which probably always applies since we just build the dove config.So I think we just skip pip in the build scripts and the
DOVE_PIP_EXECenv var whenDOVE_NIXis set.I think we can skip
DOVE_PYTHONas well and just usepythonwithout specifying the path to the python binary when using nix. I can take a look later. I think we could also setDOVE_PYTHONto the python path in the nix-store. I don't see a point in doing so, if skipping it works, though.I think the easiest way of handling this would be #65.
Otherwise we could set
DOVE_PYTHONorDOVE_PIP_DIRlike this. However, this would result in a "ModuleNotFoundError: No module named 'lxml'" error, even when usingDOVE_PIP_DIR.I am not sure how to fix this the nix way at the moment. Just wanted to throw it in here, in case you prefer setting the env var even when using nix and someone else knows how we can get the dependencies in there as well.
Thank you for caring about us nix users btw. I am thankful for your work and always happy to test and help where I can.
Thank you! I just merged your PR - that sounds like a reasonable solution for the time being. Appreciate your time and assistance.
@ddogfoodd Unrelated, but do you know whether it's possible to set environment variables globally/persistently on Nix? For typical Linux distros, Dove accomplishes this by copying
dov-env-overrides.shfile to/etc/profile.d/dove-env-overrides.sh. It'd be great if there was some way we could set environment variables for Nix users as well, will likely become more important in the future.I just verified it works for NixOS like this:
They could be set in the
flake.nixlike this.Some more information here.
However, there are two things I am unsure about.
@ddogfoodd Thanks for sharing!
The environment variables I'd intend to set this way actually aren't related to the build process, they are used and evaluated by Firefox/Thunderbird at build-time. Currently, they're mainly just used to ex. disable Mozilla's crash reporter, but I'm also looking into adding one to set the OS platform as well, so that we can potentially unify Dove and Phoenix into having one unified
.cfgfile for all platforms. Since you said these environment variables persist, they should be perfect for this use case. I'll look into adding them!Hey @ddogfoodd I've recently made some major changes to Dove and Phoenix's build systems and how they're structured - if you have a chance and don't mind, can you please test installing Phoenix and Dove on Nix from the
devbranches? Would like to make sure everything is okay on the Nix side. I did update the Nix files to point to the new paths where relevant, but want to make sure I didn't ex. miss anything and that there aren't any other places causing issues.Very much appreciated!
Thank you for your continued efforts on improving the build systems and also for keeping Nix users in mind!
I will be able to test and take a look in a few hours.
Looks good to me!
/etc/firefoxcontains:defaults/pref/phoenix.jsphoenix/assets/...with phoenix brandingphoenix/specs/...with userjs and permissions for specific servicespolicies/policies.jsonphoenix.cfgFiles content looks fine as well.
Just in case, that's not wanted behavior, I noticed Help > About now shows Phoenix - Extended.
@ddogfoodd wrote in #59 (comment):
Great, thank you for confirming!! I assume it also worked similarly for Dove?
Was this on Firefox/Phoenix or Thunderbird/Dove?
Yes, Dove worked similar.
Firefox shows Phoenix - Extended.
Thunderbird shows just Dove.
@ddogfoodd wrote in #59 (comment):
Great, thanks for confirming!!
Hmm, to confirm, did you choose to enable Phoenix Extended/use the specialized config for it? If so, I’d say that’s intended (weird it didn’t show before though), but, if not, that’s indeed a bug/unexpected and something we’ll need to investigate.
@celenity wrote in #59 (comment):
No, this is without opting for extended. So it shows Extended by default when installed via Nix it seems.
@ddogfoodd wrote in #59 (comment):
I believe I just discovered/fixed the issue, thank you for letting me know! Should now be fixed (as of
celenity/Phoenix@d88a82c717).@celenity wrote in #59 (comment):
Confirmed! Thanks!
@ddogfoodd wrote in #59 (comment):
Thanks for confirming!!