`doom compile` broken upon fresh install

What did you expect to happen?

I want to do a fresh install of Doom. Then I want to do a Doom compilation. And I'd expect the compilation to largely go off successfully and return with a zero exit code.

I know compiling is generally discouraged, but I get some comfort knowing that the code can compile, and accept the responsibility of remembering to do a doom clean as needed.

What actually happened?

I did the fresh install of Doom:

❯ rm -rf ~/.config/doom ~/.config/emacs ~/.emacs.d ~/.doom.d
❯ git clone https://github.com/hlissner/doom-emacs ~/.emacs.d && ~/.emacs.d/bin/doom install
Cloning into '/home/tnks/.emacs.d'...
remote: Enumerating objects: 112806, done.
remote: Counting objects: 100% (299/299), done.
remote: Compressing objects: 100% (149/149), done.
remote: Total 112806 (delta 164), reused 270 (delta 149), pack-reused 112507
Receiving objects: 100% (112806/112806), 27.21 MiB | 8.00 MiB/s, done.
Resolving deltas: 100% (79436/79436), done.
Installing Doom Emacs!

✓ Created ~/.doom.d/
  - Creating ~/.doom.d/init.el
  ✓ Done!
  - Creating ~/.doom.d/config.el
  ✓ Done!
  - Creating ~/.doom.d/packages.el
  ✓ Done!
Generate an envvar file? (see `doom help env` for details) (y or n) y
> Generating envvars file
  ✓ Generated ~/.emacs.d/.local/env
Installing plugins
…
Deploying commit-msg and pre-push git hooks
  - Loaded "/home/tnks/.emacs.d/ci.el"
    ✓ Created ~/.emacs.d/.git/hooks/commit-msg
    ✓ Created ~/.emacs.d/.git/hooks/pre-push
Download and install all-the-icon's fonts? (y or n) y
✓  Finished! Doom is ready to go!
…

Then I tried to do a Doom compilation:

❯ ~/.emacs.d/bin/doom compile --verbose
WARNING: Changes made to your config after compiling it won't take effect until
this command is rerun or you run 'doom clean'! It will also make error backtraces
much more difficult to decipher.

If you intend to use it anyway, remember this or it will come back to bite you!

Continue anyway? (y or n) y
> Byte-compiling your config...
  ✓ (  1/69) Compiling :core core module...
  ✓ (  2/69) Compiling :core core-ui module...
  ✓ (  3/69) Compiling :core core-start module...
  ✓ (  4/69) Compiling :core core-projects module...
  ✓ (  5/69) Compiling :core core-packages module...
…
  ✓ ( 42/69) Compiling :completion company module...
  x ( 42/69) Failed to compile modules/completion/company/config.el
  ✓ ( 43/69) Compiling :completion vertico module...
  x ( 43/69) Failed to compile modules/completion/vertico/config.el
  ✓ ( 44/69) Compiling :ui doom module...
  ✓ ( 45/69) Compiling :ui doom-dashboard module...
  ✓ ( 46/69) Compiling :ui hl-todo module...
  ✓ ( 47/69) Compiling :ui modeline module...
  x ( 47/69) Failed to compile modules/ui/modeline/config.el
  x ( 47/69) Failed to compile modules/ui/modeline/autoload.el
  ✓ ( 48/69) Compiling :ui ophints module...
  ✓ ( 49/69) Compiling :ui popup module...
  x ( 49/69) Failed to compile modules/ui/popup/config.el
  ✓ ( 50/69) Compiling :ui vc-gutter module...
  ✓ ( 51/69) Compiling :ui vi-tilde-fringe module...
  ✓ ( 52/69) Compiling :ui workspaces module...
  ✓ ( 53/69) Compiling :editor evil module...
  x ( 53/69) Failed to compile modules/editor/evil/init.el
  ✓ ( 54/69) Compiling :editor file-templates module...
  ✓ ( 55/69) Compiling :editor fold module...
  ✓ ( 56/69) Compiling :editor snippets module...
  ✓ ( 57/69) Compiling :emacs dired module...
  x There was an unexpected runtime error
    Message: (featurep! +dirvish nil nil) couldn’t figure out what module it was called from (in /home/tnks/.emacs.d/bin/doom)
    Backtrace:
      (signal error ("(featurep! +dirvish nil nil) couldn’t figure ...
      (error "(featurep! %s %s %s) couldn't figure out what module ...
      (if module (memq category (doom-module-get (car module) (cdr ...
      (let* ((module (and t (doom-module-from-path)))) (if module (...
      (cond (flag (memq flag (doom-module-get category module :flag...
      (and (cond (flag (memq flag (doom-module-get category module ...
      ((closure (doom-disabled-packages t) (category &optional modu...
      (featurep! +dirvish)
      (not (featurep! +dirvish))
      (and (not (featurep! +dirvish)) (not (featurep! +ranger)))
      (eval (and (not (featurep! +dirvish)) (not (featurep! +ranger...
      (not (eval pred t))
    ! Wrote extended backtrace to ~/.emacs.d/.local/logs/cli.doom.220703184525.40969.error

The command returns with an exit code of 255.

The full log was too long for a pastebin.com, but I've put it up the error log as a Gist.

Describe your attempts to resolve the issue

It took a while to get to get to this point. At first I was using a Emacs 28.1 with native compilation enabled. I tried using a plan Emacs 28.1 without native compilation. Both gave me the same results.

I also had a very custom private configuration. But I realized that might be confusing matters. So I moved that aside to illustrate that this problem exists for a fresh/vanilla installation.

Also, if you disable modules in the default init.el provided, it seems the same problem surfaces in other modules (but not every module). It's just easiest to see the problem with the default installation.

What I haven't done is try to do a Git bisection to determine which commit introduced the problem. If I do that, I'll come back with a followup.

Steps to reproduce

1. Have a plain 28.1 Emacs installed (doesn't matter whether with native compilation or not)
2. Do a fresh install of Doom (answering "y" to all interactive questions):

rm -rf ~/.config/doom ~/.config/emacs ~/.emacs.d ~/.doom.d
git clone https://github.com/hlissner/doom-emacs ~/.emacs.d
~/.emacs.d/bin/doom install

3. Do a Doom compilation:

~/.emacs.d/bin/doom compile --verbose

System Information

https://pastebin.com/m8nb3Dt4

I can't help myself. Tried an old commit to start a process of bisection. Probably too old, but this commit from May 9 doesn't have the problem.

Good:

• May 9: 8a27eb99bec5a955833b6d23f431c5cc39e91f7f
• May 12: 797866b59601f93647a8f9bca644ad5766a03978

Bad:

• Jun 17: 7d8b7b4fc2f3d4c3c3d2492d356d3af4606962cf
  • Failed to compile modules/lang/org/config.el

Watching a TV show and doing this in the background. I'll see if I can hone in on a commit introducing the problem. I'll update this comment with what I find.

I wonder how unreasonable it is to see if we can do a compilation of a configuration with a good amount of coverage. Seems like it would catch good errors earlier. But it also is clearly infrastructure overhead.

A bisect for me shows the error for a default installation happening somewhere in cb03d3258d05026796c4217bece9d7c01c659e8e...c4ac2ab384228f8c13eae641d7288739305af036. Unfortunately #6462 caused Doom upgrades to fail in this range so bisecting further will require some manual patching.

The specific issue in this range is that org fails to compile, and within this range 373386173a5bcf8aa7ba6bbb719b4b71cefa5f70 did bump org. However this doesn't seem like the fundamental issue, especially since more modules started failing after this range. It's likely the breaking change was somewhere before this, but didn't break modules in the default installation.

@dominicm00 yeah, if you disable the dired module you just see the same problem come up in another context/module. Your theory that there's some bug that is surfacing in various modules over time, starting with org seems plausible given what I'm seeing.

There's something I haven't gotten a good answer on yet. Does anyone have a sense of whether the problem is just with the compilation of some modules, or if there are indeed problems that would surface as runtime defects if people just went through a particular code path?

I think the safe way to go is just to fix the compilation and not have to answer this question. But I'm understanding that it takes time to triage, diagnose, and then fix this. In the meantime, just curious about what people think the scope of the problem is.

I did a lot of spelunking of Git today, and I thought I'd document what I've found.

First off, my methodology. I'm not sure if this is the best way to reset Doom when hopping around commits, but this is what I'm doing:

find ~/.config/emacs -name '*.elc' -exec rm {} +
find ~/.config/emacs -name '*.eln' -exec rm {} +
doom clean && doom sync && doom purge
doom compile

The purge might be excessive.

While hopping around commits, I found there were some unrelative breaks that covered up the bug reported by this issue. eb7329d91 fixed unbalanced parenthesis introduced by fb1c8eb11. Similarly, 343def0b1 fixes a void-function reference introduced by 74f3c1d11.

I did some interactive rebasing on a private branch to merge these fixes into the commits that introduced the problem.

That left me with one more bug that was fixed covering up this issue's bug. That bug was introduced with a major refactor of how Doom starts up.

The last known good doom compile is in commit 76431f699. Right after this good commit, 1402db512 introduces the refactor. This refactor has a bug where doom compile stops working with a void-function call to doom-initialize. This bug is fixed with 74f3c1d11. I did more rebasing, and pushed the fix down as far as I could.

After all that, I'm confident that the compilation of various modules was broken (this issue) by a commit in this refactor. I'm confident the commit responsible for this bug is somewhere in 6c0b7e153...1402db512^. It's just hard to do more Git bisection further because the problem is hidden by another problem of the void-function call to doom-initialize coming from the refactor.

There are 16 commits in my range so I've definitely narrowed it down. But it's still enough lines of change that it's not obvious to me where the problem is.

Okay, I moved around unrelated commits while rebasing, and I'm sure the problem was introduced in one of two refactor commits:

• 1402db512 - refactor: how Doom starts up
• 6c0b7e153 - refactor!(cli): rewrite CLI framework libraries

These aren't small changes, but 2 commits are better than 16. I think this is as far as I can go with Git bisection. I need to look at the actual code more now.

I'm glad I got down some record of my work above, but I didn't spell it out in a way that was clearly reproducible. Today I retraced my Git steps and minimized them to the following so others can follow along:

1. Start with the last known good commit: 76431f6 [fix(tree-sitter): ensure load order]
2. Cherrypick 1402db5 [refactor: how Doom starts up]
3. Cherrypick 6c0b7e1 [refactor!(cli): rewrite CLI framework libraries]
4. Cherrypick 74f3c1d [fix(cli): doom {compile,clean} errors]
5. Cherrypick 343def0 [fix(cli): void-function relpath on 'doom compile']

This is the smallest number of commits I could minimize to that retains the same bug that we have on master right now.

The last two cherrypicks are small fixes that just let you be able to run doom compile at all. I don't think they are related to the problem. As I mentioned before, it's the big refactor commits that I believe introduced the problem.

any updates?

here is my current output

pi@pi:~ $ ~/.emacs.d/bin/doom compile
WARNING: Changes made to your config after compiling it won't take effect until
this command is rerun or you run 'doom clean'! It will also make error backtraces
much more difficult to decipher.

If you intend to use it anyway, remember this or it will come back to bite you!

Continue anyway? (y or n) y
> Byte-compiling your config...
  ✓ (  1/68) Compiling :core doom module...
  ✓ (  2/68) Compiling :core doom-ui module...
  ✓ (  3/68) Compiling :core doom-start module...
  ✓ (  4/68) Compiling :core doom-projects module...
  ✓ (  5/68) Compiling :core doom-packages module...
  ✓ (  6/68) Compiling :core doom-modules module...
  ✓ (  7/68) Compiling :core doom-lib module...
  ✓ (  8/68) Compiling :core doom-keybinds module...
  ✓ (  9/68) Compiling :core doom-editor module...
  ✓ ( 10/68) Compiling :core doom-cli module...
  ✓ ( 11/68) Compiling :core upgrade module...
  ✓ ( 12/68) Compiling :core test module...
  ✓ ( 13/68) Compiling :core sync module...
  ✓ ( 14/68) Compiling :core run module...
  ✓ ( 15/68) Compiling :core make module...
  ✓ ( 16/68) Compiling :core install module...
  ✓ ( 17/68) Compiling :core info module...
  ✓ ( 18/68) Compiling :core env module...
  ✓ ( 19/68) Compiling :core compile module...
  ✓ ( 20/68) Compiling :core ci module...
  ✓ ( 21/68) Compiling :core autoloads module...
  ✓ ( 22/68) Compiling :core completions module...
  ✓ ( 23/68) Compiling :core ui module...
  ✓ ( 24/68) Compiling :core themes module...
  ✓ ( 25/68) Compiling :core text module...
  ✓ ( 26/68) Compiling :core system module...
  ✓ ( 27/68) Compiling :core store module...
  ✓ ( 28/68) Compiling :core sessions module...
  ✓ ( 29/68) Compiling :core scratch module...
  ✓ ( 30/68) Compiling :core sandbox module...
  ✓ ( 31/68) Compiling :core projects module...
  ✓ ( 32/68) Compiling :core process module...
  ✓ ( 33/68) Compiling :core print module...
  ✓ ( 34/68) Compiling :core plist module...
  ✓ ( 35/68) Compiling :core help module...
  ✓ ( 36/68) Compiling :core git module...
  ✓ ( 37/68) Compiling :core fonts module...
  ✓ ( 38/68) Compiling :core files module...
  ✓ ( 39/68) Compiling :core docs module...
  ✓ ( 40/68) Compiling :core debug module...
  ✓ ( 41/68) Compiling :core config module...
  ✓ ( 42/68) Compiling :core buffers module...
  ✓ ( 43/68) Compiling :completion company module...
  x ( 43/68) Failed to compile modules/completion/company/config.el
  ✓ ( 44/68) Compiling :completion vertico module...
Source file `/home/pi/.emacs.d/.local/straight/build-27.1/compat/compat.el' newer than byte-compiled file; using older file
Source file `/home/pi/.emacs.d/.local/straight/build-27.1/compat/compat-24.el' newer than byte-compiled file; using older file
Source file `/home/pi/.emacs.d/.local/straight/build-27.1/compat/compat-25.el' newer than byte-compiled file; using older file
Source file `/home/pi/.emacs.d/.local/straight/build-27.1/compat/compat-26.el' newer than byte-compiled file; using older file
Source file `/home/pi/.emacs.d/.local/straight/build-27.1/compat/compat-27.el' newer than byte-compiled file; using older file
Source file `/home/pi/.emacs.d/.local/straight/build-27.1/compat/compat-28.el' newer than byte-compiled file; using older file
Source file `/home/pi/.emacs.d/.local/straight/build-27.1/compat/compat-28.el' newer than byte-compiled file; using older file
  x ( 44/68) Failed to compile modules/completion/vertico/config.el
  ✓ ( 45/68) Compiling :ui doom module...
  ✓ ( 46/68) Compiling :ui doom-dashboard module...
  ✓ ( 47/68) Compiling :ui hl-todo module...
  ✓ ( 48/68) Compiling :ui modeline module...
  x ( 48/68) Failed to compile modules/ui/modeline/config.el
  x ( 48/68) Failed to compile modules/ui/modeline/autoload.el
  ✓ ( 49/68) Compiling :editor evil module...
  x ( 49/68) Failed to compile modules/editor/evil/init.el
  ✓ ( 50/68) Compiling :editor file-templates module...
  ✓ ( 51/68) Compiling :editor fold module...
  ✓ ( 52/68) Compiling :editor lispy module...
Source file `/home/pi/.emacs.d/.local/straight/build-27.1/project/project.el' newer than byte-compiled file; using older file
  ✓ ( 53/68) Compiling :editor parinfer module...
  ✓ ( 54/68) Compiling :editor snippets module...
  ✓ ( 55/68) Compiling :emacs dired module...

I tried to compile to speedup on Raspberry Pi

doom compile and doom clean were removed in 63c470bff32875098fa5a7f205e0b5eb08247686, so I will close this issue.