Add build jobgraph support by waruqi · Pull Request #6209 · xmake-io/xmake

waruqi · 2025-03-11T14:55:19Z

waruqi · 2025-04-06T14:55:16Z

gcc does not seem to support c++modules with pch very well

ruki@0738bc1c5a27:/mnt/xmake/tests/projects/c++/modules/hello_with_pch$ xmake -rv
[  1%]: compiling.release src/test.h
/usr/bin/gcc -c -m64 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -I build/.objs/hello/linux/x86_64/release/src/cxx -fmodules-ts -D_GLIBCXX_USE_CXX11_ABI=0 -DNDEBUG -x c++-header -o build/.objs/hello/linux/x86_64/release/src/cxx/test.h.gch src/test.h
[  2%]: <hello> generating.module.deps src/main.cpp
/usr/bin/gcc -E -x c++ -m64 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -I build/.objs/hello/linux/x86_64/release/src/cxx -include test.h -D_GLIBCXX_USE_CXX11_ABI=0 -DNDEBUG src/main.cpp -o build/.gens/hello/linux/x86_64/release/rules/bmi/cache/modules/2d9b7c0b/main.cpp.i
[  3%]: <hello> generating.module.deps src/hello.mpp
/usr/bin/gcc -E -x c++ -m64 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -I build/.objs/hello/linux/x86_64/release/src/cxx -include test.h -D_GLIBCXX_USE_CXX11_ABI=0 -DNDEBUG src/hello.mpp -o build/.gens/hello/linux/x86_64/release/rules/bmi/cache/modules/2d9b7c0b/hello.mpp.i
[ 25%]: <hello> compiling.module.release hello
/usr/bin/gcc -c -m64 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -I build/.objs/hello/linux/x86_64/release/src/cxx -include test.h -fmodules-ts -D_GLIBCXX_USE_CXX11_ABI=0 -DNDEBUG -fmodule-mapper=/tmp/.xmake999/250406/hello/src/hello.mpp -x c++ -o build/.objs/hello/linux/x86_64/release/src/hello.mpp.o src/hello.mpp
error: src/hello.mpp:1:1: error: module-declaration only permitted as first declaration, or ending a global module fragment
    1 | module;
      | ^~~~~~
src/hello.mpp:4:8: error: module-declaration only permitted as first declaration, or ending a global module fragment
    4 | export module hello;
      |        ^~~~~~
src/hello.mpp:6:1: error: 'export' may only occur after a module interface declaration
    6 | export namespace hello {
      | ^~~~~~

Arthapz · 2025-04-06T15:38:32Z

gcc does not seem to support c++modules with pch very well

ruki@0738bc1c5a27:/mnt/xmake/tests/projects/c++/modules/hello_with_pch$ xmake -rv
[  1%]: compiling.release src/test.h
/usr/bin/gcc -c -m64 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -I build/.objs/hello/linux/x86_64/release/src/cxx -fmodules-ts -D_GLIBCXX_USE_CXX11_ABI=0 -DNDEBUG -x c++-header -o build/.objs/hello/linux/x86_64/release/src/cxx/test.h.gch src/test.h
[  2%]: <hello> generating.module.deps src/main.cpp
/usr/bin/gcc -E -x c++ -m64 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -I build/.objs/hello/linux/x86_64/release/src/cxx -include test.h -D_GLIBCXX_USE_CXX11_ABI=0 -DNDEBUG src/main.cpp -o build/.gens/hello/linux/x86_64/release/rules/bmi/cache/modules/2d9b7c0b/main.cpp.i
[  3%]: <hello> generating.module.deps src/hello.mpp
/usr/bin/gcc -E -x c++ -m64 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -I build/.objs/hello/linux/x86_64/release/src/cxx -include test.h -D_GLIBCXX_USE_CXX11_ABI=0 -DNDEBUG src/hello.mpp -o build/.gens/hello/linux/x86_64/release/rules/bmi/cache/modules/2d9b7c0b/hello.mpp.i
[ 25%]: <hello> compiling.module.release hello
/usr/bin/gcc -c -m64 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -I build/.objs/hello/linux/x86_64/release/src/cxx -include test.h -fmodules-ts -D_GLIBCXX_USE_CXX11_ABI=0 -DNDEBUG -fmodule-mapper=/tmp/.xmake999/250406/hello/src/hello.mpp -x c++ -o build/.objs/hello/linux/x86_64/release/src/hello.mpp.o src/hello.mpp
error: src/hello.mpp:1:1: error: module-declaration only permitted as first declaration, or ending a global module fragment
    1 | module;
      | ^~~~~~
src/hello.mpp:4:8: error: module-declaration only permitted as first declaration, or ending a global module fragment
    4 | export module hello;
      |        ^~~~~~
src/hello.mpp:6:1: error: 'export' may only occur after a module interface declaration
    6 | export namespace hello {
      | ^~~~~~

well, modules and headerunits are glorified pch, so i don't think it's a real use case anyway

Arthapz

lgtm

Arthapz · 2025-04-07T14:58:46Z

i have a PR that touch a lot module support, should i wait for this to be merge and update this PR or rebase on this PR and wait for this to merge ?

waruqi · 2025-04-07T15:23:11Z

i have a PR that touch a lot module support, should i wait for this to be merge and update this PR or rebase on this PR and wait for this to merge ?

you can open a pr to this prepare branch. the current dev branch will be released in these days.

Arthapz · 2025-04-08T11:24:50Z

why only some of the jobs are actually ran by the jobgraph ?

local scanjob = group .. "/scan_module_dependencies"
jobgraph:add(scanjob, function()
      for _, sourcefile in ipairs(sourcebatch.sourcefiles) do
          local fileconfig = target:fileconfig(sourcefile)
          local external = fileconfig and fileconfig.external

          local jobname = group .. "/scan/" .. sourcefile
          jobgraph:add(jobname, function(_, _, opt)
              local changed = _scanner(target).scan_dependency_for(target, sourcefile, opt)
              if changed then
                  support.memcache():set2(target:fullname(), "modules.changed", true)
              end
          end, {groups = {group .. "/scan"}})
          jobgraph:add_orders(jobname, parsejob)
      end
      print(jobgraph:jobs())
end, {groups = {group}})
jobgraph:add_orders(scanjob, parsejob)

...
 "CppUtils/modules/scan/modules\Language\TreeParser.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Language\TreeParser.mpp"
  },
  "CppUtils/modules/scan/modules\Network\Server.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Network\Server.mpp"
  },
  "CppUtils/modules/scan/modules\Math\Float.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Math\Float.mpp"
  },
  "CppUtils/modules/scan/modules\FileSystem\Directory.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\FileSystem\Directory.mpp"
  },
  "CppUtils/modules/scan/modules\String\Hash.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\String\Hash.mpp"
  },
  "CppUtils/modules/scan/modules\Log\LogRotate.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Log\LogRotate.mpp"
  },
  "CppUtils/modules/scan/modules\Type\Variant.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Type\Variant.mpp"
  },
  "CppUtils/modules/scan/modules\Container\Size2d.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Container\Size2d.mpp"
  },
  "CppUtils/modules/scan/modules\Container\NetworkPtr.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Container\NetworkPtr.mpp"
  },
  "CppUtils/modules/scan/modules\Container\Vector.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Container\Vector.mpp"
  },
  "CppUtils/modules/scan/modules\Network\Client.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Network\Client.mpp"
  },
  "CppUtils/modules/scan/modules\Math\Math.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Math\Math.mpp"
  },
  "CppUtils/modules/scan/modules\Network\Network.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Network\Network.mpp"
  },
  "CppUtils/modules/scan/modules\Container\BTree.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Container\BTree.mpp"
  },
  "CppUtils/end_prepare" = {
    run = function @programdir\modules\private\action\build\target.lua:294,
    name = "CppUtils/end_prepare"
  },
  "CppUtils/modules/scan/modules\String\HashTable.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\String\HashTable.mpp"
  },
  "CppUtils/modules/scan_module_dependencies" = {
    run = function @programdir\rules\c++\modules\scanner.lua:386,
    name = "CppUtils/modules/scan_module_dependencies"
  },
  "CppUtils/modules/scan/modules\Container\Stack.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Container\Stack.mpp"
  }
}

[  1%]: <CppUtils> generating.module.deps modules\Chrono\Chrono.mpp
[  2%]: <CppUtils> generating.module.deps modules\Chrono\Chronometer.mpp
[  3%]: <CppUtils> generating.module.deps modules\Chrono\Concept.mpp
[  3%]: <CppUtils> generating.module.deps modules\Container\BidirectionalMap.mpp
[  4%]: <CppUtils> generating.module.deps modules\Container\BTree.mpp
[  5%]: <CppUtils> generating.module.deps modules\Container\Container.mpp
[ 38%]: compiling.release tests\main.cpp
error: tests\main.cpp:1:8: fatal error: module 'std' not found
    1 | import std;
      | ~~~~~~~^~~
1 error generated.

waruqi · 2025-04-08T13:19:45Z

why only some of the jobs are actually ran by the jobgraph ?

local scanjob = group .. "/scan_module_dependencies"
jobgraph:add(scanjob, function()
      for _, sourcefile in ipairs(sourcebatch.sourcefiles) do
          local fileconfig = target:fileconfig(sourcefile)
          local external = fileconfig and fileconfig.external

          local jobname = group .. "/scan/" .. sourcefile
          jobgraph:add(jobname, function(_, _, opt)
              local changed = _scanner(target).scan_dependency_for(target, sourcefile, opt)
              if changed then
                  support.memcache():set2(target:fullname(), "modules.changed", true)
              end
          end, {groups = {group .. "/scan"}})
          jobgraph:add_orders(jobname, parsejob)
      end
      print(jobgraph:jobs())
end, {groups = {group}})
jobgraph:add_orders(scanjob, parsejob)

...
 "CppUtils/modules/scan/modules\Language\TreeParser.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Language\TreeParser.mpp"
  },
  "CppUtils/modules/scan/modules\Network\Server.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Network\Server.mpp"
  },
  "CppUtils/modules/scan/modules\Math\Float.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Math\Float.mpp"
  },
  "CppUtils/modules/scan/modules\FileSystem\Directory.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\FileSystem\Directory.mpp"
  },
  "CppUtils/modules/scan/modules\String\Hash.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\String\Hash.mpp"
  },
  "CppUtils/modules/scan/modules\Log\LogRotate.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Log\LogRotate.mpp"
  },
  "CppUtils/modules/scan/modules\Type\Variant.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Type\Variant.mpp"
  },
  "CppUtils/modules/scan/modules\Container\Size2d.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Container\Size2d.mpp"
  },
  "CppUtils/modules/scan/modules\Container\NetworkPtr.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Container\NetworkPtr.mpp"
  },
  "CppUtils/modules/scan/modules\Container\Vector.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Container\Vector.mpp"
  },
  "CppUtils/modules/scan/modules\Network\Client.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Network\Client.mpp"
  },
  "CppUtils/modules/scan/modules\Math\Math.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Math\Math.mpp"
  },
  "CppUtils/modules/scan/modules\Network\Network.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Network\Network.mpp"
  },
  "CppUtils/modules/scan/modules\Container\BTree.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Container\BTree.mpp"
  },
  "CppUtils/end_prepare" = {
    run = function @programdir\modules\private\action\build\target.lua:294,
    name = "CppUtils/end_prepare"
  },
  "CppUtils/modules/scan/modules\String\HashTable.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\String\HashTable.mpp"
  },
  "CppUtils/modules/scan_module_dependencies" = {
    run = function @programdir\rules\c++\modules\scanner.lua:386,
    name = "CppUtils/modules/scan_module_dependencies"
  },
  "CppUtils/modules/scan/modules\Container\Stack.mpp" = {
    run = function @programdir\rules\c++\modules\scanner.lua:404,
    name = "CppUtils/modules/scan/modules\Container\Stack.mpp"
  }
}

[  1%]: <CppUtils> generating.module.deps modules\Chrono\Chrono.mpp
[  2%]: <CppUtils> generating.module.deps modules\Chrono\Chronometer.mpp
[  3%]: <CppUtils> generating.module.deps modules\Chrono\Concept.mpp
[  3%]: <CppUtils> generating.module.deps modules\Container\BidirectionalMap.mpp
[  4%]: <CppUtils> generating.module.deps modules\Container\BTree.mpp
[  5%]: <CppUtils> generating.module.deps modules\Container\Container.mpp
[ 38%]: compiling.release tests\main.cpp
error: tests\main.cpp:1:8: fatal error: module 'std' not found
    1 | import std;
      | ~~~~~~~^~~
1 error generated.

use group jobgraph:add(scanjob, -> jobgraph:group(scanjob,

waruqi · 2025-04-08T15:07:37Z

@Arthapz try

local scangroup = group .. "/scan_module_dependencies"
jobgraph:group(scangroup, function()
      for _, sourcefile in ipairs(sourcebatch.sourcefiles) do
          local fileconfig = target:fileconfig(sourcefile)
          local external = fileconfig and fileconfig.external
          local jobname = group .. "/scan/" .. sourcefile
          jobgraph:add(jobname, function(_, _, opt)
              local changed = _scanner(target).scan_dependency_for(target, sourcefile, opt)
              if changed then
                  support.memcache():set2(target:fullname(), "modules.changed", true)
              end
          end)
      end
end)
jobgraph:add_orders(scangroup, parsejob)

Arthapz · 2025-04-09T01:09:04Z

@Arthapz try

local scangroup = group .. "/scan_module_dependencies"
jobgraph:group(scangroup, function()
      for _, sourcefile in ipairs(sourcebatch.sourcefiles) do
          local fileconfig = target:fileconfig(sourcefile)
          local external = fileconfig and fileconfig.external
          local jobname = group .. "/scan/" .. sourcefile
          jobgraph:add(jobname, function(_, _, opt)
              local changed = _scanner(target).scan_dependency_for(target, sourcefile, opt)
              if changed then
                  support.memcache():set2(target:fullname(), "modules.changed", true)
              end
          end)
      end
end)
jobgraph:add_orders(scangroup, parsejob)

it work thx

waruqi · 2025-04-09T14:10:29Z

@Arthapz I have merged it, you can open a pr to dev later.

Arthapz · 2025-04-10T06:31:27Z

@Arthapz I have merged it, you can open a pr to dev later.

Okay

waruqi added this to the v2.9.9 milestone Mar 11, 2025

waruqi force-pushed the prepare branch 3 times, most recently from d77bb37 to 5fa6d4a Compare March 23, 2025 13:47

waruqi changed the title ~~add on_prepare stages~~ Add build jobgraph support Mar 27, 2025

waruqi force-pushed the prepare branch 2 times, most recently from 2de8920 to 6dd8d89 Compare April 3, 2025 01:00

waruqi mentioned this pull request Apr 3, 2025

feat(verilator): Support Verilator target build to shared library #6281

Merged

waruqi force-pushed the prepare branch from 6dd8d89 to 8f1ed36 Compare April 3, 2025 02:05

waruqi marked this pull request as ready for review April 6, 2025 05:16

waruqi requested a review from Arthapz April 6, 2025 14:55

Arthapz approved these changes Apr 6, 2025

View reviewed changes

waruqi mentioned this pull request Apr 7, 2025

Improve rule API and build dependency order #6202

Closed

14 tasks

waruqi force-pushed the prepare branch from 9929f67 to 7b0a0e2 Compare April 7, 2025 14:31

waruqi modified the milestones: v2.9.9, v3.0.0 Apr 7, 2025

waruqi force-pushed the prepare branch 2 times, most recently from b98d617 to b595022 Compare April 8, 2025 02:02

waruqi added 8 commits April 8, 2025 15:31

add on_prepare stages

3d00c91

add missing target config

a1d0e9c

add prepare stub

3a813e0

add jobgraph stub

00ebba3

add jobgraph.new

1621db0

improve runjobs tests

6785ce5

improve jobgraph tests

f8346ab

add build job queue stub

08fade0

waruqi added 11 commits April 8, 2025 15:31

fix pcheader for objc

162f554

disable module and pch for gcc

b2d2727

Delete tests/projects/c++/modules/hello_with_pch/test.lua

72870df

remove across_targets_in_parallel policy

c091d7b

add warings tip

279801b

improve build.fence wrap

3ebc4d1

fix distcc

07ac1c4

move target files

6fc28a7

prepare targets in cmakelists generator

f72d519

prepare targets for more generators

abd58c9

improve verilator rules

bd6f305

waruqi force-pushed the prepare branch from b595022 to bd6f305 Compare April 8, 2025 07:32

compatible with order deps

117ae68

waruqi merged commit 48275f8 into dev Apr 9, 2025
44 of 46 checks passed

waruqi deleted the prepare branch April 9, 2025 14:09

This was referenced Apr 9, 2025

v3.0 Roadmap #5527

Closed

c++ modules 和预编译头文件冲突问题 #4051

Closed

waruqi added 7 commits April 9, 2025 23:01

fix target_utils

7b4ba90

rename to target_buildutils

811f877

improve target_cmds

77d90d2

ignore rules and stages

4109abc

fix link jobs

95e69ee

improve makefile generator

260f071

move rule_group to deprecated

699e40f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add build jobgraph support#6209

Add build jobgraph support#6209
waruqi merged 127 commits intodevfrom
prepare

waruqi commented Mar 11, 2025

Uh oh!

waruqi commented Apr 6, 2025

Uh oh!

Arthapz commented Apr 6, 2025 •

edited

Loading

Uh oh!

Arthapz left a comment

Uh oh!

Arthapz commented Apr 7, 2025

Uh oh!

waruqi commented Apr 7, 2025

Uh oh!

Arthapz commented Apr 8, 2025 •

edited

Loading

Uh oh!

waruqi commented Apr 8, 2025

Uh oh!

waruqi commented Apr 8, 2025

Uh oh!

Arthapz commented Apr 9, 2025 •

edited

Loading

Uh oh!

Uh oh!

waruqi commented Apr 9, 2025

Uh oh!

Arthapz commented Apr 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

waruqi commented Mar 11, 2025

Uh oh!

waruqi commented Apr 6, 2025

Uh oh!

Arthapz commented Apr 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Arthapz left a comment

Choose a reason for hiding this comment

Uh oh!

Arthapz commented Apr 7, 2025

Uh oh!

waruqi commented Apr 7, 2025

Uh oh!

Arthapz commented Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

waruqi commented Apr 8, 2025

Uh oh!

waruqi commented Apr 8, 2025

Uh oh!

Arthapz commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

waruqi commented Apr 9, 2025

Uh oh!

Arthapz commented Apr 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Arthapz commented Apr 6, 2025 •

edited

Loading

Arthapz commented Apr 8, 2025 •

edited

Loading

Arthapz commented Apr 9, 2025 •

edited

Loading