Skip to content

refactor: incremental rebuild make failed modules and dependencies#9393

Merged
jerrykingxyz merged 3 commits intomainfrom
jerry/make
Feb 20, 2025
Merged

refactor: incremental rebuild make failed modules and dependencies#9393
jerrykingxyz merged 3 commits intomainfrom
jerry/make

Conversation

@jerrykingxyz
Copy link
Contributor

@jerrykingxyz jerrykingxyz commented Feb 19, 2025

Summary

  1. Make will not always rebuild all of failed module and failed dependencies when recompile. The MakeArtifact will save make_failed_module and make_failed_dependencies as statistical data, which are only used to find failed module and failed dependencies as indexes.
struct MakeArtifact {
// data
-  pub make_failed_module: IdentifierSet,
-  pub make_failed_dependencies: HashSet<DependencyId>,
// statistical data
+  pub make_failed_module: IdentifierSet,
+  pub make_failed_dependencies: HashSet<DependencyId>,
...
}
  1. Add factorize_info to all of ModuleDependency and ContextDependency, and FactorizeInfo will save *_dependencies and diagnostics like Module.BuildInfo (PS. no need to save those info if factorize success).
pub enum FactorizeInfo {
  Success,
  Failed {
    related_dep_ids: Vec<DependencyId>,
    file_dependencies: HashSet<ArcPath>,
    context_dependencies: HashSet<ArcPath>,
    missing_dependencies: HashSet<ArcPath>,
    #[cacheable(with=Skip)]
    diagnostics: Vec<Diagnostic>,
  },
}

trait ModuleContext {
  ..
  fn factorize_info(&self) -> &FactorizeInfo;
  fn factorize_info_mut(&mut self) -> &mut FactorizeInfo;
}
  1. Modified and deleted files will rebuild the affected make_failed_dependencies instead of all make_failed_dependencies.
impl Cutout {
  fn cutout_artifact(..) {
    match param {
        ..
        MakeParam::RemovedFiles(files) {
          ..
          for dep_id in &artifact.make_failed_dependencies {
            let dep = module_graph
              .dependency_by_id(dep_id)
              .expect("should have dependency");
            let info = FactorizeInfo::get_from(dep).expect("should have factorize info");
            if info.depends_on(&files) {
              force_build_deps.insert(*dep_id);
            }
          }
        }
    }
  }
}

Checklist

  • Tests updated (or not required).
  • Documentation updated (or not required).

@github-actions github-actions bot added the team The issue/pr is created by the member of Rspack. label Feb 19, 2025
@netlify
Copy link

netlify bot commented Feb 19, 2025

Deploy Preview for rspack canceled.

Name Link
🔨 Latest commit 4542440
🔍 Latest deploy log https://app.netlify.com/sites/rspack/deploys/67b6edbdc783d400084fb99a

@github-actions
Copy link
Contributor

github-actions bot commented Feb 19, 2025

📝 Benchmark detail: Open

Name Base (2025-02-19 a81bd20) Current Change
10000_big_production-mode_disable-minimize + exec 36.9 s ± 476 ms 37.7 s ± 1.22 s +2.17 %
10000_development-mode + exec 1.76 s ± 20 ms 1.7 s ± 18 ms -3.82 %
10000_development-mode_hmr + exec 672 ms ± 7 ms 663 ms ± 6.6 ms -1.35 %
10000_production-mode + exec 2.23 s ± 54 ms 2.19 s ± 145 ms -1.61 %
10000_production-mode_persistent-cold + exec 2.37 s ± 98 ms 2.3 s ± 33 ms -2.77 %
10000_production-mode_persistent-hot + exec 1.63 s ± 48 ms 1.62 s ± 36 ms -0.94 %
arco-pro_development-mode + exec 1.76 s ± 60 ms 1.72 s ± 51 ms -2.30 %
arco-pro_development-mode_hmr + exec 386 ms ± 3.5 ms 376 ms ± 1.9 ms -2.50 %
arco-pro_production-mode + exec 3.67 s ± 70 ms 3.53 s ± 170 ms -3.84 %
arco-pro_production-mode_generate-package-json-webpack-plugin + exec 3.66 s ± 258 ms 3.6 s ± 282 ms -1.76 %
arco-pro_production-mode_persistent-cold + exec 3.71 s ± 205 ms 3.57 s ± 96 ms -3.99 %
arco-pro_production-mode_persistent-hot + exec 2.3 s ± 79 ms 2.36 s ± 241 ms +2.77 %
arco-pro_production-mode_traverse-chunk-modules + exec 3.56 s ± 131 ms 3.52 s ± 154 ms -1.03 %
large-dyn-imports_development-mode + exec 1.99 s ± 47 ms 1.96 s ± 61 ms -1.42 %
large-dyn-imports_production-mode + exec 2.04 s ± 31 ms 2.04 s ± 38 ms -0.30 %
threejs_development-mode_10x + exec 1.52 s ± 7.6 ms 1.53 s ± 17 ms +0.70 %
threejs_development-mode_10x_hmr + exec 774 ms ± 5.1 ms 791 ms ± 40 ms +2.30 %
threejs_production-mode_10x + exec 5.15 s ± 62 ms 5.21 s ± 349 ms +1.06 %
threejs_production-mode_10x_persistent-cold + exec 5.2 s ± 92 ms 5.18 s ± 36 ms -0.35 %
threejs_production-mode_10x_persistent-hot + exec 4.44 s ± 201 ms 4.42 s ± 55 ms -0.46 %
10000_big_production-mode_disable-minimize + rss memory 8666 MiB ± 41 MiB 8673 MiB ± 53.5 MiB +0.08 %
10000_development-mode + rss memory 647 MiB ± 23.7 MiB 681 MiB ± 19 MiB +5.30 %
10000_development-mode_hmr + rss memory 1326 MiB ± 157 MiB 1334 MiB ± 174 MiB +0.60 %
10000_production-mode + rss memory 617 MiB ± 8.92 MiB 661 MiB ± 31.4 MiB +7.13 %
10000_production-mode_persistent-cold + rss memory 724 MiB ± 23 MiB 760 MiB ± 31.7 MiB +5.00 %
10000_production-mode_persistent-hot + rss memory 690 MiB ± 28.4 MiB 734 MiB ± 22.7 MiB +6.44 %
arco-pro_development-mode + rss memory 566 MiB ± 42.3 MiB 593 MiB ± 39.3 MiB +4.75 %
arco-pro_development-mode_hmr + rss memory 654 MiB ± 47.4 MiB 662 MiB ± 50.6 MiB +1.22 %
arco-pro_production-mode + rss memory 716 MiB ± 21.3 MiB 733 MiB ± 25.4 MiB +2.28 %
arco-pro_production-mode_generate-package-json-webpack-plugin + rss memory 729 MiB ± 38.1 MiB 719 MiB ± 19.7 MiB -1.30 %
arco-pro_production-mode_persistent-cold + rss memory 798 MiB ± 61 MiB 790 MiB ± 26.1 MiB -0.98 %
arco-pro_production-mode_persistent-hot + rss memory 646 MiB ± 20.1 MiB 667 MiB ± 30.1 MiB +3.34 %
arco-pro_production-mode_traverse-chunk-modules + rss memory 724 MiB ± 42.3 MiB 722 MiB ± 27.8 MiB -0.18 %
large-dyn-imports_development-mode + rss memory 636 MiB ± 1.7 MiB 652 MiB ± 10 MiB +2.53 %
large-dyn-imports_production-mode + rss memory 519 MiB ± 4.18 MiB 526 MiB ± 9.64 MiB +1.41 %
threejs_development-mode_10x + rss memory 550 MiB ± 14.7 MiB 558 MiB ± 10.1 MiB +1.36 %
threejs_development-mode_10x_hmr + rss memory 1138 MiB ± 114 MiB 1098 MiB ± 219 MiB -3.51 %
threejs_production-mode_10x + rss memory 837 MiB ± 31.6 MiB 845 MiB ± 5.72 MiB +0.98 %
threejs_production-mode_10x_persistent-cold + rss memory 930 MiB ± 31.9 MiB 924 MiB ± 67.7 MiB -0.64 %
threejs_production-mode_10x_persistent-hot + rss memory 794 MiB ± 26.3 MiB 817 MiB ± 35.9 MiB +2.84 %

@codspeed-hq
Copy link

codspeed-hq bot commented Feb 19, 2025

CodSpeed Performance Report

Merging #9393 will not alter performance

Comparing jerry/make (4542440) with main (b46f13c)

🎉 Hooray! codspeed-rust just leveled up to 2.7.2!

A heads-up, this is a breaking change and it might affect your current performance baseline a bit. But here's the exciting part - it's packed with new, cool features and promises improved result stability 🥳!
Curious about what's new? Visit our releases page to delve into all the awesome details about this new version.

Summary

✅ 7 untouched benchmarks

@github-actions
Copy link
Contributor

github-actions bot commented Feb 20, 2025

📝 Benchmark detail: Open

Name Base (2025-02-20 2101c1b) Current Change
10000_big_production-mode_disable-minimize + exec 37.5 s ± 434 ms 37.5 s ± 830 ms -0.04 %
10000_development-mode + exec 1.73 s ± 37 ms 1.69 s ± 7 ms -2.24 %
10000_development-mode_hmr + exec 674 ms ± 4.7 ms 671 ms ± 21 ms -0.49 %
10000_production-mode + exec 2.21 s ± 43 ms 2.16 s ± 53 ms -2.13 %
10000_production-mode_persistent-cold + exec 2.35 s ± 168 ms 2.34 s ± 80 ms -0.45 %
10000_production-mode_persistent-hot + exec 1.59 s ± 22 ms 1.63 s ± 68 ms +2.08 %
arco-pro_development-mode + exec 1.75 s ± 105 ms 1.76 s ± 137 ms +0.34 %
arco-pro_development-mode_hmr + exec 385 ms ± 3.1 ms 375 ms ± 0.55 ms -2.61 %
arco-pro_production-mode + exec 3.57 s ± 201 ms 3.52 s ± 140 ms -1.33 %
arco-pro_production-mode_generate-package-json-webpack-plugin + exec 3.65 s ± 103 ms 3.61 s ± 217 ms -0.95 %
arco-pro_production-mode_persistent-cold + exec 3.69 s ± 248 ms 3.57 s ± 198 ms -3.28 %
arco-pro_production-mode_persistent-hot + exec 2.33 s ± 131 ms 2.31 s ± 153 ms -0.98 %
arco-pro_production-mode_traverse-chunk-modules + exec 3.58 s ± 113 ms 3.58 s ± 145 ms -0.26 %
large-dyn-imports_development-mode + exec 1.98 s ± 26 ms 1.96 s ± 34 ms -0.90 %
large-dyn-imports_production-mode + exec 2.04 s ± 53 ms 2.04 s ± 129 ms -0.04 %
threejs_development-mode_10x + exec 1.51 s ± 24 ms 1.56 s ± 43 ms +2.65 %
threejs_development-mode_10x_hmr + exec 776 ms ± 34 ms 811 ms ± 25 ms +4.56 %
threejs_production-mode_10x + exec 5.2 s ± 394 ms 5.23 s ± 176 ms +0.58 %
threejs_production-mode_10x_persistent-cold + exec 5.27 s ± 332 ms 5.24 s ± 98 ms -0.58 %
threejs_production-mode_10x_persistent-hot + exec 4.44 s ± 275 ms 4.44 s ± 44 ms 0.00 %
10000_big_production-mode_disable-minimize + rss memory 8671 MiB ± 16.8 MiB 8692 MiB ± 54.1 MiB +0.23 %
10000_development-mode + rss memory 658 MiB ± 28.2 MiB 654 MiB ± 19.7 MiB -0.60 %
10000_development-mode_hmr + rss memory 1266 MiB ± 241 MiB 1253 MiB ± 282 MiB -1.03 %
10000_production-mode + rss memory 645 MiB ± 23.5 MiB 629 MiB ± 27.6 MiB -2.40 %
10000_production-mode_persistent-cold + rss memory 736 MiB ± 20.5 MiB 733 MiB ± 30.1 MiB -0.33 %
10000_production-mode_persistent-hot + rss memory 728 MiB ± 27.3 MiB 734 MiB ± 22.2 MiB +0.84 %
arco-pro_development-mode + rss memory 580 MiB ± 34.2 MiB 571 MiB ± 33.3 MiB -1.46 %
arco-pro_development-mode_hmr + rss memory 648 MiB ± 49.5 MiB 639 MiB ± 55.6 MiB -1.40 %
arco-pro_production-mode + rss memory 730 MiB ± 44.8 MiB 715 MiB ± 30.4 MiB -2.12 %
arco-pro_production-mode_generate-package-json-webpack-plugin + rss memory 740 MiB ± 22.9 MiB 722 MiB ± 18 MiB -2.43 %
arco-pro_production-mode_persistent-cold + rss memory 811 MiB ± 34.9 MiB 792 MiB ± 43.9 MiB -2.38 %
arco-pro_production-mode_persistent-hot + rss memory 671 MiB ± 24 MiB 669 MiB ± 34.1 MiB -0.40 %
arco-pro_production-mode_traverse-chunk-modules + rss memory 736 MiB ± 26.8 MiB 707 MiB ± 28.3 MiB -3.93 %
large-dyn-imports_development-mode + rss memory 652 MiB ± 3.22 MiB 644 MiB ± 4.55 MiB -1.16 %
large-dyn-imports_production-mode + rss memory 537 MiB ± 9.03 MiB 524 MiB ± 8.62 MiB -2.48 %
threejs_development-mode_10x + rss memory 550 MiB ± 25.2 MiB 557 MiB ± 8.16 MiB +1.29 %
threejs_development-mode_10x_hmr + rss memory 1091 MiB ± 142 MiB 1154 MiB ± 47.9 MiB +5.78 %
threejs_production-mode_10x + rss memory 825 MiB ± 47.2 MiB 837 MiB ± 14.2 MiB +1.45 %
threejs_production-mode_10x_persistent-cold + rss memory 931 MiB ± 39.6 MiB 939 MiB ± 41.1 MiB +0.87 %
threejs_production-mode_10x_persistent-hot + rss memory 811 MiB ± 46.3 MiB 808 MiB ± 29.2 MiB -0.39 %

@jerrykingxyz jerrykingxyz changed the title refactor: make failed modules and dependencies support incremental refactor: incremental rebuild make failed modules and dependencies Feb 20, 2025
@jerrykingxyz jerrykingxyz enabled auto-merge (squash) February 20, 2025 09:39
@jerrykingxyz jerrykingxyz merged commit 6c8b2c4 into main Feb 20, 2025
31 checks passed
@jerrykingxyz jerrykingxyz deleted the jerry/make branch February 20, 2025 10:37
@h-a-n-a h-a-n-a mentioned this pull request Feb 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

team The issue/pr is created by the member of Rspack.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants