Skip to content

add compile cache support to wd_?s_bundle, enable for node#3044

Merged
mikea merged 2 commits intomainfrom
maizatskyi/2024-11-01-node-compile-caches
Nov 12, 2024
Merged

add compile cache support to wd_?s_bundle, enable for node#3044
mikea merged 2 commits intomainfrom
maizatskyi/2024-11-01-node-compile-caches

Conversation

@mikea
Copy link
Contributor

@mikea mikea commented Nov 1, 2024 •
edited
Loading

Extracted from #3004, not used on load yet.

downstream pr 9121

@mikea mikea requested review from anonrig and jasnell November 1, 2024 21:56
@mikea mikea requested review from a team as code owners November 1, 2024 21:56
@mikea mikea requested a review from a team November 1, 2024 21:56
@mikea
Copy link
Contributor Author

mikea commented Nov 1, 2024 

$ cat bazel-bin/src/node/node.capnp
@0xbcc8f57c63814005;

# generated by @workerd//build/wd_js_bundle.bzl

using Modules = import "/workerd/jsg/modules.capnp";

const nodeBundle :Modules.Bundle = (
  modules = [
    (name = "node:_stream_duplex", src = embed "node__stream_duplex", type = builtin, compileCache = embed "node__stream_duplex_cache", ),
    (name = "node:_stream_passthrough", src = embed "node__stream_passthrough", type = builtin, compileCache = embed "node__stream_passthrough_cache", ),
    (name = "node:_stream_readable", src = embed "node__stream_readable", type = builtin, compileCache = embed "node__stream_readable_cache", ),

.....

@mikea
Copy link
Contributor Author

mikea commented Nov 1, 2024 

$ du -b bazel-out/k8-fastbuild/bin/src/node/*_cache
552     bazel-out/k8-fastbuild/bin/src/node/node_assert_cache
560     bazel-out/k8-fastbuild/bin/src/node/node_assert_strict_cache
744     bazel-out/k8-fastbuild/bin/src/node/node_async_hooks_cache
1824    bazel-out/k8-fastbuild/bin/src/node/node_buffer_cache
6424    bazel-out/k8-fastbuild/bin/src/node/node_crypto_cache
3712    bazel-out/k8-fastbuild/bin/src/node/node_diagnostics_channel_cache
456     bazel-out/k8-fastbuild/bin/src/node/node_events_cache
3248    bazel-out/k8-fastbuild/bin/src/node/node-internal_constants_cache
3264    bazel-out/k8-fastbuild/bin/src/node/node-internal_crypto_dh_cache
3328    bazel-out/k8-fastbuild/bin/src/node/node-internal_crypto_hash_cache
1960    bazel-out/k8-fastbuild/bin/src/node/node-internal_crypto_hkdf_cache
5456    bazel-out/k8-fastbuild/bin/src/node/node-internal_crypto_keys_cache
1288    bazel-out/k8-fastbuild/bin/src/node/node-internal_crypto_pbkdf2_cache
3304    bazel-out/k8-fastbuild/bin/src/node/node-internal_crypto_random_cache
1584    bazel-out/k8-fastbuild/bin/src/node/node-internal_crypto_scrypt_cache
1312    bazel-out/k8-fastbuild/bin/src/node/node-internal_crypto_spkac_cache
1872    bazel-out/k8-fastbuild/bin/src/node/node-internal_crypto_util_cache
4896    bazel-out/k8-fastbuild/bin/src/node/node-internal_crypto_x509_cache
872     bazel-out/k8-fastbuild/bin/src/node/node-internal_debuglog_cache
8608    bazel-out/k8-fastbuild/bin/src/node/node-internal_events_cache
4528    bazel-out/k8-fastbuild/bin/src/node/node-internal_internal_assert_cache
2616    bazel-out/k8-fastbuild/bin/src/node/node-internal_internal_assertionerror_cache
18016   bazel-out/k8-fastbuild/bin/src/node/node-internal_internal_buffer_cache
3568    bazel-out/k8-fastbuild/bin/src/node/node-internal_internal_comparisons_cache
1408    bazel-out/k8-fastbuild/bin/src/node/node-internal_internal_diffs_cache
13704   bazel-out/k8-fastbuild/bin/src/node/node-internal_internal_errors_cache
18512   bazel-out/k8-fastbuild/bin/src/node/node-internal_internal_inspect_cache
3936    bazel-out/k8-fastbuild/bin/src/node/node-internal_internal_path_cache
4936    bazel-out/k8-fastbuild/bin/src/node/node-internal_internal_querystring_cache
2400    bazel-out/k8-fastbuild/bin/src/node/node-internal_internal_stringdecoder_cache
5072    bazel-out/k8-fastbuild/bin/src/node/node-internal_internal_types_cache
3768    bazel-out/k8-fastbuild/bin/src/node/node-internal_internal_url_cache
1856    bazel-out/k8-fastbuild/bin/src/node/node-internal_internal_utils_cache
8440    bazel-out/k8-fastbuild/bin/src/node/node-internal_internal_zlib_base_cache
7640    bazel-out/k8-fastbuild/bin/src/node/node-internal_internal_zlib_cache
1904    bazel-out/k8-fastbuild/bin/src/node/node-internal_internal_zlib_constants_cache
5872    bazel-out/k8-fastbuild/bin/src/node/node-internal_legacy_url_cache
4168    bazel-out/k8-fastbuild/bin/src/node/node-internal_mock_cache
1528    bazel-out/k8-fastbuild/bin/src/node/node-internal_process_cache
3456    bazel-out/k8-fastbuild/bin/src/node/node-internal_streams_adapters_cache
1656    bazel-out/k8-fastbuild/bin/src/node/node-internal_streams_compose_cache
4280    bazel-out/k8-fastbuild/bin/src/node/node-internal_streams_duplex_cache
1360    bazel-out/k8-fastbuild/bin/src/node/node-internal_streams_legacy_cache
2920    bazel-out/k8-fastbuild/bin/src/node/node-internal_streams_pipeline_cache
1016    bazel-out/k8-fastbuild/bin/src/node/node-internal_streams_promises_cache
13928   bazel-out/k8-fastbuild/bin/src/node/node-internal_streams_readable_cache
1896    bazel-out/k8-fastbuild/bin/src/node/node-internal_streams_transform_cache
8768    bazel-out/k8-fastbuild/bin/src/node/node-internal_streams_util_cache
8016    bazel-out/k8-fastbuild/bin/src/node/node-internal_streams_writable_cache
3320    bazel-out/k8-fastbuild/bin/src/node/node-internal_validators_cache
2552    bazel-out/k8-fastbuild/bin/src/node/node_module_cache
1440    bazel-out/k8-fastbuild/bin/src/node/node_path_cache
1456    bazel-out/k8-fastbuild/bin/src/node/node_path_posix_cache
1456    bazel-out/k8-fastbuild/bin/src/node/node_path_win32_cache
456     bazel-out/k8-fastbuild/bin/src/node/node_process_cache
1152    bazel-out/k8-fastbuild/bin/src/node/node_querystring_cache
3064    bazel-out/k8-fastbuild/bin/src/node/node_stream_cache
1176    bazel-out/k8-fastbuild/bin/src/node/node_stream_consumers_cache
784     bazel-out/k8-fastbuild/bin/src/node/node__stream_duplex_cache
592     bazel-out/k8-fastbuild/bin/src/node/node__stream_passthrough_cache
576     bazel-out/k8-fastbuild/bin/src/node/node_stream_promises_cache
960     bazel-out/k8-fastbuild/bin/src/node/node__stream_readable_cache
592     bazel-out/k8-fastbuild/bin/src/node/node__stream_transform_cache
2000    bazel-out/k8-fastbuild/bin/src/node/node_stream_web_cache
824     bazel-out/k8-fastbuild/bin/src/node/node__stream_writable_cache
640     bazel-out/k8-fastbuild/bin/src/node/node_string_decoder_cache
784     bazel-out/k8-fastbuild/bin/src/node/node_test_cache
2048    bazel-out/k8-fastbuild/bin/src/node/node_url_cache
4952    bazel-out/k8-fastbuild/bin/src/node/node_util_cache
2688    bazel-out/k8-fastbuild/bin/src/node/node_util_types_cache
13184   bazel-out/k8-fastbuild/bin/src/node/node_zlib_cache

@mikea mikea force-pushed the maizatskyi/2024-11-01-node-compile-caches branch from 06279dd to 5296fcb Compare November 1, 2024 22:12
anonrig
anonrig reviewed Nov 2, 2024
View reviewed changes
Copy link
Member

@anonrig anonrig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't also enable it for consumption on load right? I think we should do it on the same PR.

@mikea
Copy link
Contributor Author

mikea commented Nov 4, 2024

This doesn't also enable it for consumption on load right? I think we should do it on the same PR.

why? This thing is brittle so I'd like to land it together with EW side. Also as of now this is a no-op and has very low (0?) risk in production. Load is different and has to be treated differently. Makes all sense to me to break this down in separete PRs.

@mikea mikea force-pushed the maizatskyi/2024-11-01-node-compile-caches branch from 5296fcb to c3095e4 Compare November 4, 2024 18:27
@mikea
Copy link
Contributor Author

mikea commented Nov 8, 2024

Unfortunately this is not landable now: as predicted there are issues with cross-compilation downstream. I'll need to massage the run rule to support running under qemu.

@mikea mikea force-pushed the maizatskyi/2024-11-01-node-compile-caches branch from c3095e4 to 72ac271 Compare November 12, 2024 19:41
@mikea mikea force-pushed the maizatskyi/2024-11-01-node-compile-caches branch from 72ac271 to a3455a1 Compare November 12, 2024 20:25
fhanau
fhanau approved these changes Nov 12, 2024
View reviewed changes
Copy link
Contributor

@fhanau fhanau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (Didn't take a close look at wd_js_bundle.bzl changes but Yagiz already approved previously)

@mikea mikea merged commit 7f72d55 into main Nov 12, 2024
@mikea mikea deleted the maizatskyi/2024-11-01-node-compile-caches branch November 12, 2024 21:01
danlapid pushed a commit that referenced this pull request Dec 3, 2024
@mikea @danlapid
add compile cache support to wd_?s_bundle, enable for node (#3044)
8477713 
Is not used on load yet.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@anonrig anonrig anonrig approved these changes

@fhanau fhanau fhanau approved these changes

@jasnell jasnell Awaiting requested review from jasnell

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mikea @anonrig @fhanau