Skip to content

refactor(engine): optimize sandbox config storage#8035

Merged
rgrinberg merged 1 commit intomainfrom
ps/rr/refactor_engine___optimize_sandbox_config_storage
Nov 14, 2023
Merged

refactor(engine): optimize sandbox config storage#8035
rgrinberg merged 1 commit intomainfrom
ps/rr/refactor_engine___optimize_sandbox_config_storage

Conversation

@rgrinberg
Copy link
Copy Markdown
Member

We store a sandbox configuration set for every single action we create.
For a large build, this can add up.

Our old representation was taking 6 words to represent this. This PR
changes it to use only a single word. Moreover, this new set now opaque
to the GC, speeds up comparison, hashing, etc.

Signed-off-by: Rudi Grinberg me@rgrinberg.com

@rgrinberg rgrinberg requested a review from snowleopard June 24, 2023 09:16
@rgrinberg rgrinberg force-pushed the ps/rr/refactor_engine___optimize_sandbox_config_storage branch 2 times, most recently from f0e663f to 3735283 Compare June 24, 2023 09:39
@Alizter Alizter mentioned this pull request Jun 27, 2023
3 tasks
@rgrinberg rgrinberg force-pushed the ps/rr/refactor_engine___optimize_sandbox_config_storage branch 5 times, most recently from 6c4eac8 to b031433 Compare August 6, 2023 20:44
@rgrinberg
Copy link
Copy Markdown
Member Author

@snowleopard should be ready now

@rgrinberg rgrinberg force-pushed the ps/rr/refactor_engine___optimize_sandbox_config_storage branch 8 times, most recently from f1ce156 to 97cf6ec Compare August 12, 2023 08:54
Copy link
Copy Markdown
Collaborator

@snowleopard snowleopard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good but I left some suggestions/questions.

Also: have you run any benchmarks?

@rgrinberg
Copy link
Copy Markdown
Member Author

Also: have you run any benchmarks?

Right now the benchmarking server is broken. Once it's back up, I'll bring some numbers.

@rgrinberg rgrinberg force-pushed the ps/rr/refactor_engine___optimize_sandbox_config_storage branch 5 times, most recently from 3eb2635 to e6794f7 Compare August 23, 2023 10:53
@Alizter
Copy link
Copy Markdown
Collaborator

Alizter commented Sep 25, 2023

@rgrinberg Benchmarking is working again, can you resurrect this?

@rgrinberg rgrinberg force-pushed the ps/rr/refactor_engine___optimize_sandbox_config_storage branch from e6794f7 to d2d1f12 Compare September 25, 2023 10:11
@rgrinberg
Copy link
Copy Markdown
Member Author

It does reduce major and promoted words by around 0.3-0.4%. Not sure if it's worth it, I'll let @snowleopard decide.

We did end up using bit_set for something else, so I will merge that at least.

@Alizter
Copy link
Copy Markdown
Collaborator

Alizter commented Sep 25, 2023

@rgrinberg patch-back-source-tree.t test needs updating too.

@rgrinberg rgrinberg force-pushed the ps/rr/refactor_engine___optimize_sandbox_config_storage branch 2 times, most recently from 477abce to b4d1c9f Compare September 27, 2023 12:42
@rgrinberg
Copy link
Copy Markdown
Member Author

ping @snowleopard

@rgrinberg rgrinberg force-pushed the ps/rr/refactor_engine___optimize_sandbox_config_storage branch 2 times, most recently from a0f0eb3 to 7d7f058 Compare November 4, 2023 05:00
Copy link
Copy Markdown
Collaborator

@snowleopard snowleopard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good though see one question. I haven't had time to run any benchmarks internally but it should be a small win.

We store a sandbox configuration set for every single action we create.
For a large build, this can add up.

Our old representation was taking 6 words to represent this. This PR
changes it to use only a single word. Moreover, this new set now opaque
to the GC, speeds up comparison, hashing, etc.

Signed-off-by: Rudi Grinberg <me@rgrinberg.com>

<!-- ps-id: 8d6ab81b-57f7-4c94-b39c-617e78e9479d -->
@rgrinberg rgrinberg force-pushed the ps/rr/refactor_engine___optimize_sandbox_config_storage branch from 7d7f058 to 419127f Compare November 14, 2023 06:22
@rgrinberg rgrinberg merged commit 2b97b01 into main Nov 14, 2023
@rgrinberg rgrinberg deleted the ps/rr/refactor_engine___optimize_sandbox_config_storage branch November 14, 2023 08:36
voodoos pushed a commit to voodoos/dune that referenced this pull request Nov 14, 2023
We store a sandbox configuration set for every single action we create.
For a large build, this can add up.

Our old representation was taking 6 words to represent this. This PR
changes it to use only a single word. Moreover, this new set now opaque
to the GC, speeds up comparison, hashing, etc.

Signed-off-by: Rudi Grinberg <me@rgrinberg.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants