Skip to content

Run garbage collection once when the main process is waiting for#5396

Merged
kit-ty-kate merged 1 commit intoocaml:masterfrom
kkeundotnet:mem2
Nov 27, 2024
Merged

Run garbage collection once when the main process is waiting for#5396
kit-ty-kate merged 1 commit intoocaml:masterfrom
kkeundotnet:mem2

Conversation

@kkeundotnet
Copy link
Copy Markdown
Contributor

@kkeundotnet kkeundotnet commented Dec 20, 2022 •
edited
Loading

In #5376, @dra27 suggested running Gc.compact when the main process is waiting for the children processes for the first time.

what I was thinking here was that possibly at the point where opamProcess first is going to wait
for the completion of running jobs, we could add a Gc.compact?

In my local running on opam install ppxlib with this PR, "GC compact" ran in the middle of parallel processing of actions.

The following actions will be performed:
=== install 1 package
  ∗ ppxlib 0.28.0
00:07.216  XSYS                            Adding to env { LC_ALL=C }
00:09.099  STATE                           depexts loaded in 1.883s
00:09.100  SOLUTION                        parallel_apply
00:09.100  SOLUTION                        Regroup shared source packages: {}

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫
00:09.106  PARALLEL                        Iterate over 3 task(s) with 11 process(es)
00:09.106  PARALLEL                        Starting job 444950918 (worker -/11 -/1 1/3): ⬇ ppxlib.0.28.0
00:09.106  SOLUTION                        Fetching sources for ppxlib.0.28.0
00:09.106  ACTION                          download_package: ppxlib.0.28.0
00:09.106  SYSTEM                          rmdir /Users/kkeundotnet/.opam/4.14.0/.opam-switch/sources/ppxlib.0.28.0
00:09.109  SYSTEM                          mkdir /var/folders/gx/6809fgsd3nndyyf69y3h12_h0000gn/T/opam-14220-67d907
00:09.110  PARALLEL                        Next task in job 444950918: /usr/bin/tar xfj /Users/kkeundotnet/.opam/download-cache/sha256/d8/d87ae5f9a081206308ca964809b50a66aeb8e83d254801e8b9675448b60cf377 -C /var/folders/gx/6809fgsd3nndyyf69y3h12_h0000gn/T/opam-14220-67d907
00:09.619  PARALLEL                        GC compact (heap 490 MB -> 328 MB)
00:09.619  PARALLEL                        Collected task for job 444950918 (ret:0)
00:10.158  SYSTEM                          rmdir /var/folders/gx/6809fgsd3nndyyf69y3h12_h0000gn/T/opam-14220-67d907
⬇ retrieved ppxlib.0.28.0  (cached)
00:10.321  PARALLEL                        Job 444950918 finished

Similar to #5376, this PR enabled my 1GB-RAM machine to install ppxlib or js_of_ocaml without OOM.

@kkeundotnet kkeundotnet mentioned this pull request Dec 20, 2022
Closed
@rjbou
Copy link
Copy Markdown
Collaborator

rjbou commented Jan 6, 2023

Thanks for the PR! we'll look at it

rjbou
rjbou reviewed Jan 6, 2023
View reviewed changes
@dra27 dra27 self-assigned this Jan 6, 2023
@kit-ty-kate
Copy link
Copy Markdown
Member

kit-ty-kate commented Nov 1, 2023

@kkeundotnet was there any specific reason why you closed this PR? Sorry we haven’t had time to review this properly but there is only 1 person working on opam full-time and we’ve been extremely busy working on the release of 2.2.0 which includes Windows support (which is not an easy task)

@kkeundotnet kkeundotnet reopened this Nov 1, 2023
@kkeundotnet
Copy link
Copy Markdown
Contributor Author

kkeundotnet commented Nov 1, 2023

No, just because it has been stcuk long time and the maintainers look busy to address this minor issue, supporting 1GB-ram machine by running GC once at starting. Also, I found 1GB is small enough to OOM on building js_of_ocaml even with this PR these days. So, I need to find another way anyhow.

I reopen the PR, but feel free to close it if you have not much of time. Actually, "Windows support" sounds much much cooler, more exciting, and important!!

@kit-ty-kate kit-ty-kate added this to the 2.4.0~alpha1 milestone Sep 25, 2024
@kit-ty-kate kit-ty-kate requested a review from rjbou November 14, 2024 21:44
@kkeundotnet @rjbou
Run garbage collection once when the main process is waiting for
b69aa45 
In ocaml#5376, @dra27 suggested running `Gc.compact` when the main
process is waiting for the children processes for the first time.

> what I was thinking here was that possibly at the point where opamProcess first is going to wait
> for the completion of running jobs, we could add a Gc.compact?

In my local running on `opam install ppxlib`, "GC compact" ran in the middle of parallel processing
of actions.

```
The following actions will be performed:
=== install 1 package
  ∗ ppxlib 0.28.0
00:07.216  XSYS                            Adding to env { LC_ALL=C }
00:09.099  STATE                           depexts loaded in 1.883s
00:09.100  SOLUTION                        parallel_apply
00:09.100  SOLUTION                        Regroup shared source packages: {}

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫
00:09.106  PARALLEL                        Iterate over 3 task(s) with 11 process(es)
00:09.106  PARALLEL                        Starting job 444950918 (worker -/11 -/1 1/3): ⬇ ppxlib.0.28.0
00:09.106  SOLUTION                        Fetching sources for ppxlib.0.28.0
00:09.106  ACTION                          download_package: ppxlib.0.28.0
00:09.106  SYSTEM                          rmdir /Users/scho/.opam/4.14.0/.opam-switch/sources/ppxlib.0.28.0
00:09.109  SYSTEM                          mkdir /var/folders/gx/6809fgsd3nndyyf69y3h12_h0000gn/T/opam-14220-67d907
00:09.110  PARALLEL                        Next task in job 444950918: /usr/bin/tar xfj /Users/scho/.opam/download-cache/sha256/d8/d87ae5f9a081206308ca964809b50a66aeb8e83d254801e8b9675448b60cf377 -C /var/folders/gx/6809fgsd3nndyyf69y3h12_h0000gn/T/opam-14220-67d907
00:09.619  PARALLEL                        GC compact (heap 490 MB -> 328 MB)
00:09.619  PARALLEL                        Collected task for job 444950918 (ret:0)
00:10.158  SYSTEM                          rmdir /var/folders/gx/6809fgsd3nndyyf69y3h12_h0000gn/T/opam-14220-67d907
⬇ retrieved ppxlib.0.28.0  (cached)
00:10.321  PARALLEL                        Job 444950918 finished
```

Similar to ocaml#5376, this PR enabled my 1GB-RAM machine to install `ppxlib` or `js_of_ocaml` without OOM.
rjbou
rjbou approved these changes Nov 27, 2024
View reviewed changes
Copy link
Copy Markdown
Collaborator

@rjbou rjbou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR!

@kit-ty-kate
Copy link
Copy Markdown
Member

kit-ty-kate commented Nov 27, 2024

Thanks!

@kit-ty-kate kit-ty-kate merged commit b2a4da8 into ocaml:master Nov 27, 2024
@kit-ty-kate kit-ty-kate mentioned this pull request May 19, 2025
Open
@kit-ty-kate kit-ty-kate mentioned this pull request Aug 26, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kit-ty-kate kit-ty-kate kit-ty-kate approved these changes

@rjbou rjbou rjbou approved these changes

Assignees

@dra27 dra27

Labels

AREA: PERFORMANCE STATE: READY TO MERGE

Projects

None yet

Milestone

2.4.0~alpha1

Development

Successfully merging this pull request may close these issues.

4 participants

@kkeundotnet @rjbou @kit-ty-kate @dra27