Skip to content

Use default MaxDownloadParallelism based on number of CPU cores#2047

Merged
sethiay merged 2 commits into
masterfrom
default_pd_pl
Jun 27, 2024
Merged

Use default MaxDownloadParallelism based on number of CPU cores#2047
sethiay merged 2 commits into
masterfrom
default_pd_pl

Conversation

@sethiay

@sethiay sethiay commented Jun 21, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

Use default MaxDownloadParallelism based on number of CPU cores. Also set a minimum value of 16 in case the cores of CPU is less (2 or 4).

Using default 2 * number of cores as with the testing so far with a workload, this value looks good.

And couple of more things to tell upfront :

  1. This is based on CPU cores count of machine, so on GKE this will translate to machine's CPU cores and not on the CPU limit on pod. We are fine here because pod limit <= machine CPU cores. So, we are just assigning higher value to max parallel downloads there.
  2. It's possible to mount multiple GCSFuse mounts on single GCE VM/GKE node. In that case, one GCSFuse default is not affected by the other.

Link to the issue in case of a bug fix.

NA

Testing details

  1. Manual - NA
  2. Unit tests - NA
  3. Integration tests - NA

@codecov

codecov Bot commented Jun 24, 2024 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.94%. Comparing base (dd1542d) to head (42783e6).

Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #2047      +/-   ##
==========================================
- Coverage   71.95%   71.94%   -0.02%     
==========================================
  Files          99       99              
  Lines       10870    10872       +2     
==========================================
  Hits         7822     7822              
- Misses       2715     2717       +2     
  Partials      333      333
Flag Coverage Δ
unittests 71.94% <100.00%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@sethiay sethiay added the execute-integration-tests Run only integration tests label Jun 24, 2024
@sethiay sethiay marked this pull request as ready for review June 24, 2024 08:37
@sethiay sethiay requested a review from a team as a code owner June 24, 2024 08:37
@sethiay sethiay requested review from charith87 and raj-prince June 24, 2024 08:37
@kislaykishore

kislaykishore commented Jun 24, 2024

Copy link
Copy Markdown
Collaborator

@sethiay could you update the params.yaml as well? You'll need to make the following changes:

  • Add "runtime" here
  • Update the default here to "int64(math.Max(16, 2*runtime.NumCPU()))". Alternatively, you can create a function in cfg/ and reference it here.
  • Regenerate the config.go by calling go generate ./...

charith87
charith87 reviewed Jun 24, 2024
View reviewed changes
Comment thread internal/config/config_util.go
@sethiay

sethiay commented Jun 24, 2024

Copy link
Copy Markdown
Contributor Author

@sethiay could you update the params.yaml as well? You'll need to make the following changes:

  • Add "runtime" here
  • Update the default here to "int64(math.Max(16, 2*runtime.NumCPU()))". Alternatively, you can create a function in cfg/ and reference it here.
  • Regenerate the config.go by calling go generate ./...

As discussed offline, we will handle this in a separate PR.

@sethiay sethiay requested a review from kislaykishore as a code owner June 26, 2024 07:10
@sethiay sethiay requested a review from charith87 June 26, 2024 07:27
charith87
charith87 reviewed Jun 26, 2024
View reviewed changes
Comment thread internal/config/config_util_test.go
@charith87 charith87 self-requested a review June 26, 2024 18:45
@sethiay sethiay merged commit ef2179c into master Jun 27, 2024
@BrewTestBot BrewTestBot mentioned this pull request Jun 28, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@charith87 charith87 charith87 approved these changes

@raj-prince raj-prince Awaiting requested review from raj-prince raj-prince is a code owner automatically assigned from GoogleCloudPlatform/gcsfuse-codeowners

@kislaykishore kislaykishore Awaiting requested review from kislaykishore

Assignees

No one assigned

Labels

execute-integration-tests Run only integration tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@sethiay @kislaykishore @charith87