Skip to content

Realtime scheduling#1118

Merged
sporksmith merged 6 commits intoshadow:devfrom
sporksmith:rwails/realtime
Feb 24, 2021
Merged

Realtime scheduling#1118
sporksmith merged 6 commits intoshadow:devfrom
sporksmith:rwails/realtime

Conversation

@sporksmith
Copy link
Copy Markdown
Contributor

@sporksmith sporksmith commented Feb 23, 2021

  • Adds some sched_yields inside shadow to prevent poor performance or deadlocks under CPU contention when using the realtime scheduler. These also make performance less-bad when there's CPU contention, even with the regular scheduler.
  • Adds a command-line option for using the realtime scheduler

Some initial analysis at https://github.com/sporksmith/phantom-paper/blob/realtime-experiment/experiments/realtime/analysis.ipynb

More analysis in progress; looks like cpu-pinning+realtime is a significant win in preload mode.

The ptrace modes are currently broken with the realtime scheduler.

Fixes #964

@codecov
Copy link
Copy Markdown

codecov bot commented Feb 24, 2021

Codecov Report

Merging #1118 (6b7a060) into dev (20c3413) will increase coverage by 0.02%.
The diff coverage is 33.33%.

Impacted file tree graph

@@            Coverage Diff             @@
##              dev    #1118      +/-   ##
==========================================
+ Coverage   55.92%   55.95%   +0.02%     
==========================================
  Files         136      136              
  Lines       20065    20075      +10     
  Branches     4768     4770       +2     
==========================================
+ Hits        11221    11232      +11     
+ Misses       6003     5996       -7     
- Partials     2841     2847       +6     
Flag Coverage Δ
tests 55.95% <33.33%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/main/core/main.c 43.90% <11.11%> (-2.59%) ⬇️
.../main/core/scheduler/scheduler_policy_host_steal.c 82.12% <100.00%> (+0.06%) ⬆️
src/shim/binary_spinning_sem.cc 64.70% <100.00%> (ø)
src/main/host/thread_ptrace.c 67.20% <0.00%> (-0.88%) ⬇️
src/main/core/scheduler/scheduler.c 77.40% <0.00%> (-0.34%) ⬇️
src/main/core/slave.c 68.95% <0.00%> (+0.65%) ⬆️
src/main/utility/utility.c 31.70% <0.00%> (+6.82%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 20c3413...7acb2ef. Read the comment docs.

Copy link
Copy Markdown
Member

@robgjansen robgjansen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Easy-peasy. I just have some minor suggested changes. Thanks!

Comment on lines +33 to +36
OPTION_EXPERIMENTAL_ENTRY(
"set-sched-fifo", 0, 0, G_OPTION_ARG_INT, &_setSchedFifo,
"Use the SCHED_FIFO scheduler. Requires CAP_SYS_NICE. See sched(7), capabilities(7) [0]",
"[0|1]")
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we want this to be a boolean argument? Here is how we specify other boolean args (from options.c):

{"version", 'v', 0, G_OPTION_ARG_NONE, &(options->printSoftwareVersion),
  "Print software version and exit", NULL},
{"pin-cpus", 'z', 0, G_OPTION_ARG_NONE, &(options->pinCPUs),
 "Use experimental CPU pinning", NULL},

So I think this would just become:

OPTION_EXPERIMENTAL_ENTRY(
    "set-sched-fifo", 0, 0, G_OPTION_ARG_NONE, &_setSchedFifo,
    "Use the SCHED_FIFO scheduler. Requires CAP_SYS_NICE. See sched(7), capabilities(7)",
    NULL)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see. I'd previously looked for but couldn't find a way to suppoert e.g. --set-sched-fifo=true and --set-sched-fifo=false. It looks like if we wanted to add a way to explicitly disable it we could add another option --no-set-sched-fifo using the option G_OPTION_FLAG_REVERSE

fprintf(stderr, "Could not set SCHED_FIFO!\n");
return -1;
} else {
fprintf(stderr, "In mode SCHED_FIFO.\n");
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Log this with our logger, probably at message level. How about:

message("Successfully set real-time scheduler mode to SCHED_FIFO");

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

int rc = sched_setscheduler(0, SCHED_FIFO, &param);

if (rc != 0) {
fprintf(stderr, "Could not set SCHED_FIFO!\n");
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logger has already been set up at this point, so can we log this with our logger? How about:

error("Could not set real-time scheduler mode to SCHED_FIFO!")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@sporksmith sporksmith merged commit 886a116 into shadow:dev Feb 24, 2021
@sporksmith sporksmith deleted the rwails/realtime branch February 24, 2021 22:31
@robgjansen robgjansen added Component: Main Composing the core Shadow executable Type: Enhancement New functionality or improved design labels Feb 24, 2021
@robgjansen robgjansen added the Tag: Performance Related to improving shadow's run-time label Mar 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Component: Main Composing the core Shadow executable Tag: Performance Related to improving shadow's run-time Type: Enhancement New functionality or improved design

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants