Skip to content

Reimplement ShadowLogger in Rust and bound its memory usage#1325

Merged
sporksmith merged 7 commits intoshadow:devfrom
sporksmith:bound-logger
May 18, 2021
Merged

Reimplement ShadowLogger in Rust and bound its memory usage#1325
sporksmith merged 7 commits intoshadow:devfrom
sporksmith:bound-logger

Conversation

@sporksmith
Copy link
Copy Markdown
Contributor

@sporksmith sporksmith commented May 7, 2021

  • Reimplement ShadowLogger in Rust
  • Log system thread names instead of artificial thread names.

Fixes #1301

@codecov
Copy link
Copy Markdown

codecov bot commented May 7, 2021

Codecov Report

Merging #1325 (f45fe9b) into dev (52b1ed1) will decrease coverage by 1.43%.
The diff coverage is 69.56%.

Impacted file tree graph

@@            Coverage Diff             @@
##              dev    #1325      +/-   ##
==========================================
- Coverage   55.71%   54.27%   -1.44%     
==========================================
  Files         140      138       -2     
  Lines       20100    20608     +508     
  Branches     4898     5189     +291     
==========================================
- Hits        11199    11186      -13     
- Misses       6048     6511     +463     
- Partials     2853     2911      +58     
Flag Coverage Δ
tests 54.27% <69.56%> (-1.44%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/main/core/manager.c 68.00% <ø> (-0.20%) ⬇️
src/main/core/scheduler/scheduler.c 70.93% <ø> (-0.15%) ⬇️
src/main/utility/fork_proxy.c 0.00% <ø> (ø)
src/support/logger/rust_bindings/src/bindings.rs 0.00% <0.00%> (ø)
src/shim/shim_logger.c 72.22% <47.05%> (-11.99%) ⬇️
src/support/logger/logger.c 86.58% <77.77%> (-2.61%) ⬇️
src/main/core/logger/log_wrapper.c 90.00% <90.00%> (ø)
src/main/core/controller.c 77.09% <100.00%> (ø)
src/main/core/main.c 50.00% <100.00%> (-0.43%) ⬇️
src/main/core/worker.c 73.08% <100.00%> (-0.94%) ⬇️
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 52b1ed1...f45fe9b. Read the comment docs.

@robgjansen robgjansen added Component: Libraries Support functions like LD_PRELOAD and logging Type: Bug Error or flaw producing unexpected results labels May 7, 2021
@robgjansen robgjansen linked an issue May 10, 2021 that may be closed by this pull request
@sporksmith sporksmith force-pushed the bound-logger branch 4 times, most recently from 7d991cd to 34c7e2b Compare May 13, 2021 18:37
@sporksmith sporksmith changed the title Bound logger Reimpliment ShadowLogger in Rust and bound its memory usage May 13, 2021
@sporksmith sporksmith changed the title Reimpliment ShadowLogger in Rust and bound its memory usage Reimplement ShadowLogger in Rust and bound its memory usage May 13, 2021
@sporksmith sporksmith requested a review from robgjansen May 13, 2021 18:46
@sporksmith sporksmith marked this pull request as ready for review May 13, 2021 18:48
Copy link
Copy Markdown
Member

@robgjansen robgjansen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable to me! But let's have Steven also look at src/main/core/logger/shadow_logger.rs.

Comment on lines +33 to +34
# Alternatively, just call libc's vsprintf directly when the unstable feature
# `c_variadic` feature is stabilized, or if we decide to enable it anyway.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: one too many "feature" words.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

return logger;
}

void rustlogger_destroy(Logger* logger) { free(logger); } No newline at end of file
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: I've recently come to appreciate files ending with a blank line rather than the last line not containing a newline. When adding new lines in the future, the diff will show that the previous last line in the file is modified even though the only thing changed was the addition of a newline character.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Agreed; I'm used to vim doing it automatically. I'll see if there's a VSCode setting...


void logger_setLevel(Logger* logger, LogLevel level) {
if (!logger) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this empty line intentional? If so, and you want to keep the if statement, at least add a comment.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a comment

Comment on lines +38 to +43
let total_secs = u64::try_from(total_nanos / 1_000_000_000).unwrap();
let total_mins = u32::try_from(total_secs / 60).unwrap();
let hours = total_mins / 60;
let mins = total_mins - hours * 60;
let secs = total_secs - u64::from(total_mins) * 60;
let nanos = u64::try_from(total_nanos - u128::from(total_secs) * 1_000_000_000).unwrap();
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The difference here between total_* and just hours, mins, etc. is unclear. I'm confused why we are converting from nanos to seconds back to nanos; don't we lose information due to rounding? Probably just needs a comment.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added comments and renamed variables to hopefully clarify. Also added a test.

Comment on lines +67 to +68
// we flush the queue on the current thread instead of trying to send to
// send a command to the logger thread (because our thread-local sender
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: send to send

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

/// entries (e.g. sim time, running process, etc.). It's also designed for
/// high performance to accomodate heavy logging from multiple threads.
pub struct ShadowLogger {
// Channel used to send commands to second commands to the logger's thread.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

send commands to second commands: it's unclear what this means.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Comment on lines +349 to +399
// Flush *synchronously*, since we're likely about to crash one way or another.
self.flush_sync();
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wooo!

@robgjansen
Copy link
Copy Markdown
Member

@stevenengler I reviewed this but I think you should also review at least src/main/core/logger/shadow_logger.rs.

@robgjansen robgjansen requested a review from stevenengler May 17, 2021 17:33
Comment on lines +259 to +264
line = record
.line
.map(|l| format!("{}", l))
.as_ref()
.map(|s| s.as_str())
.unwrap_or("n/a"),
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe just print "-1" instead of "n/a" here if not available to avoid the extra string allocation from format!()?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Broke up this write! to avoid the allocation while keepin "n/a" instead of "-1".

Comment on lines +340 to +355
fn log(&self, record: &Record) {
if !self.enabled(record.metadata()) {
return;
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the individual host log level taken into account anywhere?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! It is now :)

Comment on lines +120 to +124
// A lock-free queue for individual log records. We don't put the records
// themselves in the `command_sender`, because `Sender` doesn't support
// getting the queue length. Conversely we don't put commands in this queue
// because it doesn't support blocking operations.
records: SegQueue<ShadowLogRecord>,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the logger doesn't push more than SYNC_FLUSH_QD_LINES_THRESHOLD records, does it make sense to make this an ArrayQueue to avoid the extra allocation? Maybe this doesn't matter since there are already a few other string allocations per record.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think I originally went with SegQueue because I started with a higher value for ASYNC_FLUSH_QD_LINES_THRESHOLD, and didn't want the fixed memory cost. With the current value of 100k it's not so bad, though.

* Replace the wrapper for calling Shadow's logging framework from Rust
  with a wrapper for calling Rust's logging framework from Shadow.
* Add shadow_logger.rs.
* Remove shadow_logger.c and its supporting modules.
* Delete calls to register threads with logger - no longer needed.
* No longer need to explicitly flush between rounds; log records are
  inserted into a single channel instead of multiple, so we don't need the
  "synchronization point".
@sporksmith sporksmith merged commit a3cfcee into shadow:dev May 18, 2021
@sporksmith sporksmith deleted the bound-logger branch May 18, 2021 19:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Component: Libraries Support functions like LD_PRELOAD and logging Type: Bug Error or flaw producing unexpected results

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Logger queue grows unbounded, exceeds available RAM at debug level

3 participants