Skip to content

Fix telemetry shutdown and log max filter#432

Merged
bwoebi merged 2 commits into
mainfrom
bob/fix-log-and-shutdown
May 16, 2024
Merged

Fix telemetry shutdown and log max filter#432
bwoebi merged 2 commits into
mainfrom
bob/fix-log-and-shutdown

Conversation

@bwoebi

@bwoebi bwoebi commented May 15, 2024

Copy link
Copy Markdown
Contributor

Fixing max() in max_level_hint().

And properly dropping the telemetry worker:
When the runtime is shutdown with pending apps, it will send Stop, but never actually shutdown the telemetry instances. This regressed with d1fb3bc. Now we simply drop the telemetry handle, which implicitly stops the worker as well causes the worker to be joined properly.

At least this led to the occasional deadlocked sidecar.

Fixing max() in max_level_hint().

And properly dropping the telemetry worker:
When the runtime is shutdown with pending apps, it will send Stop, but never actually shutdown the telemetry instances. This regressed with d1fb3bc.
Now we simply drop the telemetry handle, which implicitly stops the worker as well causes the worker to be joined properly.

Signed-off-by: Bob Weinand <bob.weinand@datadoghq.com>
@bwoebi bwoebi requested a review from a team as a code owner May 15, 2024 20:38
@codecov-commenter

codecov-commenter commented May 15, 2024

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 90.90909% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 66.17%. Comparing base (74c89eb) to head (5b2e6f1).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #432      +/-   ##
==========================================
+ Coverage   66.14%   66.17%   +0.02%     
==========================================
  Files         187      187              
  Lines       23166    23171       +5     
==========================================
+ Hits        15324    15334      +10     
+ Misses       7842     7837       -5     
Components Coverage Δ
crashtracker 19.34% <ø> (ø)
datadog-alloc 98.37% <ø> (ø)
data-pipeline 51.45% <ø> (ø)
data-pipeline-ffi 0.00% <ø> (ø)
ddcommon 81.23% <ø> (ø)
ddcommon-ffi 74.93% <ø> (ø)
ddtelemetry 53.72% <ø> (ø)
ipc 81.27% <ø> (ø)
profiling 76.83% <ø> (ø)
profiling-ffi 60.05% <ø> (ø)
serverless 0.00% <ø> (ø)
sidecar 29.67% <90.90%> (+0.29%) ⬆️
sidecar-ffi 0.00% <ø> (ø)
spawn-worker 54.98% <ø> (ø)
trace-mini-agent 69.12% <ø> (ø)
trace-normalization 97.79% <ø> (ø)
trace-obfuscation 95.74% <ø> (ø)
trace-protobuf 25.64% <ø> (ø)
trace-utils 68.85% <ø> (ø)

Comment thread sidecar/src/log.rs
.values()
.map(|f| f.max_level_hint())
.min()
.max()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The whole ordering implementations for Level and LevelFilter makes my head spin since the actual underlying enum for both is ordered the other way around, i.e. Debug is 1 and Warn is 3, but apparently Debug is greater than Warn.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's exactly why I made the mistake in the first place. Really a what the fuck moment :-D

@bwoebi bwoebi merged commit 1ecd94c into main May 16, 2024
@bwoebi bwoebi deleted the bob/fix-log-and-shutdown branch May 16, 2024 17:08
morrisonlevi pushed a commit that referenced this pull request May 16, 2024
Fixing max() in max_level_hint().

And properly dropping the telemetry worker:
When the runtime is shutdown with pending apps, it will send Stop, but never actually shutdown the telemetry instances. This regressed with d1fb3bc.
Now we simply drop the telemetry handle, which implicitly stops the worker as well causes the worker to be joined properly.

Signed-off-by: Bob Weinand <bob.weinand@datadoghq.com>
r1viollet added a commit that referenced this pull request May 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants