Skip to content

#12439 change parts that may be affecting the benchmark#12446

Closed
allrob23 wants to merge 6 commits intotwisted:trunkfrom
allrob23:patch-fix-test-publisher-bench
Closed

#12439 change parts that may be affecting the benchmark#12446
allrob23 wants to merge 6 commits intotwisted:trunkfrom
allrob23:patch-fix-test-publisher-bench

Conversation

@allrob23
Copy link
Contributor

@allrob23 allrob23 commented May 4, 2025

Scope and purpose

While analyzing the profiler in the codespace, I noticed that the DummyObserver class was spending some time processing events, which could influence the benchmark results. Additionally, the benchmark test a test was not adding observers in the go function, missing a part of the setup that affects the performance measurement.

Changes Made

  • Modified DummyObserver to remove event storage (last_event) and make __call__ a no-op, reducing its impact on the benchmark.
  • Updated test_log_publisher_call_dispatch to explicitly add observers to the LogPublisher inside the go function, ensuring the benchmark reflects the realistic setup where observers are added before dispatching events.
  • Changed the initialization of LogPublisher to start without observers, aligning with the new setup in go.

Related with #12439 and #12440

Contributor Checklist:

This process applies to all pull requests - no matter how small.
Have a look at our developer documentation before submitting your Pull Request.

Below is a non-exhaustive list (as a reminder):

  • The title of the PR should describe the changes and starts with the associated issue number, like “#9782 Remove twisted.news. #1234 Brief description”.
  • A release notes news fragment file was create in src/twisted/newsfragments/ (see: Release notes fragments docs.)
  • The automated tests were updated.
  • Once all checks are green, request a review by leaving a comment that contains exactly the string please review.
    Our bot will trigger the review process, by applying the pending review label
    and requesting a review from the Twisted dev team.

@codspeed-hq
Copy link

codspeed-hq bot commented May 4, 2025

CodSpeed Performance Report

Merging #12446 will degrade performances by 98.96%

Comparing allrob23:patch-fix-test-publisher-bench (7362817) with trunk (aedee3e)

Summary

❌ 1 regressions
✅ 35 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark BASE HEAD Change
test_log_publisher_call_dispatch 1.3 ms 127.1 ms -98.96%

@allrob23 allrob23 changed the title test: remove parts that may be affecting the benchmark 12439 remove parts that may be affecting the benchmark May 4, 2025
@allrob23 allrob23 changed the title 12439 remove parts that may be affecting the benchmark #12439 remove parts that may be affecting the benchmark May 4, 2025
@allrob23
Copy link
Contributor Author

allrob23 commented May 4, 2025

please review

@chevah-robot chevah-robot requested a review from a team May 4, 2025 15:06
@allrob23 allrob23 changed the title #12439 remove parts that may be affecting the benchmark #12439 change parts that may be affecting the benchmark May 4, 2025
Copy link
Member

@glyph glyph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updating in the hopes that this will pass the static checks, but can you have a look and just try to push something which gets the green checkmark?

@glyph
Copy link
Member

glyph commented Aug 11, 2025

@allrob23 It looks like the only issue here is that you forgot to file an issue describing the problem first and adding a news fragment. The fragment itself could be a .misc, so hopefully this is just a few minutes of paperwork to get things unstuck?

@adiroiban
Copy link
Member

@glyph maybe in some cases we can use the PR id instead of the issue id... to reduce the red tape

I understand that we have an issue, and that issue might not be fixed with the first proposed PR...

but if this PR is not merged, the newsfragment is also not merged

Just trying to make it easier for people to contribute

@glyph
Copy link
Member

glyph commented Aug 12, 2025

maybe in some cases we can use the PR id instead of the issue id... to reduce the red tape

I think that writing down the issue that you plan to fix before writing any code is a really good habit that I wish felt less like "red tape". But, given that we seem to be going against the grain, perhaps in the case where we would accept a .misc anyway, we can accept a .misc with the PR number? (I do think that we really need to insist on the red tape of "I explicitly considered whether this belongs in the changelog or not", even if we don't need an issue)

@adiroiban
Copy link
Member

"I explicitly considered whether this belongs in the changelog or not", even if we don't need an issue)

I fully agree with creating a news fragment file for every PR.


The question is about creating a GitHub Issue, when you already have a Github PR with a description of the "problem"

@allrob23
Copy link
Contributor Author

Hello @adiroiban and @glyph , surry but after your conversation I don't quite understand what I need to do, do I need to create the Issue? Is there something else I should do?

@glyph
Copy link
Member

glyph commented Aug 12, 2025

do I need to create the Issue? Is there something else I should do?

Adi already created the news fragment for you, so you're all set here as far as that goes. I guess the sticking point right now is that I don't understand the massive benchmark regression here. wouldn't emptying out __call__ make the benchmark faster? If you're expecting it to get 100% slower, then I guess we can move forward :)

Copy link
Member

@glyph glyph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops. A good indication of why writing an issue before writing a PR is a good idea, actually. This is based on a total misunderstanding of what the benchmark is trying to measure :)

Comment on lines +38 to +39
for obs in observers:
publisher.addObserver(obs)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, this is definitely wrong. go() is the function that the benchmark is running over and over again. This is supposed to be measuring dispatch, i.e. the amount of time that LogPublisher.__call__ takes, not the amount of time addObserver takes.

@glyph
Copy link
Member

glyph commented Aug 12, 2025

Additionally, the benchmark test a test was not adding observers in the go function, missing a part of the setup that affects the performance measurement.

The setup is excluded from the benchmark measurement on purpose. The benchmark that captures addObserver is benchmarks/test_log_publisher.py. This benchmark is testing log dispatch.

It's possible that we might want to replace DummyObserver with lambda event: None or something even lower-overhead than a single attribute set, but as it stands, this change would basically break the benchmarks, so I am going to close this PR.

@glyph glyph closed this Aug 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants