Skip to content

More useful stats output by privPrecCompare#1642

Merged
michael-schwarz merged 2 commits intomasterfrom
priv_prec_compare_stats
Jan 17, 2025
Merged

More useful stats output by privPrecCompare#1642
michael-schwarz merged 2 commits intomasterfrom
priv_prec_compare_stats

Conversation

@michael-schwarz
Copy link
Copy Markdown
Member

@michael-schwarz michael-schwarz commented Dec 17, 2024

For my thesis, I was considering many different configurations, and the pairwise output got tedious:

protection equal to mine-W    (equal: 427, more precise: 0, less precise: 0, incomparable: 0, total: 427)
protection equal to lock    (equal: 427, more precise: 0, less precise: 0, incomparable: 0, total: 427)
protection equal to write    (equal: 427, more precise: 0, less precise: 0, incomparable: 0, total: 427)
protection equal to write+lock    (equal: 427, more precise: 0, less precise: 0, incomparable: 0, total: 427)
protection less precise than protection-interval    (equal: 51, more precise: 0, less precise: 376, incomparable: 0, total: 427)
protection less precise than mine-W-interval    (equal: 51, more precise: 0, less precise: 376, incomparable: 0, total: 427)
protection less precise than lock-interval    (equal: 51, more precise: 0, less precise: 376, incomparable: 0, total: 427)
protection less precise than write-interval    (equal: 51, more precise: 0, less precise: 376, incomparable: 0, total: 427)
protection less precise than write+lock-interval    (equal: 51, more precise: 0, less precise: 376, incomparable: 0, total: 427)
protection equal to protection-tid    (equal: 427, more precise: 0, less precise: 0, incomparable: 0, total: 427)
protection equal to lock-tid    (equal: 427, more precise: 0, less precise: 0, incomparable: 0, total: 427)
protection equal to write-tid    (equal: 427, more precise: 0, less precise: 0, incomparable: 0, total: 427)
protection equal to write+lock-tid    (equal: 427, more precise: 0, less precise: 0, incomparable: 0, total: 427)
protection less precise than protection-tid-interval    (equal: 51, more precise: 0, less precise: 376, incomparable: 0, total: 427)
protection less precise than lock-tid-interval    (equal: 51, more precise: 0, less precise: 376, incomparable: 0, total: 427)
protection less precise than write-tid-interval    (equal: 51, more precise: 0, less precise: 376, incomparable: 0, total: 427)
protection less precise than write+lock-tid-interval    (equal: 51, more precise: 0, less precise: 376, incomparable: 0, total: 427)
...

I had 18 configurations and thus 288 more pairwise comparisons like this (where many have equal precision).

With this PR, we still create the detailed output, but then at the end create buckets of approaches that have the same precision and then output one comparison per bucket.

For example, this output here (all approaches equally precise, TIDs do not matter, intervals pay off) is much easier to intuitively grasp than the 306 pairwise comparisons.

Bucket 0:
  protection-tid
  write
  write+lock-tid
  write+lock
  write-tid
  protection
  mine-W
  lock
  lock-tid
Bucket 1:
  write-interval
  lock-tid-interval
  write-tid-interval
  protection-interval
  mine-W-interval
  protection-tid-interval
  lock-interval
  write+lock-interval
  write+lock-tid-interval
Comparison between bucket 0 and 1: lock less precise than write+lock-tid-interval    (equal: 51, more precise: 0, less precise: 376, incomparable: 0, total: 427)

I merged this only into my dissertation branch back then (#1431 ). I opened this PR so we can consider if we want to have this or not.

@michael-schwarz michael-schwarz mentioned this pull request Dec 17, 2024
Closed
7 tasks
@sim642 sim642 added this to the v2.6.0 milestone Jan 17, 2025
sim642
sim642 approved these changes Jan 17, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@sim642 sim642 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think at one point when I looked at this I thought maybe there's some clever way to do this with DisjointDomain.PairwiseSet but it's not worth spending time to make this particularly elegant.

@michael-schwarz michael-schwarz merged commit ea7ee83 into master Jan 17, 2025
@michael-schwarz michael-schwarz deleted the priv_prec_compare_stats branch January 17, 2025 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sim642 sim642 sim642 approved these changes

Assignees

No one assigned

Labels

benchmarking usability

Projects

None yet

Milestone

v2.6.0 Awkward Aardvark

Development

Successfully merging this pull request may close these issues.

2 participants

@michael-schwarz @sim642