Skip to content

fix(metrics): Fix metrics propagation for destination plugins with managed write mode#585

Closed
candiduslynx wants to merge 1 commit intomainfrom
fix/destinations/metrics
Closed

fix(metrics): Fix metrics propagation for destination plugins with managed write mode#585
candiduslynx wants to merge 1 commit intomainfrom
fix/destinations/metrics

Conversation

@candiduslynx
Copy link
Copy Markdown
Contributor

Refactored destination plugin metrics:

  1. incorporated metrics as well as table info into workers
  2. properly collecting metrics from workers for managed writers
  3. left only a single mutex in the destination plugin
  4. refactored map handling a bit

@candiduslynx
Copy link
Copy Markdown
Contributor Author

this PR is following in the steps of #583

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 9, 2023

⏱️ Benchmark results

Comparing with c196a74

  • DefaultConcurrencyDFS-2 resources/s: 11,210 ⬇️ 0.54% decrease vs. c196a74
  • DefaultConcurrencyRoundRobin-2 resources/s: 12,373 ⬇️ 0.65% decrease vs. c196a74
  • Glob-2 ns/op: 159 ⬇️ 27.99% decrease vs. c196a74
  • TablesWithChildrenDFS-2 resources/s: 27,960 ⬆️ 5.89% increase vs. c196a74
  • TablesWithChildrenRoundRobin-2 resources/s: 27,914 ⬆️ 5.08% increase vs. c196a74
  • TablesWithRateLimitingDFS-2 resources/s: 28.53 ⬆️ 0.63% increase vs. c196a74
  • TablesWithRateLimitingRoundRobin-2 resources/s: 790.8 ⬇️ 2.50% decrease vs. c196a74
  • BufferedScanner-2 ns/op: 9.438 ⬇️ 33.18% decrease vs. c196a74
  • LogReader-2 ns/op: 30.74 ⬇️ 23.68% decrease vs. c196a74

Copy link
Copy Markdown
Contributor

@hermanschaaf hermanschaaf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just left a few comments. It would also be good to add a test that ensures the metrics behavior stays fixed.

@candiduslynx candiduslynx force-pushed the fix/destinations/metrics branch from c33119b to 4729f2f Compare January 9, 2023 13:40
@candiduslynx candiduslynx self-assigned this Jan 9, 2023
@candiduslynx candiduslynx force-pushed the fix/destinations/metrics branch 2 times, most recently from cd7dc21 to 6d71c67 Compare January 9, 2023 16:46
@candiduslynx candiduslynx force-pushed the fix/destinations/metrics branch 2 times, most recently from e7c2d14 to 38448ea Compare January 11, 2023 16:27
Copy link
Copy Markdown
Contributor

@yevgenypats yevgenypats left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quite a big change so just reviewed partially.

@yevgenypats
Copy link
Copy Markdown
Contributor

yevgenypats commented Jan 11, 2023

Prob worth a live chat as I don't think I quite understand what this solves

@candiduslynx candiduslynx force-pushed the fix/destinations/metrics branch from 38448ea to c727fda Compare January 12, 2023 09:00
@candiduslynx
Copy link
Copy Markdown
Contributor Author

@yevgenypats I've updated the metrics code. Now we use sync.Uint64 to store locally & return the destination.Metrics as read value.

@candiduslynx
Copy link
Copy Markdown
Contributor Author

related to cloudquery/cloudquery#6761

@candiduslynx candiduslynx force-pushed the fix/destinations/metrics branch from c727fda to 50bcb13 Compare January 16, 2023 11:27
kodiakhq bot pushed a commit to cloudquery/cloudquery that referenced this pull request Jan 18, 2023
@yevgenypats
Copy link
Copy Markdown
Contributor

Closing as I didn't understand the fix and I'll prob just fix the metrics later.

@candiduslynx candiduslynx deleted the fix/destinations/metrics branch January 19, 2023 14:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants