High throughput transport implementation by maciejwalkowiak · Pull Request #1114 · getsentry/sentry-java

maciejwalkowiak · 2020-12-15T13:30:11Z

📢 Type of change

Bugfix
New feature
Enhancement
Refactoring

📜 Description

Provides high throughput transport implementation meant to be used by backend applications using the performance feature.

Extracts rate limiting functionality to separate class, making it possible to use from other implementation of ITransport.

💡 Motivation and Context

Fixes #1097.

extracted rate-limiter that can be re-used in multiple transport implementations
lifting TransportFactory as a top level interface that can be set on SentryOptions - to provide an easy way to set custom transports
Apache Http Client 5 based high throughput transport implementation that under the hood is based on non blocking IO.
Spring Boot auto-configuration for the new transport

The last two are in the PR pending review (#1136).

💚 How did you test it?

Unit & integration tests.

📝 Checklist

I reviewed submitted code
I added tests to verify changes
All tests passing
No breaking changes

🔮 Next steps

Merge Add non blocking Apache HttpClient 5 based Transport. #1136
Do a final review for High throughput transport implementation #1114

bruno-garcia

nice!

marandaneto

LGTM

…1118) Flip the relation between `AsyncConnection` and `ITransport` and make `BlockingHttpTransport` use `HttpConnection`, making it possible to provide alternative transport implementations not bound to use thread pool.

…entry-java into gh-1097-rate-limiter

codecov-io · 2020-12-22T12:30:34Z

Codecov Report

Merging #1114 (2081d2e) into main (ef1eef7) will decrease coverage by 0.31%.
The diff coverage is 78.46%.

@@             Coverage Diff              @@
##               main    #1114      +/-   ##
============================================
- Coverage     74.95%   74.64%   -0.32%     
+ Complexity     1612     1605       -7     
============================================
  Files           166      167       +1     
  Lines          5603     5609       +6     
  Branches        548      550       +2     
============================================
- Hits           4200     4187      -13     
- Misses         1150     1164      +14     
- Partials        253      258       +5

Impacted Files	Coverage Δ	Complexity Δ
...c/main/java/io/sentry/transport/NoOpTransport.java	`0.00% <0.00%> (-40.00%)`	`0.00 <0.00> (-2.00)`
.../io/sentry/transport/QueuedThreadPoolExecutor.java	`80.95% <ø> (ø)`	`7.00 <0.00> (ø)`
...main/java/io/sentry/transport/StdoutTransport.java	`77.77% <ø> (+17.77%)`	`2.00 <0.00> (ø)`
.../src/main/java/io/sentry/NoOpTransportFactory.java	`66.66% <66.66%> (ø)`	`2.00 <2.00> (?)`
sentry/src/main/java/io/sentry/SentryOptions.java	`83.90% <75.00%> (+0.06%)`	`106.00 <2.00> (ø)`
.../main/java/io/sentry/transport/HttpConnection.java	`75.00% <75.00%> (ø)`	`18.00 <18.00> (?)`
...src/main/java/io/sentry/transport/RateLimiter.java	`75.47% <75.47%> (ø)`	`26.00 <26.00> (?)`
...n/java/io/sentry/transport/AsyncHttpTransport.java	`61.85% <82.14%> (ø)`	`8.00 <5.00> (?)`
...src/main/java/io/sentry/log4j2/SentryAppender.java	`82.50% <100.00%> (ø)`	`19.00 <0.00> (ø)`
...rc/main/java/io/sentry/logback/SentryAppender.java	`86.84% <100.00%> (-1.32%)`	`21.00 <1.00> (-1.00)`
... and 12 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ef1eef7...2081d2e. Read the comment docs.

bruno-garcia · 2020-12-22T19:55:23Z

sentry/src/main/java/io/sentry/AsyncHttpTransportFactory.java

+
+    URL sentryUrl;
+    try {
+      sentryUrl = parsedDsn.getSentryUri().toURL();


Ideally we'd do this eagerly (after init) so we throw at that point and crash.
Once the SDK inits we should capture all exceptions and log it out

It is done with init. If DSN is wrong SDK init will fail.

This piece will be refactored anyways in subsequent PR as every transport needs an envelope url, so there is no point for each factory to resolve it by itself.

sentry/src/main/java/io/sentry/transport/AsyncHttpTransport.java

bruno-garcia · 2020-12-22T20:01:42Z

sentry/src/main/java/io/sentry/transport/AsyncHttpTransport.java

+        envelopeCache.discard(envelope);
+      }
+    } else {
+      executor.submit(new EnvelopeSender(filteredEnvelope, hint, currentEnvelopeCache));


What happens if we capture in a tight loop, executor drops after maxItems?

Yes, in such case once we reach max queue items are dropped.

bruno-garcia · 2020-12-30T18:19:53Z

any blockers for merging this?

maciejwalkowiak · 2020-12-31T10:21:35Z

It's a PR for branch that takes all the NIO related changes in. Once #1136 is approved, this whole feature is complete and we can merge this PR too.

maciejwalkowiak · 2021-01-12T20:00:36Z

#1136 is merged. Please do a final review of this PR.

maciejwalkowiak added 6 commits December 10, 2020 13:11

WIP: Added non-blocking connection implementation.

e2dc3c7

Polish.

3eea40c

Drop NIO Connection, fix tests.

a7aa733

Add RateLimiter tests.

187cf93

Polish.

3a383b1

Polish.

83e1d14

maciejwalkowiak requested review from bruno-garcia and marandaneto as code owners December 15, 2020 13:30

Polish.

3296e31

bruno-garcia approved these changes Dec 15, 2020

View reviewed changes

marandaneto approved these changes Dec 17, 2020

View reviewed changes

maciejwalkowiak added 3 commits December 22, 2020 12:30

Merge remote-tracking branch 'origin/main' into gh-1097-rate-limiter

3ef7e65

Merge branch 'gh-1097-rate-limiter' of https://github.com/getsentry/s…

2081d2e

…entry-java into gh-1097-rate-limiter

bruno-garcia approved these changes Dec 22, 2020

View reviewed changes

Ref: Simplify ITransport creation in ITransportFactory. (#1135)

28a0068

maciejwalkowiak changed the title ~~Extract Rate Limiter~~ High throughput transport implementation Dec 31, 2020

maciejwalkowiak mentioned this pull request Dec 31, 2020

WIP: Added non-blocking connection implementation. #1104

Closed

8 tasks

bruno-garcia mentioned this pull request Jan 5, 2021

4.0.0 GA - Performance Monitoring #1151

Closed

22 tasks

maciejwalkowiak mentioned this pull request Jan 7, 2021

Attach HTTP headers to Sentry request #594

Closed

maciejwalkowiak added 3 commits January 12, 2021 15:44

Add non blocking Apache HttpClient 5 based Transport. (#1136)

92876c8

Merge remote-tracking branch 'origin/main' into gh-1097-rate-limiter

d4c6500

Make HttpConnection final.

65658d6

Merge branch 'main' into gh-1097-rate-limiter

d476de8

bruno-garcia approved these changes Jan 12, 2021

View reviewed changes

maciejwalkowiak merged commit 8159439 into main Jan 12, 2021

maciejwalkowiak deleted the gh-1097-rate-limiter branch January 12, 2021 20:19

adinauer mentioned this pull request Jun 27, 2025

Fix javadoc on TransportResult #4528

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

High throughput transport implementation#1114

High throughput transport implementation#1114
maciejwalkowiak merged 15 commits intomainfrom
gh-1097-rate-limiter

maciejwalkowiak commented Dec 15, 2020 •

edited

Loading

Uh oh!

bruno-garcia left a comment

Uh oh!

marandaneto left a comment

Uh oh!

codecov-io commented Dec 22, 2020

Uh oh!

bruno-garcia Dec 22, 2020

Uh oh!

maciejwalkowiak Dec 23, 2020

Uh oh!

Uh oh!

bruno-garcia Dec 22, 2020

Uh oh!

maciejwalkowiak Dec 23, 2020

Uh oh!

bruno-garcia commented Dec 30, 2020

Uh oh!

maciejwalkowiak commented Dec 31, 2020

Uh oh!

maciejwalkowiak commented Jan 12, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

maciejwalkowiak commented Dec 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📢 Type of change

📜 Description

💡 Motivation and Context

💚 How did you test it?

📝 Checklist

🔮 Next steps

Uh oh!

bruno-garcia left a comment

Choose a reason for hiding this comment

Uh oh!

marandaneto left a comment

Choose a reason for hiding this comment

Uh oh!

codecov-io commented Dec 22, 2020

Codecov Report

Uh oh!

bruno-garcia Dec 22, 2020

Choose a reason for hiding this comment

Uh oh!

maciejwalkowiak Dec 23, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bruno-garcia Dec 22, 2020

Choose a reason for hiding this comment

Uh oh!

maciejwalkowiak Dec 23, 2020

Choose a reason for hiding this comment

Uh oh!

bruno-garcia commented Dec 30, 2020

Uh oh!

maciejwalkowiak commented Dec 31, 2020

Uh oh!

maciejwalkowiak commented Jan 12, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

maciejwalkowiak commented Dec 15, 2020 •

edited

Loading