Skip to content

Filter Processor Benchmarking#146

Merged
winder merged 3 commits intoalgorand:masterfrom
winder:will/filter-benchmark
Sep 26, 2023
Merged

Filter Processor Benchmarking#146
winder merged 3 commits intoalgorand:masterfrom
winder:will/filter-benchmark

Conversation

@winder
Copy link
Contributor

@winder winder commented Aug 28, 2023 •
edited
Loading

Summary

Add new benchmarks Benchmarks for filters.

The results suggest that we're doing something extremely inefficient when setting up each additional filter.

Applying one filter to 1 or 25000 transactions has an almost negligible difference.
On the other hand there seems to be a fixed 8ms overhead for each filter.

The numbers in each report are as follows:

  • number of transactions in the test block
  • number of string equality filters
  • number of regex filters
  • number of numeric filters.
num txn eq filters regex filters numeric filters
25000 1000 0 0
1 1000 0 0
25000 1 0 0
1 1 0 0
25000 0 1000 0
1 0 1000 0
25000 0 1 0
1 0 1 0
25000 0 0 1000
1 0 0 1000
25000 0 0 1
1 0 0 1 
BenchmarkProcess2/txn_25000_equal_1000_regex_0_numeric_0
BenchmarkProcess2/txn_25000_equal_1000_regex_0_numeric_0-8         	       1	7791642627 ns/op
BenchmarkProcess2/txn_0_equal_1000_regex_0_numeric_0
BenchmarkProcess2/txn_0_equal_1000_regex_0_numeric_0-8             	       1	8057424687 ns/op
BenchmarkProcess2/txn_25000_equal_1_regex_0_numeric_0
BenchmarkProcess2/txn_25000_equal_1_regex_0_numeric_0-8            	     136	   8644760 ns/op
BenchmarkProcess2/txn_1_equal_1_regex_0_numeric_0
BenchmarkProcess2/txn_1_equal_1_regex_0_numeric_0-8                	     133	   8780231 ns/op
BenchmarkProcess2/txn_25000_equal_0_regex_1000_numeric_0
BenchmarkProcess2/txn_25000_equal_0_regex_1000_numeric_0-8         	       1	2271617844 ns/op
BenchmarkProcess2/txn_1_equal_0_regex_1000_numeric_0
BenchmarkProcess2/txn_1_equal_0_regex_1000_numeric_0-8             	       1	2330539073 ns/op
BenchmarkProcess2/txn_25000_equal_0_regex_1_numeric_0
BenchmarkProcess2/txn_25000_equal_0_regex_1_numeric_0-8            	     384	   2741222 ns/op
BenchmarkProcess2/txn_1_equal_0_regex_1_numeric_0
BenchmarkProcess2/txn_1_equal_0_regex_1_numeric_0-8                	     414	   2810811 ns/op
BenchmarkProcess2/txn_25000_equal_0_regex_0_numeric_1000
BenchmarkProcess2/txn_25000_equal_0_regex_0_numeric_1000-8         	       6	 206244626 ns/op
BenchmarkProcess2/txn_25000_equal_0_regex_0_numeric_1
BenchmarkProcess2/txn_25000_equal_0_regex_0_numeric_1-8            	    2234	    489659 ns/op
BenchmarkProcess2/txn_1_equal_0_regex_0_numeric_1000
BenchmarkProcess2/txn_1_equal_0_regex_0_numeric_1000-8             	       6	 197892161 ns/op
BenchmarkProcess2/txn_1_equal_0_regex_0_numeric_1
BenchmarkProcess2/txn_1_equal_0_regex_0_numeric_1-8                	    2068	    508157 ns/op

@winder winder requested a review from a team August 28, 2023 15:19
@winder winder self-assigned this Aug 28, 2023
@winder winder requested review from Eric-Warehime, algochoi, shiqizng and tzaffi and removed request for a team August 28, 2023 15:19
@winder winder force-pushed the will/filter-benchmark branch from 1fc05d3 to 0e01345 Compare August 28, 2023 15:19
@winder winder marked this pull request as ready for review August 28, 2023 15:19
@codecov
Copy link

codecov bot commented Aug 28, 2023

Codecov Report

Merging #146 (0e01345) into master (442791a) will increase coverage by 4.05%.
Report is 55 commits behind head on master.
The diff coverage is 80.81%.

@@            Coverage Diff             @@
##           master     #146      +/-   ##
==========================================
+ Coverage   67.66%   71.71%   +4.05%     
==========================================
  Files          32       37       +5     
  Lines        1976     2747     +771     
==========================================
+ Hits         1337     1970     +633     
- Misses        570      678     +108     
- Partials       69       99      +30
Files Changed Coverage Δ
conduit/data/block_export_data.go 100.00% <ø> (+92.30%) ⬆️
conduit/metrics/metrics.go 100.00% <ø> (ø)
conduit/pipeline/metadata.go 69.11% <ø> (ø)
conduit/plugins/config.go 100.00% <ø> (ø)
...duit/plugins/exporters/filewriter/file_exporter.go 81.63% <ø> (-1.06%) ⬇️
conduit/plugins/importers/algod/metrics.go 100.00% <ø> (ø)
...gins/processors/filterprocessor/fields/searcher.go 77.50% <ø> (ø)
...ins/processors/filterprocessor/filter_processor.go 83.82% <ø> (+3.54%) ⬆️
...plugins/processors/filterprocessor/gen/generate.go 34.28% <ø> (ø)
conduit/plugins/processors/noop/noop_processor.go 64.70% <ø> (+6.81%) ⬆️
... and 21 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@shiqizng
Copy link
Contributor

shiqizng commented Aug 28, 2023

how do you tell from the output that there's fixed 8ms overhead for each filter?

@tzaffi
Copy link
Contributor

tzaffi commented Aug 28, 2023

BenchmarkProcess2/txn_1_equal_0_regex_0_numeric_1000
BenchmarkProcess2/txn_1_equal_0_regex_0_numeric_1000-8 6 197892161 ns/op

It seems that way because when running 1000 filters, it takes 8 seconds. EG:

BenchmarkProcess2/txn_0_equal_1000_regex_0_numeric_0-8             	       1	8057424687 ns/op
image

@tzaffi
Copy link
Contributor

tzaffi commented Aug 28, 2023

197892161

But numeric filters seems to be faster.

@winder
Copy link
Contributor Author

winder commented Aug 28, 2023

1 filter / 1 txn takes about 8ms
1 filter / 25000 txn takes about 8ms
1000 filters / 1 txn takes about 8 seconds
1000 filters / 25000 txn takes about 8 seconds

I think it follows that the number of txn seems to be nicely optimized, but there seems to be some per-filter overhead somewhere.

@winder
Copy link
Contributor Author

winder commented Aug 28, 2023

197892161

But numeric filters seems to be faster.

Good point, I was definitely looking at the first few results more than the other filter types.

@winder winder merged commit 83dec6a into algorand:master Sep 26, 2023
@winder winder deleted the will/filter-benchmark branch September 26, 2023 18:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@algorandskiy algorandskiy algorandskiy approved these changes

@tzaffi tzaffi Awaiting requested review from tzaffi

@Eric-Warehime Eric-Warehime Awaiting requested review from Eric-Warehime

@shiqizng shiqizng Awaiting requested review from shiqizng

@algochoi algochoi Awaiting requested review from algochoi

Assignees

@winder winder

Labels

Team Lamprey

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@winder @shiqizng @tzaffi @algorandskiy