feat: add erigon_getLatestLogs as a new feature API.#5875
Conversation
feat: add `erigon_getLatestLogs` as a new feature API. 1. `erigon_getLatestLogs` returns latest logs that match the filter with `logCount` length. Implements is similar to `erigon_getLogs` but it uses `ReverseIterator` which makes it more efficient to fetch latest logs.
fix: 1. logs index should be returned in descend order. 2. return logs and reduce the useless file I/O.
|
@fenghaojiang let's add some test? |
1. add simple test for `erigon_getLatestLogs`
It's too simple. I am wondering How can I add tests to |
|
Can take |
| db := m.DB | ||
| agg := m.HistoryV3Components() | ||
| api := NewErigonAPI(NewBaseApi(nil, stateCache, br, agg, false, rpccfg.DefaultEvmCallTimeout), db, nil) | ||
| logs, err := api.GetLatestLogs(context.Background(), filters.FilterCriteria{}, 1) |
There was a problem hiding this comment.
I can see some logs by:
logs, _ := api.GetLogs(ctx, filters.FilterCriteria{FromBlock: big.NewInt(0), ToBlock: big.NewInt(1024)})
for _, l := range logs {
fmt.Printf("alex: %v\n", l.BlockNumber)
}
There was a problem hiding this comment.
Thank you. Really appreciated for your help. I will add it tomorrow.
|
I am wondering why sometimes CI goes wrong... |
seems some unstable test TestEmptyBlock |
All tests have passed. Plz have a review. |
|
@fenghaojiang hi, we just added some test-cases: https://github.com/ledgerwatch/erigon/pull/5922/files#diff-b51f30a672ac364062718f3365716830f69ffed44ff545ce79f14e41945d7fcd |
| } | ||
| timestamp := header.Time | ||
|
|
||
| blockHash, err := rawdb.ReadCanonicalHash(tx, blockNumber) |
There was a problem hiding this comment.
can use header.Hash()
There was a problem hiding this comment.
Fixed it in following 2 commits.
|
Please add new method with description to |
Done in #5994 |
|
Fetching indices from 0 to latest may go OOM and in-general very bad. Need optimize it. |
How about giving a max |
|
the charts show how many decoded logs/s you can get from geth or erigon with a split up and parallelized query. didn't expect such abysmal performance from geth. it ended up being 21.4x slower than erigon, with many of the combinations timing out. Source Tweet: https://twitter.com/bantg/status/1547515905944555520 How about setting up a max |
Now it looks like “post-filter” we need something to avoid passing 0 to getTopicsBitmap. I don’t have opinion. eth_getLogs has no such problem, only erigon_getLatestLogs |
Sry I didn't get your point. You mean I should add limitation before the bloom filter, not the block numbers when iterating? |
|
@fenghaojiang i'm about this line: https://github.com/ledgerwatch/erigon/pull/5875/files#diff-8d2abcdcfbc55e04a41440a32062ea49b7aaa4d5d7f5a00279cb62c8c6d8d2dfR233 some accounts/topics may be old and popular |
But it's just a bloom filter to calculate the possible block numbers that may have the logs. The real iteration is in the following kv iterate the possible blocks and I add a limitation of blockCount in this line. |
I got it. When I looked into the |
|
Here is my opinion of enhancement. for i := latest; i >= 0; i-= 50000 {
end := latest
if latest >= 50000 {
begin := latest - 50000
}
if err := applyFilters(blockNumbers, tx, begin, end, crit); err != nil {
return nil, err
}
...
iter := blockNumbers.Iterator()
for iter.HasNext() {
//if count satisfied break the `for` loop and return
}
}What do you think? Looks not so neat and may need some advice. Thx. |
|
It’s fine for me. |
feat: add
erigon_getLatestLogsas a new feature API.erigon_getLatestLogsreturns latest logs that match the filter withlogCountlength. Implementation is similar toerigon_getLogsbut it usesReverseIteratorwhich makes it more efficient to fetch the latest logs.