indexer: move deduplication functionality purely to the kvindexer#9473
indexer: move deduplication functionality purely to the kvindexer#9473
Conversation
thanethomson
left a comment
There was a problem hiding this comment.
Could you please add a test here to test specifically for the problem outlined in #9460?
I've added a test. As part of it, I've added a flag in the constructor which will terminate the indexing service upon the first error. That way I can detect that the postgres indexer has successfully processes the transactions |
Co-authored-by: Jasmina Malicevic <jasmina.dustinac@gmail.com>
| batch.Ops, err = DeduplicateBatch(batch.Ops, is.txIdxr) | ||
| if err != nil { | ||
| is.Logger.Error("deduplicate batch", "height", height) | ||
| is.Logger.Info("indexed block exents", "height", height) |
There was a problem hiding this comment.
Why is the error no longer needed here. Is this only because we are getting rid of the "deduplicate batch"
|
I will add the changelog for the v0.34 and v0.37 backports |
| err = eventBus.PublishEventTx(types.EventDataTx{TxResult: *txResult2}) | ||
| require.NoError(t, err) | ||
|
|
||
| time.Sleep(100 * time.Millisecond) |
There was a problem hiding this comment.
We really need to refactor our abstractions in such a way that allows us to avoid time.Sleep in future when "awaiting" some kind of asynchronous consequence.
There was a problem hiding this comment.
Yeah, currently the indexer is such an opaque system to debug. One improvement I thought of would be to track the latest height indexed, then we could call to make sure that it had indexed these heights without error
| Height: 1, | ||
| Index: uint32(1), | ||
| Tx: types.Tx("bar"), | ||
| Result: abci.ResponseDeliverTx{Code: 1}, |
There was a problem hiding this comment.
The Postgres indexer would error in this case, right?
There was a problem hiding this comment.
Yes, previously it would error
) (cherry picked from commit 4fd19a2)
) (cherry picked from commit 4fd19a2)
|
I couldn't find the docs for creating initial migration or relation in DB |
…ndermint#9473) (#228) (cherry picked from commit 4fd19a2) Co-authored-by: Callum Waters <cmwaters19@gmail.com>
Closes: #9460
The deduplication logic which prevents older failed transactions from overwriting newer successful transactions called the
Getmethod on theTxIndexer. For postgresql, this wasn't implemented and would always return an error. Errors were only logged which meant that the postgresql would batchnilfor that height` and then continue. This resolves the issue by moving the logic purely within the kvindexer