(VDB-1144) Enable diffs to come in out of order#97
Conversation
| db = test_config.NewTestDB(test_config.NewTestNode()) | ||
| flapContractAddress = test_data.FlapAddress() | ||
| repository flap.FlapStorageRepository | ||
| repository = &flap.FlapStorageRepository{ContractAddress: flapContractAddress} |
There was a problem hiding this comment.
I pulled this out of the BeforeEach due to getting some pointer-related weirdness where fields in the struct weren't getting set sometimes.
There was a problem hiding this comment.
Cool - not opposed, but I wonder if we'd get similar results if we declared this var as *flap.FlapStorageRepository and assigned with & in the BeforeEach
There was a problem hiding this comment.
Weirdly, I tried that and ended up getting a nil pointer dereference error. So now I'm afraid to touch it 😅
| type commonBidSnapshot struct { | ||
| AddressID string `db:"address_id"` | ||
| BlockNumber string `db:"block_number"` | ||
| BidID string `db:"bid_id"` | ||
| Guy sql.NullString | ||
| Tic sql.NullString | ||
| End sql.NullString | ||
| Lot sql.NullString | ||
| Bid sql.NullString | ||
| Updated string | ||
| } | ||
|
|
||
| type flipBidSnapshot struct { | ||
| commonBidSnapshot | ||
| Usr sql.NullString | ||
| Gal sql.NullString | ||
| Tab sql.NullString | ||
| } | ||
|
|
||
| func commonBidSnapshotFromValues(bidID int, blockNumber, addressID int64, updated string, bidValues map[string]interface{}) commonBidSnapshot { | ||
| parsedUpdated, parseErr := strconv.ParseInt(updated, 10, 64) | ||
| Expect(parseErr).NotTo(HaveOccurred()) | ||
| packedValues := bidValues[mcdStorage.Packed].(map[int]string) | ||
|
|
||
| return commonBidSnapshot{ | ||
| AddressID: strconv.FormatInt(addressID, 10), | ||
| BlockNumber: strconv.FormatInt(blockNumber, 10), | ||
| BidID: strconv.Itoa(bidID), | ||
| Guy: test_helpers.GetValidNullString(packedValues[0]), | ||
| Tic: test_helpers.GetValidNullString(packedValues[1]), | ||
| End: test_helpers.GetValidNullString(packedValues[2]), | ||
| Lot: test_helpers.GetValidNullString(bidValues[mcdStorage.BidLot].(string)), | ||
| Bid: test_helpers.GetValidNullString(bidValues[mcdStorage.BidBid].(string)), | ||
| Updated: FormatTimestamp(parsedUpdated), | ||
| } | ||
| } | ||
|
|
||
| func flipBidSnapshotFromValues(bidID int, blockNumber, addressID int64, updated string, bidValues map[string]interface{}) flipBidSnapshot { | ||
| return flipBidSnapshot{ | ||
| commonBidSnapshot: commonBidSnapshotFromValues(bidID, blockNumber, addressID, updated, bidValues), | ||
| Usr: test_helpers.GetValidNullString(bidValues[mcdStorage.BidUsr].(string)), | ||
| Gal: test_helpers.GetValidNullString(bidValues[mcdStorage.BidGal].(string)), | ||
| Tab: test_helpers.GetValidNullString(bidValues[mcdStorage.BidTab].(string)), | ||
| } | ||
| } | ||
|
|
||
| func assertBidSnapshot(actualBid, expectedBid commonBidSnapshot, expectedBlockNumber int64) { | ||
| Expect(actualBid.AddressID).To(Equal(expectedBid.AddressID)) | ||
| Expect(actualBid.BlockNumber).To(Equal(strconv.FormatInt(expectedBlockNumber, 10))) | ||
| Expect(actualBid.BidID).To(Equal(expectedBid.BidID)) | ||
| Expect(actualBid.Lot).To(Equal(expectedBid.Lot)) | ||
| Expect(actualBid.Bid).To(Equal(expectedBid.Bid)) | ||
| Expect(actualBid.Updated).To(Equal(expectedBid.Updated)) | ||
| } | ||
|
|
||
| func assertFlipSnapshot(actualFlip, expectedFlip flipBidSnapshot, expectedBlockNumber int64) { | ||
| assertBidSnapshot(actualFlip.commonBidSnapshot, expectedFlip.commonBidSnapshot, expectedBlockNumber) | ||
| Expect(actualFlip.Usr).To(Equal(expectedFlip.Usr)) | ||
| Expect(actualFlip.Gal).To(Equal(expectedFlip.Gal)) | ||
| Expect(actualFlip.Tab).To(Equal(expectedFlip.Tab)) | ||
| } |
There was a problem hiding this comment.
We already have datatypes/functions similar to these in test_helpers, but these are slightly different since they're designed to work with the maker.flip-style records from the DB, whereas the ones in test_helpers work with the api.flip_state-style types (returned by api.get_flip, etc.).
So I think it makes sense to have 2 distinct sets of helpers for 2 distinct types. Although, hopefully we'll eventually be able to expose the flip table directly instead of exposing its data via api.get_flips and all_flips, at which point we'd be able to delete the other set of helpers.
There was a problem hiding this comment.
Do we have a story for this replacement?
There was a problem hiding this comment.
Actually no, it doesn't look like we do. For some reason I thought I added stories like that at the same time that I added the ones about making better use of the ilk + urn trigger tables, but I guess not 🤔. I'll see to getting some into the backlog.
| }) | ||
| } | ||
|
|
||
| func FlipBidSnapshotTriggerTests(input BidTriggerTestInput) { |
There was a problem hiding this comment.
I had some trouble deciding how to balance abstraction vs readability with these tests. I ended up allowing some duplication for these 2 tests (the inserting a new field suite is duplicated on line 236) since they have to make assertions about the entire record, and thus are dealing with 2 distinct types for flip vs flap/flop records. Whereas the updating history with a new value suite only deals with 1 field at a time.
So, while it would've been possible to abstract the suites on lines 163 and 236 into a single suite, I felt like it would be too detrimental to the tests' (already questionable) clarity.
There was a problem hiding this comment.
Dig it - definitely a tricky balancing act here, but I'm in favor of prioritizing readability while we're already dealing with a shared test suite
| }) | ||
| } | ||
|
|
||
| func randomBidStorageValue(valueType vdbStorage.ValueType, packedValueType vdbStorage.ValueType) (interface{}, string) { |
There was a problem hiding this comment.
This function was my best effort at keeping packed-value repository logic out of the body of the tests. It returns 2 values: the first is what the repo.Create function wants to accept, and the second is the string value that will actually be inserted into the trigger-table, which we can use in assertions or raw SQL inserts. Usually these 2 values will be the same, except in the case of packed values.
There was a problem hiding this comment.
Might be worth exploring how much clutter we'd create by extracting a separate suite/function for packed values? Totally on board with the de-duplicating approach, but wondering if this is a place where there's a need for some kind of polymorphism to handle important differences between the data structures/avoid returning vals that are only useful to a subset of callers
There was a problem hiding this comment.
Yeah that's a good call. I guess in this case I was so preoccupied with whether or not I could combine all the tests suites into one, that I didn’t stop to think if I should. ;)
There was a problem hiding this comment.
The other thing I considered in order to avoid this problem entirely, is to create a utility function (just for tests) that handles inserting storage records, rather than relying on the repository. My thinking is that the trigger tests have nothing to do with the repositories (especially the 'packed values' portion of the repo logic), so the fact that we even need to think about packed values in the trigger tests feels like sort of a code smell to me. Maybe it would be worth decoupling these tests from the repository.
| }) | ||
|
|
||
| Describe("updating history with new value", func() { | ||
| It("updates field in subsequent blocks", func() { |
There was a problem hiding this comment.
This whole suite follows the general pattern of:
- insert record 1
- insert record 2
- see how record 1 reacted
c386162 to
6dbdf4a
Compare
rmulhol
left a comment
There was a problem hiding this comment.
💪
Nice work! None of my comments are blocking - just interested in grokking the details 👍
| AND flip.address_id = new_diff.address_id | ||
| AND flip.block_number >= diff_block_number | ||
| AND (next_guy_diff_block IS NULL | ||
| OR flip.block_number < next_guy_diff_block); |
There was a problem hiding this comment.
Maybe worth adding indexes on flip bid_id and block_number? Probably also relevant for flap and flop
There was a problem hiding this comment.
Good point, I hadn't put much thought into indexes. I know there should be an index on a few columns due to the primary key, which is on (bid_id, address_id, block_number), but I'm guessing that wouldn't have an effect if we're filtering on just 1 of those fields.
There was a problem hiding this comment.
I did some reading up on composite indexes, and it looks like in this case Postgres will use the primary key index, since the columns we're filtering on are included in the index. Definitely a good idea to keep indexes in mind moving forward though!
| db = test_config.NewTestDB(test_config.NewTestNode()) | ||
| flapContractAddress = test_data.FlapAddress() | ||
| repository flap.FlapStorageRepository | ||
| repository = &flap.FlapStorageRepository{ContractAddress: flapContractAddress} |
There was a problem hiding this comment.
Cool - not opposed, but I wonder if we'd get similar results if we declared this var as *flap.FlapStorageRepository and assigned with & in the BeforeEach
| ColumnName: constants.BidColumn, | ||
| } | ||
| shared_behaviors.CommonBidSnapshotTriggerTests(triggerInput) | ||
| shared_behaviors.SharedBidHistoryTriggerTests(triggerInput) |
There was a problem hiding this comment.
Is there a semantic distinction between Common and Shared here?
There was a problem hiding this comment.
Yeah although it isn't very clear. Since there are 2 separate bid structures at play, i.e. the one used by flap + flop vs the one used by flip (where the flap/flop one is a subset of the flip one), I'm trying to use the dichotomy between a "common bid" vs a "flip bid" to express that distinction. Hence why we're using CommonBidSnapshot... in this suite and FlipBidSnapshot... in the flip storage suite.
OTOH, by SharedBidHistoryTriggerTests, I just mean "tests about behavior that's shared across all bid history triggers".
So to recap, Common is being used as a way to distinguish between types of bids, and Shared is being used to describe what the tests are about. I'll try to think of a less confusing way to imply those ideas haha. Open to suggestions!
There was a problem hiding this comment.
I wonder if removing the Common and Shared would be okay? So just have BidSnapshotTriggerTests and BidHistoryTriggerTests and we can infer that they're common or shared by the fact that they're used in multiple places. And then FlipBidSnapshot... would stay named that way to explicitly point out that Flips are different?
There was a problem hiding this comment.
Great idea! Will update. FYI I've been enacting feedback from this PR onto the reorg branch, just because a lot of the suggested changes are also applicable to that one, and it's easier to make them all at once.
| queryErr := db.Select(&valueHistory, getFieldQuery) | ||
| Expect(queryErr).NotTo(HaveOccurred()) | ||
| Expect(len(valueHistory)).To(Equal(2)) | ||
| Expect(valueHistory[1].String).To(Equal(newColumnVal)) |
There was a problem hiding this comment.
Worth maybe asserting on indexes 0 and 1, just to clarify that the earlier block's val is set for both? Or is that not what's happening here?
There was a problem hiding this comment.
I like it! I originally left out assertions about the other record, just because in most cases they would still pass even if I hadn't written the triggers that these tests cover. But I think you're right that they could add clarity, and test that unwanted behavior isn't happening.
| Describe("updating history with new value", func() { | ||
| It("updates field in subsequent blocks", func() { | ||
| _, initialColumnVal := randomBidStorageValue(input.Metadata.Type, input.PackedValueType) | ||
| _, setupErr := db.Exec(insertFieldQuery, addressID, headerTwo.BlockNumber, bidID, initialColumnVal, timestampTwo) |
There was a problem hiding this comment.
Can we use repo.Create here? I sense the answer is probably no but I'm not really grokking why 🤔
There was a problem hiding this comment.
Ok think I'm making progress here 😅
So this inserts a record directly into the trigger table, which means it gets updated when an earlier record comes in and there's no corresponding row in the storage table for this val - is that right? Then the repo.Create approach behaves differently because there is a subsequent row on the storage table for that block?
There was a problem hiding this comment.
Yeah exactly, the trigger would find the initial record in the storage table, and thus not update the trigger-table at (or after) that block.
In general, I used straight-up db.Execs to model the concept of a trigger-table record-insertion having been triggered by a storage table that isn't the one being tested.
There was a problem hiding this comment.
Not sure, but do you think it would help to clarify if the test description included something like "updates field in subsequent blocks when no associated diff records"? That also may make things more complicated... I sometimes get too verbose with my test descriptions. 😊
| queryErr := db.Select(&valueHistory, getFieldQuery) | ||
| Expect(queryErr).NotTo(HaveOccurred()) | ||
| Expect(len(valueHistory)).To(Equal(2)) | ||
| Expect(valueHistory[1].String).To(Equal(initialColumnVal)) |
There was a problem hiding this comment.
I think it might be helpful to include assertions for both of the valueHistory records here
| Expect(valueHistory[1].String).To(Equal(initialColumnVal)) | ||
| }) | ||
|
|
||
| It("ignores rows from earlier blocks", func() { |
There was a problem hiding this comment.
I dig testing this case, but it makes me a little bit nervous that the db is cool with rows in the trigger table without corresponding records on the raw storage tables. I suppose this should maybe be addressed by the reorg safety story, but would be up for creating something new if not and/or we want to investigate further anchoring the trigger tables to underlying storage table rows
There was a problem hiding this comment.
Yeah, at least in theory the reorg triggers should preclude this scenario from ever happening in production data, but I like the idea of making a stronger guarantee. I'm just not sure how something like that would be implemented, aside from using a view instead of a trigger-table, which I think we decided against due to Postgres's lack of introspection functionality wrt views. Definitely open to the idea though, if we can find a good way of going about it.
| }) | ||
| } | ||
|
|
||
| func FlipBidSnapshotTriggerTests(input BidTriggerTestInput) { |
There was a problem hiding this comment.
Dig it - definitely a tricky balancing act here, but I'm in favor of prioritizing readability while we're already dealing with a shared test suite
| expectedBid := flipBidSnapshotFromValues(bidID, headerTwo.BlockNumber, addressID, headerTwo.Timestamp, | ||
| initialBidValues) | ||
| assertFlipSnapshot(bidSnapshots[1], expectedBid, headerTwo.BlockNumber) | ||
| assertSingleField(bidSnapshots[1], expectedBid, input.ColumnName) |
There was a problem hiding this comment.
Sorry to be a broken record here, but I'd be inclined to add assertions for both records whenever we expect 2 results
| type commonBidSnapshot struct { | ||
| AddressID string `db:"address_id"` | ||
| BlockNumber string `db:"block_number"` | ||
| BidID string `db:"bid_id"` | ||
| Guy sql.NullString | ||
| Tic sql.NullString | ||
| End sql.NullString | ||
| Lot sql.NullString | ||
| Bid sql.NullString | ||
| Updated string | ||
| } | ||
|
|
||
| type flipBidSnapshot struct { | ||
| commonBidSnapshot | ||
| Usr sql.NullString | ||
| Gal sql.NullString | ||
| Tab sql.NullString | ||
| } | ||
|
|
||
| func commonBidSnapshotFromValues(bidID int, blockNumber, addressID int64, updated string, bidValues map[string]interface{}) commonBidSnapshot { | ||
| parsedUpdated, parseErr := strconv.ParseInt(updated, 10, 64) | ||
| Expect(parseErr).NotTo(HaveOccurred()) | ||
| packedValues := bidValues[mcdStorage.Packed].(map[int]string) | ||
|
|
||
| return commonBidSnapshot{ | ||
| AddressID: strconv.FormatInt(addressID, 10), | ||
| BlockNumber: strconv.FormatInt(blockNumber, 10), | ||
| BidID: strconv.Itoa(bidID), | ||
| Guy: test_helpers.GetValidNullString(packedValues[0]), | ||
| Tic: test_helpers.GetValidNullString(packedValues[1]), | ||
| End: test_helpers.GetValidNullString(packedValues[2]), | ||
| Lot: test_helpers.GetValidNullString(bidValues[mcdStorage.BidLot].(string)), | ||
| Bid: test_helpers.GetValidNullString(bidValues[mcdStorage.BidBid].(string)), | ||
| Updated: FormatTimestamp(parsedUpdated), | ||
| } | ||
| } | ||
|
|
||
| func flipBidSnapshotFromValues(bidID int, blockNumber, addressID int64, updated string, bidValues map[string]interface{}) flipBidSnapshot { | ||
| return flipBidSnapshot{ | ||
| commonBidSnapshot: commonBidSnapshotFromValues(bidID, blockNumber, addressID, updated, bidValues), | ||
| Usr: test_helpers.GetValidNullString(bidValues[mcdStorage.BidUsr].(string)), | ||
| Gal: test_helpers.GetValidNullString(bidValues[mcdStorage.BidGal].(string)), | ||
| Tab: test_helpers.GetValidNullString(bidValues[mcdStorage.BidTab].(string)), | ||
| } | ||
| } | ||
|
|
||
| func assertBidSnapshot(actualBid, expectedBid commonBidSnapshot, expectedBlockNumber int64) { | ||
| Expect(actualBid.AddressID).To(Equal(expectedBid.AddressID)) | ||
| Expect(actualBid.BlockNumber).To(Equal(strconv.FormatInt(expectedBlockNumber, 10))) | ||
| Expect(actualBid.BidID).To(Equal(expectedBid.BidID)) | ||
| Expect(actualBid.Lot).To(Equal(expectedBid.Lot)) | ||
| Expect(actualBid.Bid).To(Equal(expectedBid.Bid)) | ||
| Expect(actualBid.Updated).To(Equal(expectedBid.Updated)) | ||
| } | ||
|
|
||
| func assertFlipSnapshot(actualFlip, expectedFlip flipBidSnapshot, expectedBlockNumber int64) { | ||
| assertBidSnapshot(actualFlip.commonBidSnapshot, expectedFlip.commonBidSnapshot, expectedBlockNumber) | ||
| Expect(actualFlip.Usr).To(Equal(expectedFlip.Usr)) | ||
| Expect(actualFlip.Gal).To(Equal(expectedFlip.Gal)) | ||
| Expect(actualFlip.Tab).To(Equal(expectedFlip.Tab)) | ||
| } |
There was a problem hiding this comment.
Do we have a story for this replacement?
| DataColumn event.ColumnName = "data" | ||
| DinkColumn event.ColumnName = "dink" | ||
| DstColumn event.ColumnName = "dst" | ||
| EndColumn event.ColumnName = "end" |
| CREATE FUNCTION get_latest_flip_bid_gal(bid_id numeric) RETURNS TEXT AS | ||
| CREATE FUNCTION flip_bid_usr_before_block(bid_id NUMERIC, address_id INTEGER, header_id INTEGER) RETURNS TEXT AS | ||
| $$ | ||
| SELECT usr |
There was a problem hiding this comment.
I feel like I should know this 😨 is usr a newish field on flip?
There was a problem hiding this comment.
Nice catch, I probably should've explained why I added this! The short answer is no, this isn't a new field.
The usr field isn't in the spec, and tbh I added it on a whim because it was very easy and I wanted the flip table's structure to match the Bid object in the solidity code. The only real benefit I see to keeping it, is that the usr is an urn identifier, so having it directly available in this table will make the urn computed column more performant, since it won't have to do a separate lookup to find the usr associated with a bid.
| -- +goose StatementEnd | ||
|
|
||
| -- +goose StatementBegin | ||
| CREATE OR REPLACE FUNCTION maker.update_flip_guys_until_next_diff(new_diff maker.flip_bid_guy) RETURNS VOID |
There was a problem hiding this comment.
Trying to make sure that I understand how this works, and am struggling a bit with the update... functions.
insert_new_flip_guy will insert a flip with the flip_bid_guy diff's guy value. And on conflict it will update the guy to the current diff's value. But, that current diff may not be the most recent recent diff, so then the update_flip_guys_until_next_diff function becomes important.
And so then in update_flip_guys_until_next_diff it looks like next_guy_diff_block will get a block number if there's a flip_bid_guy diff that is more recent than our current diff. The thing I'm a bit stuck on is that on line 219 it looks like we're allowing the flip record to be updated with the current diff's guy value, even though there is a more recent diff (i.e. the current block number is less than the next block number) - wouldn't that make the flip guy value out of date?
There was a problem hiding this comment.
The flip table is basically a collection of snapshots, each representing the state of a bid at a given block. So when a diff from the past is inserted, we need to update all the snapshots which would've been affected by the diff that was inserted. This trigger's job is to figure out which snapshots would be affected, and update them accordingly.
Using flip_bid_guy as an example: the only snapshots we care about updating are the ones from blocks in between the flip_bid_guy from the past (which was just inserted), and the next-most-recent flip_bid_guy diff. That's because the next one will essentially overwrite the guy field for that bid, but we still want a record of what the bid's state was before that later diff happened.
Hope that makes sense! Lmk if I can clarify anything.
There was a problem hiding this comment.
🙏 thank you, that makes sense.
elizabethengelman
left a comment
There was a problem hiding this comment.
nice work! thanks for all the explanation about the rationale too!
| Describe("updating history with new value", func() { | ||
| It("updates field in subsequent blocks", func() { | ||
| _, initialColumnVal := randomBidStorageValue(input.Metadata.Type, input.PackedValueType) | ||
| _, setupErr := db.Exec(insertFieldQuery, addressID, headerTwo.BlockNumber, bidID, initialColumnVal, timestampTwo) |
There was a problem hiding this comment.
Not sure, but do you think it would help to clarify if the test description included something like "updates field in subsequent blocks when no associated diff records"? That also may make things more complicated... I sometimes get too verbose with my test descriptions. 😊
6dbdf4a to
26538ef
Compare
This PR enables the auction (flip/flap/flop) trigger tables to react appropriately to diffs coming in out of order. Reorg-safety still has not been implemented.
All of flop/flap's triggers are just copy-pasted flip triggers with some columns removed, plus a
:%s/flip/flop/g(though flip has some triggers that the other two don't). So it might not be necessary for reviewers to go through all 3 migrations with a fine-tooth comb, since they're all almost identical.