Skip to content

(VDB-1132) Async diff arrival#94

Merged
gslaughl merged 3 commits intostagingfrom
vdb-1132-async-diff-arrival
Jan 22, 2020
Merged

(VDB-1132) Async diff arrival#94
gslaughl merged 3 commits intostagingfrom
vdb-1132-async-diff-arrival

Conversation

@gslaughl
Copy link
Copy Markdown
Contributor

This PR removes the implicit assumption that kick events will arrive before storage diffs. Now the data should stay up-to-date regardless of whether events or diffs are transformed first (though there is still a dependency on diffs arriving in order).

var flipKickModel event.InsertionModel

BeforeEach(func() {
flipKickModel = test_data.FlipKickModel()
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kind of annoying how we can't do this on the same line where we invoke the shared suite, but the problem is that there are some Expects in the CopyModel function, which break if they're not called from within an It or setup/teardown function.

(SELECT get_latest_flip_bid_tab(NEW.bid_id)),
INTO maker.flip (bid_id, address_id, block_number, guy, tic, "end", lot, bid, gal, tab, updated, created)
VALUES (NEW.bid_id, NEW.address_id, (SELECT block_number FROM diff_block), NEW.guy,
get_latest_flip_bid_tic(NEW.bid_id),
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe I should have done some of this reformatting/refactoring in a separate commit 😅 but the only behavior change in all these insert_updated_x trigger functions are cascading changes from updating the flip table. I.e. removing block_hash and updating the PK.

Copy link
Copy Markdown
Contributor

@rmulhol rmulhol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks awesome!

Just to make sure I understand - this story means that we can have events/storage diffs arrive out of order and preserve data consistency, but we need additional work to enable reorg safety. Is that right? Also, do we need a separate story to gracefully handle diffs arriving out of order (e.g. if we got diffs from block 7, 1, 5, 3 - in that order)?

ON CONFLICT (bid_id, block_number) DO UPDATE SET guy = NEW.guy;
return NEW;
flip_bid_time_created(NEW.address_id, NEW.bid_id))
ON CONFLICT (block_number, bid_id, address_id) DO UPDATE SET guy = NEW.guy;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems important 👍

return NEW;
flip_bid_time_created(NEW.address_id, NEW.bid_id))
ON CONFLICT (block_number, bid_id, address_id) DO UPDATE SET lot = NEW.lot;
RETURN NEW;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious - why is RETURN NEW necessary? Seems like this function could just return void

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the docs

A trigger function must return either NULL or a record/row value having exactly the structure of the table the trigger was fired for.

So RETURN NULL would work just as well here (though I don't think it would work to omit the return entirely). I don't know of any reason to pick one over the other, so I randomly settled on NEW. 🤷‍♂

RETURN NEW;
END
$$
LANGUAGE plpgsql;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like this is a legit use of plpgsql (and not added by this PR), but I was mildly surprised to see that sql is preferred for performance - just dropping here for knowledge sharing's sake

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to know!

func TestTriggers(t *testing.T) {
RegisterFailHandler(Fail)
RunSpecs(t, "Trigger Suite")
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this suite emit any logs when you run make test from the cli?

Copy link
Copy Markdown
Contributor Author

@gslaughl gslaughl Jan 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that I look again, I guess event.PersistModels can emit logs under the right conditions. I don't think we're likely to encounter those conditions, but I guess I might as well make sure they get discarded just in case

})

It("generates log note signature", func() {
It("generates log value signature", func() {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥇

@gslaughl
Copy link
Copy Markdown
Contributor Author

@rmulhol Yeah exactly, this PR does not include the changes that will allow diffs to come in out of order-- that's covered by VDB-1144, which is in progress right now.

@gslaughl gslaughl force-pushed the vdb-1132-async-diff-arrival branch from 6c09602 to a28c6fd Compare January 22, 2020 22:27
@gslaughl gslaughl merged commit 14ad297 into staging Jan 22, 2020
@gslaughl gslaughl deleted the vdb-1132-async-diff-arrival branch January 22, 2020 22:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants