(VDB-1045) Isolate execute container by rmulhol · Pull Request #45 · sky-ecosystem/vdb-mcd-transformers

rmulhol · 2019-12-06T17:57:56Z

Remove headerSync so it can run in a separate container

This includes a change to point at vdb staging and makes associated changes to the deps, which is necessary to use the new storage repository interface. I'd be fine with cutting a new version of vdb whenever we're ready (potentially awaiting open PRs over there) and then updating things here accordingly.

When I run this locally with a config file that includes storage transformers but without a valid path to the CSV, "Waiting for to appear" spams the logs. Thinking that shouldn't be an issue when actually running this because you'd either include a valid path or turn off the storage transformers.

m0ar

:yay:

m0ar · 2019-12-09T08:18:39Z

plugins/transformerExporter.go

-			vow.StorageTransformerInitializer,
-		},
-		[]interface1.ContractTransformerInitializer{}
+	return []interface1.EventTransformerInitializer{yank.EventTransformerInitializer, bite.EventTransformerInitializer, cat_file_flip.EventTransformerInitializer, cat_file_vow.EventTransformerInitializer, jug_file_vow.EventTransformerInitializer, vat_fork.EventTransformerInitializer, vat_heal.EventTransformerInitializer, flop_kick.EventTransformerInitializer, vat_file_ilk.EventTransformerInitializer, vow_file.EventTransformerInitializer, flap_kick.EventTransformerInitializer, new_cdp.EventTransformerInitializer, log_value.EventTransformerInitializer, pot_file_dsr.EventTransformerInitializer, deal.EventTransformerInitializer, dent.EventTransformerInitializer, jug_drip.EventTransformerInitializer, jug_file_base.EventTransformerInitializer, vat_grab.EventTransformerInitializer, vat_move.EventTransformerInitializer, vow_fess.EventTransformerInitializer, pot_cage.EventTransformerInitializer, spot_file_mat.EventTransformerInitializer, spot_poke.EventTransformerInitializer, tick.EventTransformerInitializer, vat_frob.EventTransformerInitializer, vat_flux.EventTransformerInitializer, vat_fold.EventTransformerInitializer, cat_file_chop_lump.EventTransformerInitializer, spot_file_pip.EventTransformerInitializer, tend.EventTransformerInitializer, vat_init.EventTransformerInitializer, vat_slip.EventTransformerInitializer, flip_kick.EventTransformerInitializer, jug_file_ilk.EventTransformerInitializer, pot_file_vow.EventTransformerInitializer, vat_file_debt_ceiling.EventTransformerInitializer, vow_flog.EventTransformerInitializer, jug_init.EventTransformerInitializer, vat_suck.EventTransformerInitializer}, []interface1.StorageTransformerInitializer{cdp_manager.StorageTransformerInitializer, flop_storage.StorageTransformerInitializer, spot.StorageTransformerInitializer, jug.StorageTransformerInitializer, vow.StorageTransformerInitializer, bat_flip.StorageTransformerInitializer, eth_flip_a.StorageTransformerInitializer, vat.StorageTransformerInitializer, cat.StorageTransformerInitializer, flap_storage.StorageTransformerInitializer}, []interface1.ContractTransformerInitializer{}


I'm kind of in favor of having each of these transformer initializers on a new line so it's easier to see what has been included, though it does make the file a lot longer. 🤷‍♀ But, don't feel super strongly either way.

Yeah happy to add line breaks- do y'all know a way to automate that? I just copied over the file generated by ./vulcanizedb compose

Oh right it's the generated exporter, no idea. No biggie either.

m0ar · 2019-12-09T08:19:15Z

environments/docker.toml

        "log_value",
        "pot_cage",
        "pot_file_dsr",
+        "pot_file_vow",


What's this got to do with this PR? 🤔

I think this PR just needs a rebase - these pot_file_vow changes were merged into staging already

elizabethengelman

LGTM! 💯

Mostly just questions out of curiosity.

elizabethengelman · 2019-12-11T16:31:30Z

environments/docker.toml

        "log_value",
        "pot_cage",
        "pot_file_dsr",
+        "pot_file_vow",


I think this PR just needs a rebase - these pot_file_vow changes were merged into staging already

elizabethengelman · 2019-12-11T16:33:47Z

plugins/transformerExporter.go

-			vow.StorageTransformerInitializer,
-		},
-		[]interface1.ContractTransformerInitializer{}
+	return []interface1.EventTransformerInitializer{yank.EventTransformerInitializer, bite.EventTransformerInitializer, cat_file_flip.EventTransformerInitializer, cat_file_vow.EventTransformerInitializer, jug_file_vow.EventTransformerInitializer, vat_fork.EventTransformerInitializer, vat_heal.EventTransformerInitializer, flop_kick.EventTransformerInitializer, vat_file_ilk.EventTransformerInitializer, vow_file.EventTransformerInitializer, flap_kick.EventTransformerInitializer, new_cdp.EventTransformerInitializer, log_value.EventTransformerInitializer, pot_file_dsr.EventTransformerInitializer, deal.EventTransformerInitializer, dent.EventTransformerInitializer, jug_drip.EventTransformerInitializer, jug_file_base.EventTransformerInitializer, vat_grab.EventTransformerInitializer, vat_move.EventTransformerInitializer, vow_fess.EventTransformerInitializer, pot_cage.EventTransformerInitializer, spot_file_mat.EventTransformerInitializer, spot_poke.EventTransformerInitializer, tick.EventTransformerInitializer, vat_frob.EventTransformerInitializer, vat_flux.EventTransformerInitializer, vat_fold.EventTransformerInitializer, cat_file_chop_lump.EventTransformerInitializer, spot_file_pip.EventTransformerInitializer, tend.EventTransformerInitializer, vat_init.EventTransformerInitializer, vat_slip.EventTransformerInitializer, flip_kick.EventTransformerInitializer, jug_file_ilk.EventTransformerInitializer, pot_file_vow.EventTransformerInitializer, vat_file_debt_ceiling.EventTransformerInitializer, vow_flog.EventTransformerInitializer, jug_init.EventTransformerInitializer, vat_suck.EventTransformerInitializer}, []interface1.StorageTransformerInitializer{cdp_manager.StorageTransformerInitializer, flop_storage.StorageTransformerInitializer, spot.StorageTransformerInitializer, jug.StorageTransformerInitializer, vow.StorageTransformerInitializer, bat_flip.StorageTransformerInitializer, eth_flip_a.StorageTransformerInitializer, vat.StorageTransformerInitializer, cat.StorageTransformerInitializer, flap_storage.StorageTransformerInitializer}, []interface1.ContractTransformerInitializer{}


I'm kind of in favor of having each of these transformer initializers on a new line so it's easier to see what has been included, though it does make the file a lot longer. 🤷‍♀ But, don't feel super strongly either way.

elizabethengelman · 2019-12-11T16:43:53Z

Dockerfile


 RUN apk --update --no-cache add make git g++ linux-headers
-# DEBUG
-RUN apk add busybox-extras


just out of curiosity - do you know what we need this package for in headerSync?

Which package are you referring to? I removed busybox-extras bc it didn't seem necessary

Yeah, busybox-extras - I just wasn't sure what it was even. But I'm up for ✂️ it if it's not needed.

I think it's helpful tools for tinkering inside the container. May be worth re-adding if we're shing in there a lot to mess around, but I'm hoping we can set-it-and-forget-it with minimal interference

elizabethengelman · 2019-12-11T16:49:23Z

Dockerfile

 WORKDIR /go/src/github.com/makerdao
 RUN git clone https://github.com/makerdao/vulcanizedb.git
 WORKDIR /go/src/github.com/makerdao/vulcanizedb
-RUN git checkout v0.0.10


I'm trying to think if we'd need any sort of change to our workflow with pointing at VDB staging, in case we make a change to VDB that's incompatible with mcd transformers. e.g. the situation we'd be in if sky-ecosystem/vulcanizedb#8 gets merged in before #52

yeah good point. I think this should probably be pointing at a release once a new one is cut. Maybe it would make sense to add an arg that you pass at build time to specify the branch/release of vdb you want to target?

👍 I like that idea.

elizabethengelman · 2019-12-11T16:53:10Z

Dockerfile

-
-# setup environment
-ENV GOPATH $HOME/go
-ENV GO111MODULE on


how come we're able to remove these ENV variables? Don't we still need to make sure that the GOPATH is set properly, and go mods are on?

Don't need them since we built the plugin in the builder container, so this container doesn't need to peruse the GOPATH or load modules

elizabethengelman · 2019-12-11T16:55:48Z

Dockerfile

-# chown first so dir is writable
-# note: using $USER is merged, but not in the stable release yet
-COPY --chown=5000:5000 --from=builder /go/src/github.com/makerdao/vulcanizedb .
-COPY --chown=5000:5000 --from=builder /go/src/github.com/makerdao/vdb-mcd-transformers /go/src/github.com/makerdao/vdb-mcd-transformers


also wondering how we're able to remove these lines, specifically why we're able to remove the --chown

With the plugin already built, we don't need write access to create an .so file or assemble migrations

elizabethengelman · 2019-12-13T16:43:00Z

Wondering if this may require an update to the Docker README too?

rmulhol · 2019-12-13T22:03:20Z

Wondering if this may require an update to the Docker README too?

Done 🎉

- Remove headerSync so it can run in a separate container

rmulhol requested review from elizabethengelman, gslaughl, m0ar and yaoandrew December 6, 2019 17:57

m0ar approved these changes Dec 9, 2019

View reviewed changes

elizabethengelman approved these changes Dec 11, 2019

View reviewed changes

rmulhol force-pushed the vdb-1045-isolate-execute-container branch 3 times, most recently from 29b2f76 to b133058 Compare December 11, 2019 19:42

rmulhol force-pushed the vdb-1045-isolate-execute-container branch from b133058 to 23705d9 Compare December 13, 2019 22:00

rmulhol force-pushed the vdb-1045-isolate-execute-container branch 2 times, most recently from ba41e9e to 1bd9ff3 Compare December 18, 2019 20:10

(VDB-1045) Isolate execute container

1dbc59b

- Remove headerSync so it can run in a separate container

rmulhol force-pushed the vdb-1045-isolate-execute-container branch from 1bd9ff3 to 1dbc59b Compare December 19, 2019 23:35

rmulhol merged commit 6045865 into staging Dec 19, 2019

rmulhol deleted the vdb-1045-isolate-execute-container branch December 19, 2019 23:48

Conversation

rmulhol commented Dec 6, 2019

Uh oh!

m0ar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rmulhol Dec 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elizabethengelman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elizabethengelman Dec 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elizabethengelman commented Dec 13, 2019

Uh oh!

rmulhol commented Dec 13, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rmulhol Dec 11, 2019 •

edited

Loading

elizabethengelman Dec 13, 2019 •

edited

Loading