Vdb 1294 update docker compose config and directions#192
Vdb 1294 update docker compose config and directions#192paytonrules merged 4 commits intostagingfrom
Conversation
rmulhol
left a comment
There was a problem hiding this comment.
Looking good! Happy to approve as long as we can verify the existing CD pipeline will continue working as expected with these changes
| The database needs to be initialized once before bringing up the system. That can be by running, from the root directory, | ||
|
|
||
| ```bash | ||
| docker-compose -f dockerfiles/docker-compose.yml run execute ./run_migrations.sh |
There was a problem hiding this comment.
Just to double check - have you run this locally and verified the migrations run before the containers run their own migrations? Asking because I've seen the migrations take a long time (~3 minutes) before
There was a problem hiding this comment.
Just double-checked and when that command is run locally it only starts the postgres image, and the execute image. The others aren't run, and the execute image actually shuts down when run_migration is done, because it overrides the startup command.
|
|
||
| ### Docker Setup | ||
|
|
||
| This repository provides a docker-compose setup to manage the multiple container-based deployment for you, if at all possible you should use that setup. |
There was a problem hiding this comment.
This seems like the right approach, but maybe worth mentioning the need to be sure the resources allocated to Docker are substantial - I think we'll hit lower limits really fast with the ethereum node included
| contract data via [event](./transformers/events) and [storage](./transformers/storage) transformers. The VulcanizeDB | ||
| repository includes a [general description](https://github.com/makerdao/vulcanizedb/blob/staging/documentation/custom-transformers.md) | ||
| about transformers. | ||
| 1. `execute` uses the raw Ethereum data that has been synced into Postgres and applies transformations to configured MCD contract data via [event](./transformers/events) and [storage](./transformers/storage) transformers. The VulcanizeDB repository includes a [general description](https://github.com/makerdao/vulcanizedb/blob/staging/documentation/custom-transformers.md) about transformers. |
There was a problem hiding this comment.
Maybe doesn't need to be part of this PR, but you now also need to run extractDiffs to get the storage diffs used by the storage transformers
There was a problem hiding this comment.
That makes a lot of sense - probably a separate PR?
| 1. Create a user for yourself that is able run migrations and add extensions. | ||
| 1. `createdb vulcanize_public` | ||
| 1. Migrating the database manually is unnecessary as the commands handle this. | ||
| 1. Migrate the database using the `make migrate` task in this repository. |
There was a problem hiding this comment.
fwiw I believe execute and composeAndExecute do handle running the migrations for you, but if you went that path you would need to be sure to run those commands first since the vulcanizedb migrations alone will moot out migrations in this repo.
There was a problem hiding this comment.
You're correct - I changed the directions for this reason. It might be good to re-warn the reader here of what will happen if you do things in the wrong order.
| echo "Could not run migrations. Are the database details correct?" | ||
| exit 1 | ||
| fi | ||
| source run_migrations.sh |
There was a problem hiding this comment.
So this will still run the migrations automatically, right? Just double checking we don't need to modify the CD pipeline with these changes
There was a problem hiding this comment.
It did locally. Is there a good CD pipeline test you can think of?
There was a problem hiding this comment.
I was able to build the execute Docker container and run execute, and sure enough it ran the migrations.
This allows running the migrations as a separate step, and a one-off before running docker-compose up.
Running it this way should just work.
d56d8e8 to
180bfd5
Compare
Extract run-migrations from startup-script, and then reference the docker-compose setup in the directions first.
This PR has a bit of a chicken-and-egg problem as the Docker image needs to be updated before the directions are correct. I validated this using a local image, but that's not a perfect setup.