Coming Soon

The Local Setup is used as the the baseline, its a postgres running in docker on your local machine.

Setup your python environment (just a note Dataflow from google only supports Python 3.5)

pyenv virtualenv -f 3.5.9 venv
pyenv activate venv

pip install -r requirements.txt

Start a Postgres instance

docker-compose -f docker/docker-compose.yaml up -d

once up and running you can run:

./environments/local/run.sh

this will run the pipeline twice, once with new data and once to update the existing data,

you should get output as such:

First Run
#############################################
...
INFO:root:Run Time => 0:00:53.553058
#############################################
Second Run With Updates
#############################################
...
INFO:root:Run Time => 0:00:51.237524

Requires a Google Cloud Subscription