Skip to content

Automatic-er database initialization.#4887

Merged
martenson merged 1 commit intogalaxyproject:devfrom
jmchilton:database_initialization
Nov 1, 2017
Merged

Automatic-er database initialization.#4887
martenson merged 1 commit intogalaxyproject:devfrom
jmchilton:database_initialization

Conversation

@jmchilton
Copy link
Member

  • Use sqlalchemy-utils to create postgres / mysql database if they don't exist the way it happens automatically for sqlite.
  • Add a new configuration parameter called database_template that allows creating postgres databases from an existing database using a TEMPLATE statement.
  • Use the database template parameter to give each integration test case class a clean postgres database.

@jmchilton jmchilton added area/database Galaxy's database or data access layer kind/enhancement area/testing labels Oct 27, 2017
@jmchilton
Copy link
Member Author

Hopefully something like this allows us to fix failing integration tests on the job handler and collection state PRs that work when you run them alone.

from sqlalchemy.orm.collections import attribute_mapped_collection
from sqlalchemy.types import BigInteger

from sqlalchemy_utils import database_exists, create_database
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Import order

@galaxybot galaxybot added this to the 18.01 milestone Oct 27, 2017
- Use sqlalchemy-utils to create postgres / mysql database if they don't exist the way it happens automatically for sqlite.
- Add a new configuration parameter called database_template that allows creating postgres databases from an existing database using a TEMPLATE statement.
- Use the database template parameter to give each integration test case class a clean postgres database.
@jmchilton jmchilton force-pushed the database_initialization branch from 38dc0fb to 9b90999 Compare October 27, 2017 20:03
@jmchilton
Copy link
Member Author

@nsoranzo Thanks I fixed that and another linting problem.

jmchilton added a commit to jmchilton/galaxy that referenced this pull request Oct 30, 2017
- Bake Selenium (chrome, chrome-wrapper, xvfb, and Selenium server) into Docker test image. (1)
- Correct usage of common_startup.sh to fetch dev wheels (don't use pip directly).
- Fix/enhance the test wrapper file to not set GALAXY_CONFIG_OVERRIDE_DATABASE_CONNECATION - this prevents integration database templating (galaxyproject#4887) from working with the Docker image.
- Update target Dockerfile.

1) The Docker compose setup for Selenium tests currently used by Jenkins is nice in that it uses official Selenium and Postgres Docker images - but it is problematic in that it takes more resources than the other single container Docker jobs, it doesn't cleanup as nicely, it doesn't have a pre-migrated database or pre-installed dependencies so it takes longer to run, and isn't easily exposed to developers via run_tests.sh. A more pragmatic Jenkins job based on this work should be runnable for every PR - the current variant simply takes too long and requires too many resources.
@martenson martenson merged commit 6b68c5d into galaxyproject:dev Nov 1, 2017
jmchilton added a commit to jmchilton/planemo that referenced this pull request Feb 23, 2018
Add a new profile database type option "postgres_docker". The existing "postgres" type option expects a local database to be accessible via the "psql" command on Planemo's PATH - probably few people have postgres running directly on their development machine and fewer still have it configured to allow passwordless entry. This newer variant is a much simpler interface - it starts a named container for the official Docker postgres image and uses that to manage the profile. This should make using profiles easier for most developers.

This new modality does have some shortcomings - it should probably be redone to persist the database somewhere -or- start and stop the same container (currently when the container is stopped the data is gone).

This also updates the workflow test that encounters Galaxy locks to pickup the environment variable PLANEMO_TEST_WORKFLOW_RUN_PROFILE and PLANEMO_TEST_WORKFLOW_RUN_PROFILE_DATABASE_TYPE and use these to create and use a profile during testing if set as well as setting that variable in Travis to leverage the postgres service already configured for other tests. This more less fixes what I had in mind with galaxyproject#772 but it would be nice to leverage galaxyproject/galaxy#4887 for greater test isolation.
jmchilton added a commit to jmchilton/planemo that referenced this pull request Feb 23, 2018
Add a new profile database type option "postgres_docker". The existing "postgres" type option expects a local database to be accessible via the "psql" command on Planemo's PATH - probably few people have postgres running directly on their development machine and fewer still have it configured to allow passwordless entry. This newer variant is a much simpler interface - it starts a named container for the official Docker postgres image and uses that to manage the profile. This should make using profiles easier for most developers.

This new modality does have some shortcomings - it should probably be redone to persist the database somewhere -or- start and stop the same container (currently when the container is stopped the data is gone).

This also updates the workflow test that encounters Galaxy locks to pickup the environment variable PLANEMO_TEST_WORKFLOW_RUN_PROFILE and PLANEMO_TEST_WORKFLOW_RUN_PROFILE_DATABASE_TYPE and use these to create and use a profile during testing if set as well as setting that variable in Travis to leverage the postgres service already configured for other tests. This more less fixes what I had in mind with galaxyproject#772 but it would be nice to leverage galaxyproject/galaxy#4887 for greater test isolation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/database Galaxy's database or data access layer area/testing kind/enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants