Fix default connection handling with three-tier config by eileencodes · Pull Request #32271 · rails/rails

eileencodes · 2018-03-16T15:51:56Z

If you had a three-tier config, the establish_connection that's called
in the Railtie on load can't figure out how to access the default
configuration.

This is because Rails assumes that the config is the first value in the
hash and always associated with the key from the environment. With a
three tier config however we need to go one level deeper.

This commit includes 2 changes. 1) removes a line from resolve_all
which was parsing out the the environment from the config so instead of
getting

{
  :development => {
    :primary => {
      :database => "whatever"
    }
  },
    :animals => {
      :database => "whatever-animals"
    }
  },
  etc with test / prod
}

We'd instead end up with a config that had no attachment to it's
envioronment.

{
  :primary => {
    :database => "whatever"
  }
  :animals => {
    :database => "whatever-animals"
  }
  etc - without test and prod
}

Not only did this mean that Active Record didn't know how to establish a
connection, it didn't have the other necessary configs along with it in
the configs list.

So fix this I removed the line that deletes these configs.

The second thing this commit changes is adding this line to
establish_connection

spec = spec[spec_name.to_sym] if spec[spec_name.to_sym]

When you have a three-tier config and don't pass any hash/symbol/env etc
to establish_connection the resolver will automatically return both
the primary and secondary (in this case animals db) configurations.
We'll get an database configuration does not specify adapter error
because AR will try to establish a connection on the primary key
rather than the primary key's config. It assumes that the
development or default env automatically will return a config hash,
but with a three-tier config we actually get a key and config primary => config.

This fix is a bit of a bandaid because it's not the "correct" way to
handle this situation, but it does solve our immediate problem. The new
code here is saying "if the config returned from the resolver (I know
it's called spec in here but we interchange our meanings a LOT and what
is returned is a three-tier config) has a key matching the "primary"
spec name, grab the config from the spec and pass that to the
estalbish_connection method".

This works because if we pass :animals or a hash, or :primary we'll
already have the correct configuration to connect with.

This fixes the case where we want Rail to connect with the default
connection.

Coming soon is a refactoring that should eliminate the need to do this
but I need this fix in order to write the multi-db rake tasks that I
promised in my RailsConf submission. @tenderlove and I are working on
the refactoring of the internals for connection management but it won't
be ready for a few weeks and this issue has been blocking progress.

cc/ @tenderlove

If you had a three-tier config, the `establish_connection` that's called in the Railtie on load can't figure out how to access the default configuration. This is because Rails assumes that the config is the first value in the hash and always associated with the key from the environment. With a three tier config however we need to go one level deeper. This commit includes 2 changes. 1) removes a line from `resolve_all` which was parsing out the the environment from the config so instead of getting ``` { :development => { :primary => { :database => "whatever" } }, :animals => { :database => "whatever-animals" } }, etc with test / prod } ``` We'd instead end up with a config that had no attachment to it's envioronment. ``` { :primary => { :database => "whatever" } :animals => { :database => "whatever-animals" } etc - without test and prod } ``` Not only did this mean that Active Record didn't know how to establish a connection, it didn't have the other necessary configs along with it in the configs list. So fix this I removed the line that deletes these configs. The second thing this commit changes is adding this line to `establish_connection` ``` spec = spec[spec_name.to_sym] if spec[spec_name.to_sym] ``` When you have a three-tier config and don't pass any hash/symbol/env etc to `establish_connection` the resolver will automatically return both the primary and secondary (in this case animals db) configurations. We'll get an `database configuration does not specify adapter` error because AR will try to establish a connection on the `primary` key rather than the `primary` key's config. It assumes that the `development` or default env automatically will return a config hash, but with a three-tier config we actually get a key and config `primary => config`. This fix is a bit of a bandaid because it's not the "correct" way to handle this situation, but it does solve our immediate problem. The new code here is saying "if the config returned from the resolver (I know it's called spec in here but we interchange our meanings a LOT and what is returned is a three-tier config) has a key matching the "primary" spec name, grab the config from the spec and pass that to the estalbish_connection method". This works because if we pass `:animals` or a hash, or `:primary` we'll already have the correct configuration to connect with. This fixes the case where we want Rail to connect with the default connection. Coming soon is a refactoring that should eliminate the need to do this but I need this fix in order to write the multi-db rake tasks that I promised in my RailsConf submission. `@tenderlove` and I are working on the refactoring of the internals for connection management but it won't be ready for a few weeks and this issue has been blocking progress.

In a three-tier config environment `configurations[environment].presence` will return `{ :primary => { :key => value, :key => value }, :secondary => { :key => value, :key => value} }, which means it's not given a single config to connect to. If we flip these however it will connect to primary because that's the default connection, and on a two tier it will be `nil` so the code will select the connection from the configurations rather than the connection.

bogdanvlviv · 2018-03-17T16:05:24Z

activerecord/test/cases/connection_adapters/connection_handler_test.rb

+
+          ActiveRecord::Base.establish_connection
+
+          assert_equal "db/primary.sqlite3", ActiveRecord::Base.connection.pool.spec.config[:database]


ActiveRecord::Base.connection.pool.spec.config[:database] creates extra rails/activerecord/db/primary.sqlite3 and this test fails if run via bin/test, the same problem with the test that below in this file.

What the what. This works fine with bundle exec ruby -Ilib:test test/cases/connection_adapters/connection_handler_test.rb 🤔

Anyway there's another bug in here that I need to look at and will fix on Monday.

Creating extra rails/activerecord/db/primary.sqlite3 was fixed in #32400.

@tenderlove

While the three-tier config makes it easier to define databases for multiple database applications, it quickly became clear to offer full support for multiple databases we need to change the way the connections hash was handled. A three-tier config means that when Rails needed to choose a default configuration (in the case a user doesn't ask for a specific configuration) it wasn't clear to Rails which the default was. I [bandaid fixed this so the rake tasks could work](rails#32271) but that fix wasn't correct because it actually doubled up the configuration hashes. Instead of attemping to manipulate the hashes @tenderlove and I decided that it made more sense if we converted the hashes to objects so we can easily ask those object questions. In a three tier config like this: ``` development: primary: database: "my_primary_db" animals: database; "my_animals_db" ``` We end up with an object like this: ``` @Configurations=[ #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbded10 @env_name="development",@spec_name="primary", @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>, #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbdea90 @env_name="development",@spec_name="animals", @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}> ]> ``` The configurations setter takes the database configuration set by your application and turns them into an `ActiveRecord::DatabaseConfigurations` object that has one getter - `@configurations` which is an array of all the database objects. The configurations getter returns this object by default since it acts like a hash in most of the cases we need. For example if you need to access the default `development` database we can simply request it as we did before: ``` ActiveRecord::Base.configurations["development"] ``` This will return primary development database configuration hash: ``` { "database" => "my_primary_db" } ``` Internally all of Active Record has been converted to use the new objects. I've built this to be backwards compatible but allow for accessing the hash if needed for a deprecation period. To get the original hash instead of the object you can either add `to_h` on the configurations call or pass `legacy: true` to `configurations. ``` ActiveRecord::Base.configurations.to_h => { "development => { "database" => "my_primary_db" } } ActiveRecord::Base.configurations(legacy: true) => { "development => { "database" => "my_primary_db" } } ``` The new configurations object allows us to iterate over the Active Record configurations without losing the known environment or specification name for that configuration. You can also select all the configs for an env or env and spec. With this we can always ask any object what environment it belongs to: ``` db_configs = ActiveRecord::Base.configurations.configurations_for("development") => #<ActiveRecord::DatabaseConfigurations:0x00007fd1acbdf800 @Configurations=[ #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbded10 @env_name="development",@spec_name="primary", @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>, #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbdea90 @env_name="development",@spec_name="animals", @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}> ]> db_config.env_name => "development" db_config.spec_name => "primary" db_config.config => { "adapter"=>"sqlite3", "database"=>"db/development.sqlite3" } ``` The configurations object is more flexible than the configurations hash and will allow us to build on top of the connection management in order to add support for primary/replica connections, sharding, and constructing queries for associations that live in multiple databases.

@tenderlove

While the three-tier config makes it easier to define databases for multiple database applications, it quickly became clear to offer full support for multiple databases we need to change the way the connections hash was handled. A three-tier config means that when Rails needed to choose a default configuration (in the case a user doesn't ask for a specific configuration) it wasn't clear to Rails which the default was. I [bandaid fixed this so the rake tasks could work](rails#32271) but that fix wasn't correct because it actually doubled up the configuration hashes. Instead of attemping to manipulate the hashes @tenderlove and I decided that it made more sense if we converted the hashes to objects so we can easily ask those object questions. In a three tier config like this: ``` development: primary: database: "my_primary_db" animals: database; "my_animals_db" ``` We end up with an object like this: ``` @Configurations=[ #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbded10 @env_name="development",@spec_name="primary", @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>, #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbdea90 @env_name="development",@spec_name="animals", @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}> ]> ``` The configurations setter takes the database configuration set by your application and turns them into an `ActiveRecord::DatabaseConfigurations` object that has one getter - `@configurations` which is an array of all the database objects. The configurations getter returns this object by default since it acts like a hash in most of the cases we need. For example if you need to access the default `development` database we can simply request it as we did before: ``` ActiveRecord::Base.configurations["development"] ``` This will return primary development database configuration hash: ``` { "database" => "my_primary_db" } ``` Internally all of Active Record has been converted to use the new objects. I've built this to be backwards compatible but allow for accessing the hash if needed for a deprecation period. To get the original hash instead of the object you can either add `to_h` on the configurations call or pass `legacy: true` to `configurations. ``` ActiveRecord::Base.configurations.to_h => { "development => { "database" => "my_primary_db" } } ActiveRecord::Base.configurations(legacy: true) => { "development => { "database" => "my_primary_db" } } ``` The new configurations object allows us to iterate over the Active Record configurations without losing the known environment or specification name for that configuration. You can also select all the configs for an env or env and spec. With this we can always ask any object what environment it belongs to: ``` db_configs = ActiveRecord::Base.configurations.configurations_for("development") => #<ActiveRecord::DatabaseConfigurations:0x00007fd1acbdf800 @Configurations=[ #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbded10 @env_name="development",@spec_name="primary", @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>, #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbdea90 @env_name="development",@spec_name="animals", @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}> ]> db_config.env_name => "development" db_config.spec_name => "primary" db_config.config => { "adapter"=>"sqlite3", "database"=>"db/development.sqlite3" } ``` The configurations object is more flexible than the configurations hash and will allow us to build on top of the connection management in order to add support for primary/replica connections, sharding, and constructing queries for associations that live in multiple databases.

@tenderlove

While the three-tier config makes it easier to define databases for multiple database applications, it quickly became clear to offer full support for multiple databases we need to change the way the connections hash was handled. A three-tier config means that when Rails needed to choose a default configuration (in the case a user doesn't ask for a specific configuration) it wasn't clear to Rails which the default was. I [bandaid fixed this so the rake tasks could work](rails#32271) but that fix wasn't correct because it actually doubled up the configuration hashes. Instead of attemping to manipulate the hashes @tenderlove and I decided that it made more sense if we converted the hashes to objects so we can easily ask those object questions. In a three tier config like this: ``` development: primary: database: "my_primary_db" animals: database; "my_animals_db" ``` We end up with an object like this: ``` @Configurations=[ #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbded10 @env_name="development",@spec_name="primary", @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>, #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbdea90 @env_name="development",@spec_name="animals", @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}> ]> ``` The configurations setter takes the database configuration set by your application and turns them into an `ActiveRecord::DatabaseConfigurations` object that has one getter - `@configurations` which is an array of all the database objects. The configurations getter returns this object by default since it acts like a hash in most of the cases we need. For example if you need to access the default `development` database we can simply request it as we did before: ``` ActiveRecord::Base.configurations["development"] ``` This will return primary development database configuration hash: ``` { "database" => "my_primary_db" } ``` Internally all of Active Record has been converted to use the new objects. I've built this to be backwards compatible but allow for accessing the hash if needed for a deprecation period. To get the original hash instead of the object you can either add `to_h` on the configurations call or pass `legacy: true` to `configurations. ``` ActiveRecord::Base.configurations.to_h => { "development => { "database" => "my_primary_db" } } ActiveRecord::Base.configurations(legacy: true) => { "development => { "database" => "my_primary_db" } } ``` The new configurations object allows us to iterate over the Active Record configurations without losing the known environment or specification name for that configuration. You can also select all the configs for an env or env and spec. With this we can always ask any object what environment it belongs to: ``` db_configs = ActiveRecord::Base.configurations.configurations_for("development") => #<ActiveRecord::DatabaseConfigurations:0x00007fd1acbdf800 @Configurations=[ #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbded10 @env_name="development",@spec_name="primary", @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}>, #<ActiveRecord::DatabaseConfigurations::HashConfig:0x00007fd1acbdea90 @env_name="development",@spec_name="animals", @config={"adapter"=>"sqlite3", "database"=>"db/development.sqlite3"}> ]> db_config.env_name => "development" db_config.spec_name => "primary" db_config.config => { "adapter"=>"sqlite3", "database"=>"db/development.sqlite3" } ``` The configurations object is more flexible than the configurations hash and will allow us to build on top of the connection management in order to add support for primary/replica connections, sharding, and constructing queries for associations that live in multiple databases.

eileencodes added the activerecord label Mar 16, 2018

eileencodes added 2 commits March 16, 2018 12:28

eileencodes force-pushed the fix-three-tier-default-connection branch from c524f9b to 4027643 Compare March 16, 2018 16:43

eileencodes merged commit 9700dac into rails:master Mar 16, 2018

eileencodes deleted the fix-three-tier-default-connection branch March 16, 2018 19:18

eileencodes mentioned this pull request Mar 16, 2018

Part 1 Easy Multi db in Rails: Add basic rake tasks for multi db setup #32274

Merged

bogdanvlviv reviewed Mar 17, 2018

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix default connection handling with three-tier config#32271

Fix default connection handling with three-tier config#32271
eileencodes merged 2 commits intorails:masterfrom
eileencodes:fix-three-tier-default-connection

eileencodes commented Mar 16, 2018

Uh oh!

bogdanvlviv Mar 17, 2018

Uh oh!

eileencodes Mar 17, 2018

Uh oh!

eileencodes Mar 17, 2018

Uh oh!

bogdanvlviv Apr 3, 2018 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		ActiveRecord::Base.establish_connection

		assert_equal "db/primary.sqlite3", ActiveRecord::Base.connection.pool.spec.config[:database]

Conversation

eileencodes commented Mar 16, 2018

Uh oh!

bogdanvlviv Mar 17, 2018

Choose a reason for hiding this comment

Uh oh!

eileencodes Mar 17, 2018

Choose a reason for hiding this comment

Uh oh!

eileencodes Mar 17, 2018

Choose a reason for hiding this comment

Uh oh!

bogdanvlviv Apr 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bogdanvlviv Apr 3, 2018 •

edited

Loading