Skip to content

Allow overriding SQLite defaults from database.yml#50460

Merged
dhh merged 3 commits intorails:mainfrom
fractaledmind:ar-overridable-sqlite-pragmas
Jan 2, 2024
Merged

Allow overriding SQLite defaults from database.yml#50460
dhh merged 3 commits intorails:mainfrom
fractaledmind:ar-overridable-sqlite-pragmas

Conversation

@fractaledmind
Copy link
Contributor

Any PRAGMA configuration set under the pragmas key in the configuration file take precedence over Rails' defaults, and additional PRAGMAs can be set as well.

database: storage/development.sqlite3
timeout: 5000
pragmas:
  journal_mode: off
  temp_store: memory

Motivation / Background

This is an alternative to #49691, following @matthewd's suggestions.

Detail

This Pull Request allows the 7.1 default SQLite configuration to be overridden via values set under the pragmas key in the database.yml file. Moreover, it allows other SQLite PRAGMAs to be set via the pragmas key, and will ignore and warn for any unknown pragmas.

Checklist

Before submitting the PR make sure the following are checked:

  • This Pull Request is related to one change. Changes that are unrelated should be opened in separate PRs.
  • Commit message has a detailed description of what changed and why. If this PR fixes a related issue include it in the commit message. Ex: [Fix #issue-number]
  • Tests are added or updated if you fix a bug or add a feature.
  • CHANGELOG files are updated for the changed libraries if there is a behavior change or additional feature. Minor bug fixes and documentation changes should not be included.

@fractaledmind
Copy link
Contributor Author

@rafaelfranca @eileencodes @matthewd:

Following the conversation in #49691, I put together this alternative which follows Matthew's preferences expressed in the discussion there.

Honestly, hearing @suwyn's situation, it was an oversight on my part to not include the ability to override the new defaults in the first place.

@fractaledmind fractaledmind force-pushed the ar-overridable-sqlite-pragmas branch 2 times, most recently from de62858 to 3055e70 Compare December 28, 2023 08:21
@zzak
Copy link
Member

zzak commented Dec 28, 2023

Maybe we could add test cases for setting each of the pragmas, in either string or symbol? I know it's kind of testing #{pragma}= button, but we are kind of contracting what users can or can't put in the yaml file so it might be nice to know when and/or if that contract breaks. 🤔

@fractaledmind
Copy link
Contributor Author

Yeah, low cost to add the tests, and then CI will definitely catch any regressions, if introduced, on either Rails' side of with the SQLite3 gem. I'll add more tests today

@fractaledmind fractaledmind force-pushed the ar-overridable-sqlite-pragmas branch from 3055e70 to 627adee Compare December 28, 2023 23:05
@fractaledmind
Copy link
Contributor Author

@zzak: Tests added for each pragma (in addition to the one test for setting a non-default pragma and for setting an unknown pragma). Covered positive and error conditions for each pragma

@fractaledmind fractaledmind force-pushed the ar-overridable-sqlite-pragmas branch 2 times, most recently from e0daaee to 6589b90 Compare December 29, 2023 19:49
@fractaledmind
Copy link
Contributor Author

@zzak I pushed a fix for the test failures, but CI didn't rerun. Not sure what I did differently/wrong, as I'm pretty sure I followed the same process that I always do. Anyway, if there is any way to rerun CI, I'm confident tests will go green.

rubys added a commit to rubys/showcase that referenced this pull request Dec 31, 2023
Hopefully this can be retired once this can be configured:

rails/rails#50460
@dhh
Copy link
Member

dhh commented Dec 31, 2023

Real errors on 3.3:

image

fractaledmind and others added 2 commits December 31, 2023 23:15
Any PRAGMA configuration set under the `pragmas` key in the configuration file take precedence over Rails' defaults, and additional PRAGMAs can be set as well.

```yaml
database: storage/development.sqlite3
timeout: 5000
pragmas:
  synchronous: full
  temp_store: memory
```
@fractaledmind fractaledmind force-pushed the ar-overridable-sqlite-pragmas branch 3 times, most recently from 42c4d98 to df75954 Compare December 31, 2023 22:17
Any PRAGMA configuration set under the `pragmas` key in the configuration file take precedence over Rails' defaults, and additional PRAGMAs can be set as well.

```yaml
database: storage/development.sqlite3
timeout: 5000
pragmas:
  synchronous: full
  temp_store: memory
```
@fractaledmind fractaledmind force-pushed the ar-overridable-sqlite-pragmas branch from df75954 to caaf868 Compare December 31, 2023 22:41
@fractaledmind
Copy link
Contributor Author

@dhh: No idea what is up with the ujs test failures 🤷 but the rest is good to go. I simplified the error message expectation to be more resilient by only checking the core part of the error message. Aside from whatever is up with these ujs tests, everything related to this PR is green. And thanks for the style fix.

@dhh dhh merged commit 8ac2af2 into rails:main Jan 2, 2024
dhh added a commit that referenced this pull request Jan 2, 2024
* main:
  Fix test
  Allow overriding SQLite defaults from `database.yml` (#50460)
  Fix a flaky test in `query_assertions_test.rb` (#50503)
  Document apply_rubocop_autocorrect_after_generate! in configuration guide (#50532)
  bundle update rubocop --conservative (#50515)
@fractaledmind fractaledmind deleted the ar-overridable-sqlite-pragmas branch August 2, 2024 14:06
rossta added a commit to joyofrails/joyofrails.com that referenced this pull request Oct 3, 2024
…hancedsqlite3-adapter

Most of the enhancements provided by
activerecord-enhancedsqlite3-adapter have been upstreamed into Rails,
including:

* Add support for generated columns in SQLite3 adapter
  rails/rails#49346
* Add SQLite3 support for supports_insert_returning?
  rails/rails#49290
* Allow overriding SQLite defaults from database.yml
  rails/rails#50460
* Performance tune the SQLite3 adapter connection configuration
  rails/rails#49349
* Allow SQLite3 busy_handler to be configured with simple max number of
  retries rails/rails#49352

One enhancement Joy still needs is a way to load SQLite3 extensions.
There is a sqlpkg gem that would provide this, which includes generating
an initializer that overrides the Rails SQLite3 adapter. Instead, I‘ve
just adapted the enhancedsqlite3-adapter code and droppped into the lib/
directory. The code here also knows how to load an extension by gem name
which is makes it useful for the sqlite_ulid gem currently in use.
rossta added a commit to joyofrails/joyofrails.com that referenced this pull request Oct 3, 2024
…hancedsqlite3-adapter

Most of the enhancements provided by
activerecord-enhancedsqlite3-adapter have been upstreamed into Rails,
including:

* Add support for generated columns in SQLite3 adapter
  rails/rails#49346
* Add SQLite3 support for supports_insert_returning?
  rails/rails#49290
* Allow overriding SQLite defaults from database.yml
  rails/rails#50460
* Performance tune the SQLite3 adapter connection configuration
  rails/rails#49349
* Allow SQLite3 busy_handler to be configured with simple max number of
  retries rails/rails#49352

One enhancement Joy still needs is a way to load SQLite3 extensions.
There is a sqlpkg gem that would provide this, which includes generating
an initializer that overrides the Rails SQLite3 adapter. Instead, I‘ve
just adapted the enhancedsqlite3-adapter code and droppped into the lib/
directory. The code here also knows how to load an extension by gem name
which is makes it useful for the sqlite_ulid gem currently in use.
rossta added a commit to joyofrails/joyofrails.com that referenced this pull request Oct 3, 2024
…hancedsqlite3-adapter

Most of the enhancements provided by
activerecord-enhancedsqlite3-adapter have been upstreamed into Rails,
including:

* Add support for generated columns in SQLite3 adapter
  rails/rails#49346
* Add SQLite3 support for supports_insert_returning?
  rails/rails#49290
* Allow overriding SQLite defaults from database.yml
  rails/rails#50460
* Performance tune the SQLite3 adapter connection configuration
  rails/rails#49349
* Allow SQLite3 busy_handler to be configured with simple max number of
  retries rails/rails#49352

One enhancement Joy still needs is a way to load SQLite3 extensions.
There is a sqlpkg gem that would provide this, which includes generating
an initializer that overrides the Rails SQLite3 adapter. Instead, I‘ve
just adapted the enhancedsqlite3-adapter code and droppped into the lib/
directory. The code here also knows how to load an extension by gem name
which is makes it useful for the sqlite_ulid gem currently in use.
yoones pushed a commit to yoones/rails that referenced this pull request Mar 6, 2025
* Allow overriding SQLite defaults from `database.yml`

Any PRAGMA configuration set under the `pragmas` key in the configuration file take precedence over Rails' defaults, and additional PRAGMAs can be set as well.

```yaml
database: storage/development.sqlite3
timeout: 5000
pragmas:
  synchronous: full
  temp_store: memory
```

* Style

* Allow overriding SQLite defaults from `database.yml`

Any PRAGMA configuration set under the `pragmas` key in the configuration file take precedence over Rails' defaults, and additional PRAGMAs can be set as well.

```yaml
database: storage/development.sqlite3
timeout: 5000
pragmas:
  synchronous: full
  temp_store: memory
```

---------

Co-authored-by: David Heinemeier Hansson <david@hey.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants