Skip to content

Rely on emacsql-sqlite-open to pick the best available back-end#2486

Merged
dustinfarris merged 5 commits intoorg-roam:mainfrom
tarsiiformes:sqlite-open
Feb 18, 2025
Merged

Rely on emacsql-sqlite-open to pick the best available back-end#2486
dustinfarris merged 5 commits intoorg-roam:mainfrom
tarsiiformes:sqlite-open

Conversation

@tarsius
Copy link
Copy Markdown
Contributor

@tarsius tarsius commented Nov 17, 2024

That function was added two years ago and the first release that
provided it was v4.0.0. It automatically picks the best available
back-end, emacsql-sqlite-builtin or emacsql-sqlite-module.
In v4.0.0 it could also fall back to the legacy emacsql-sqlite.
The inferior third-party back-ends are no longer supported.

Emacsql v4.1.0, which will be releases in early December, removes
the legacy emacsql-sqlite back-end (which used a custom binary).

This means that we now require an Emacs that was built with SQLite
and/or module support.


Motivation for this change

EmacSQL now takes care of picking the best back-end itself. Let's
use that.

By removing an option that offers no longer valid choices, we make
sure users cannot pick an invalid choice.

@tarsius
Copy link
Copy Markdown
Contributor Author

tarsius commented Nov 18, 2024

Re #2485.

brsvh added a commit to brsvh/shelf that referenced this pull request Nov 19, 2024
Signed-off-by: Burgess Chang <bsc@brsvh.org>
wnka added a commit to wnka/piwonka-doom-emacs that referenced this pull request Nov 19, 2024
@Delapouite Delapouite added 2. db Issues regarding the database 1. upstream bug/issue dependent on upstream behavior labels Nov 19, 2024
@aragaer
Copy link
Copy Markdown
Contributor

aragaer commented Dec 18, 2024

Unfortunately this implementation of org-roam-db results in endless recursion when file doesn't exist. When org-roam-db--setup checks if file exists, it is already created by emacsql.

Here's the updated version that works:

(defun org-roam-db ()
  "Entrypoint to the Org-roam sqlite database.
Initializes and stores the database, and the database connection.
Performs a database upgrade when required."
    (or (and-let* ((conn (org-roam-db--get-connection)))
          (and (emacsql-live-p conn) conn))
        (puthash (expand-file-name (file-name-as-directory org-roam-directory))
                 (emacsql-sqlite-open org-roam-db-location nil
                                      (if (file-exists-p org-roam-db-location)
                                          #'org-roam-db--setup
                                        #'org-roam-db--init))
                 org-roam-db--connection)))

@tarsius
Copy link
Copy Markdown
Contributor Author

tarsius commented Dec 20, 2024

emacsql-sqlite-open calls its SETUP function with the database as its only argument and expects it to only access the db via that binding. When I wrote 118be8a I did not notice, that, while none of the functions called in the old org-roam-db do that directly, they do call functions that in turn use org-roam-db, leading to this infinite loop.

I.e., we have to give up on using the SETUP argument, which means that Org-Roam doesn't follow the (refined) conventions. Alternatively the problematic functions could be changed to take the database as an optional argument, but I don't understand enough about what Org-Roam is doing here (and why), so it would be too risky for me to do it.

@tarsius
Copy link
Copy Markdown
Contributor Author

tarsius commented Dec 20, 2024

I've pushed an additional commit to address the compiler warning (which are unrelated to the changes in the other commits). Normally that should be done in a separate pull-request, but due to the radio silence, I have chosen to append that here, so I don't have to juggle with two branches when/if I decide to play with Org-Roam.

@aragaer
Copy link
Copy Markdown
Contributor

aragaer commented Dec 20, 2024

I think that org-roam-db--setup is for setting up database when it already exists and org-roam-db--init for its first creation, so if we pass the correct one, that should work. Unfortunately the org-roam-db--setup checks if upgrade is needed and that uses (org-roam-db-sync t) which actually closes and reopens the connection. While emacql approach is fine we might want to rewrite the setup function so that it detects whether an update is needed and postpones it until after return from setup/open.

@dustinfarris
Copy link
Copy Markdown
Contributor

@tarsius I just merged a couple of related changes in #2503 that do just enough to get CI passing. Would you be willing to update this branch based off of latest main?

That function was added two years ago and the first release that
provided it was v4.0.0.  It automatically picks the best available
back-end, `emacsql-sqlite-builtin' or `emacsql-sqlite-module'.
In v4.0.0 it could also fall back to the legacy `emacsql-sqlite'.
The inferior third-party back-ends are no longer supported.

Emacsql v4.1.0, which will be releases in early December, removes
the legacy `emacsql-sqlite' back-end (which used a custom binary).
Starting with EmacSQL v4.1.0, `emacsql-sqlite-open' takes care of that.
To avoid having to create a wrapper function and silence the byte-
compiler for both when using a pre-rename and post-rename Org, keep
it simple and just depend on Org 9.6.
Depend on Org 9.6 for `org-fold-show-context'.
@tarsius
Copy link
Copy Markdown
Contributor Author

tarsius commented Feb 18, 2025

I've rebased, but note that I have not tested, since I don't use this package.

@dustinfarris
Copy link
Copy Markdown
Contributor

It is working well for me locally. Thank you so much for this thorough PR! ❤️

@dustinfarris dustinfarris merged commit ed272ea into org-roam:main Feb 18, 2025
@tarsius
Copy link
Copy Markdown
Contributor Author

tarsius commented Feb 18, 2025

Thanks for reviving this package!

@tarsius tarsius deleted the sqlite-open branch February 18, 2025 17:14
dustinfarris added a commit that referenced this pull request Feb 18, 2025
Now that emacsql handles choosing an appropriate backend/connector, we
do not need to provide these instructions.  If there are issues, emacsql
has good error messages that tell the user what they need to do to get
things working.

In general:
  - If the user is on Emacs >= 29, emacsql will use the built-in sqlite
    functionality that comes with Emacs.
  - If the user is on Emacs < 29, emacsql will use a module-based
    connector that requires the user to have certain libraries available
    on their machine.  emacsql will tell the user what is needed if they
    do not have it already.

Ref: #2486
Close: #2502
Close: #2415
@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Jun 7, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

1. upstream bug/issue dependent on upstream behavior 2. db Issues regarding the database

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants