I'm recently toying with populating org-mem tables from the output of (org-element-parse-buffer), so that we could maybe deprecate org-mem-parser.

Still a long time off (so if anyone wanna chip away at that part, I'd love to see code examples!).

I find myself first solving a quagmire of issues related to the fact it's significantly slower to use Org's parser in place of Org-mem's parser if used in the same way as now: creating subprocesses and then throwing them away, so they have to re-read and re-enable Org for every file every time.

Daemon

Instead, I imagine to have an "Org worker daemon" which always connected to the mother process and has permanently open Org buffers holding the contents of every file, parsed and done, benefiting from the org-element cache. When any files change mtime, it'd re-create buffers for only those files in specific.

Then this daemon would stand ready to run org-element-map in any file that the mother process requests.

The end result might be an API of a single function, with a calling convention like:

(org-saved-file-element-map FILE TYPE FUN &rest ARGS)

analogous to Org's:

(org-element-map (org-element-parse-buffer) TYPE FUN &rest ARGS)

"with-file" macro

It's not just the parse tree that may be interesting; old-fashioned buffer analysis with org-up-heading and org-get-tags etc is also handy. So add to the API something like:

(org-with-saved-file FILE
   &rest BODY)

Which is analogous to something you can already do with org-node:

(with-current-buffer (org-node--work-buffer-for FILE)
   &rest BODY)

but the latter is a lot slower if you're gonna iterate over a lot of files. Just setting up a work buffer for each of my 2,500 files takes my machine 30 seconds. By the way, that's part of why (org-roam-db-sync :force) takes so long!

And I can't just keep them open -- that would be too convenient, wouldn't it? If I do, Emacs as a whole slows down, especially window-switching and buffer-switching commands, and Org-related stuff.

Come to think, this "with-file" concept could be generalized to all major modes. It's really just a way to have a limitless amount of extra "hidden buffers" without slowing down Emacs.