force correct index for MySQL when listing entries#8211
force correct index for MySQL when listing entries#8211Alkarex merged 4 commits intoFreshRSS:edgefrom
Conversation
|
I think that would break most databases, so it will need a special case for setups using MySQL. |
|
By the way, there are some related performance problems specific to MySQL / MariaDB, which would welcome some help: |
|
Could you please check whether it is sufficient with |
|
Yes you are right, this is a fix specifically for MySQL and probably also MariaDB. I was under the impression that Postgres and SQLite do not support the I've done some more testing and based on my production dump I can report the following: Current query in
with So it seems like this fix would directly target MySQL <= 9 and MariaDB < 11. There is no difference when using When looking at your posted links, the second one seems to be this exact issue. I've seen some issues on inserting entries in the past, but these all somehow resolved on their own. So for me it's a non-issue. I think it was causing problems, when I've had multiple feeds publishing "lots of" articles. But i cannot give you exact numbers as I don't have this scenario anymore. Currently with 99 feeds subscribed to, I don't have any problems with updating at all. I must admit I'm still using MySQL 5.7, because my deployment is from 2018. I probably should upgrade my database in the near future... But as the issue still persists with MySQL 9.4 and even applies to MariaDB < 11, this would probably also help someone else, as FreshRSS currently targets MySQL 8.0+ and MariaDB 10.0.5+. |
|
Thanks for the additional tests 👍🏻 |
* force correct index for MySQL when listing entries * Make special case for MySQL / MariaDB --------- Co-authored-by: Alexandre Alapetite <alexandre@alapetite.fr>
This PR contains the following updates: | Package | Update | Change | |---|---|---| | [freshrss/freshrss](https://freshrss.org/) ([source](https://github.com/FreshRSS/FreshRSS)) | minor | `1.27.1` -> `1.28.0` | --- ### Release Notes <details> <summary>FreshRSS/FreshRSS (freshrss/freshrss)</summary> ### [`v1.28.0`](https://github.com/FreshRSS/FreshRSS/blob/HEAD/CHANGELOG.md#2025-12-24-FreshRSS-1280) [Compare Source](FreshRSS/FreshRSS@1.27.1...1.28.0) - Features - New sorting and filtering by date of *User modified* [#​7886](FreshRSS/FreshRSS#7886), [#​8090](FreshRSS/FreshRSS#8090), [#​8105](FreshRSS/FreshRSS#8105), [#​8118](FreshRSS/FreshRSS#8118), [#​8130](FreshRSS/FreshRSS#8130) - Corresponding search operator, e.g. `userdate:PT1H` for the past hour [#​8093](FreshRSS/FreshRSS#8093) - Allows finding articles marked by the local user as read/unread or starred/unstarred at specific dates for e.g. undo action. - New sorting by article length [#​8119](FreshRSS/FreshRSS#8119) - New advanced search form [#​8103](FreshRSS/FreshRSS#8103), [#​8122](FreshRSS/FreshRSS#8122), [#​8226](FreshRSS/FreshRSS#8226) - Add compatibility with PCRE word boundary `\b` and `\B` for regex search using PostgreSQL [#​8141](FreshRSS/FreshRSS#8141) - More uniform SQL search and PHP search for accents and case-sensitivity (e.g. for automatically marking as read) [#​8329](FreshRSS/FreshRSS#8329) - New overview of dates with most unread articles [#​8089](FreshRSS/FreshRSS#8089) - Allow marking as read articles older than 1 or 7 days also when sorting by publication date [#​8163](FreshRSS/FreshRSS#8163) - New option to show user labels instead of tags in RSS share [#​8112](FreshRSS/FreshRSS#8112) - Add new feed visibility (priority) *Show in its feed* [#​7972](FreshRSS/FreshRSS#7972) - New ability to share feed visibility through API (implemented by e.g. Capy Reader) [#​7583](FreshRSS/FreshRSS#7583), [#​8158](FreshRSS/FreshRSS#8158) - Configurable notification timeout [#​7942](FreshRSS/FreshRSS#7942) - OPML export/import of unicity criteria [#​8243](FreshRSS/FreshRSS#8243) - Ensure stable IDs (categories, feeds, labels) during export/import [#​7988](FreshRSS/FreshRSS#7988) - Add username and timestamp to SQLite export from Web UI [#​8169](FreshRSS/FreshRSS#8169) - Add option to apply filter actions to existing articles [#​7959](FreshRSS/FreshRSS#7959), [#​8259](FreshRSS/FreshRSS#8259) - Support CSS selector `~` *subsequent-sibling* [#​8154](FreshRSS/FreshRSS#8154) - Upstream PR [phpgt/CssXPath#231](phpgt/CssXPath#231) - Rework saving of configuration files for more reliability in case of e.g. full disk [#​8220](FreshRSS/FreshRSS#8220) - Web scraping support date format as milliseconds for Unix epoch [#​8266](FreshRSS/FreshRSS#8266) - Allow negative category sort numbers [#​8330](FreshRSS/FreshRSS#8330) - Performance - Improve SQL speed for updating cached information [#​6957](FreshRSS/FreshRSS#6957), [#​8207](FreshRSS/FreshRSS#8207), [#​8255](FreshRSS/FreshRSS#8255), [#​8254](FreshRSS/FreshRSS#8254), [#​8255](FreshRSS/FreshRSS#8255) - Fix SQL performance issue with MySQL, using an index hint [#​8211](FreshRSS/FreshRSS#8211) - Scaling of user statistics in Web UI and CLI, to help instances with 1k+ users [#​8277](FreshRSS/FreshRSS#8277) - API streaming of large responses for reducing memory consumption and increasing speed [#​8041](FreshRSS/FreshRSS#8041) - Security - 💥 Move unsafe autologin to an extension [#​7958](FreshRSS/FreshRSS#7958) - Fix some CSRFs [#​8035](FreshRSS/FreshRSS#8035) - Strengthen some crypto (login, tokens, nonces) [#​8061](FreshRSS/FreshRSS#8061), [#​8320](FreshRSS/FreshRSS#8320) - Create separate HTTP `Retry-After` rules for proxies [#​8029](FreshRSS/FreshRSS#8029), [#​8218](FreshRSS/FreshRSS#8218) - Add `data:` to CSP in subscription controller [#​8253](FreshRSS/FreshRSS#8253) - Improve anonymous authentication logic [#​8165](FreshRSS/FreshRSS#8165) - Enable GitHub [release immutability](https://github.blog/changelog/2025-10-28-immutable-releases-are-now-generally-available/) [#​8205](FreshRSS/FreshRSS#8205) - Bug fixing - Exclude local networks for domain-wide HTTP `Retry-After` [#​8195](FreshRSS/FreshRSS#8195) - Fix OpenID Connect with Debian 13 [#​8032](FreshRSS/FreshRSS#8032) - Fix MySQL / MariaDB bug wrongly sorting new articles [#​8223](FreshRSS/FreshRSS#8223) - Fix MySQL / MariaDB database size calculation [#​8282](FreshRSS/FreshRSS#8282) - Fix SQLite bind bug when adding user label [#​8101](FreshRSS/FreshRSS#8101) - Fix SQL auto-update of field `f.kind` to ease migrations from FreshRSS versions older than 1.20.0 [#​8148](FreshRSS/FreshRSS#8148) - Fix search encoding and quoting [#​8311](FreshRSS/FreshRSS#8311), [#​8324](FreshRSS/FreshRSS#8324), [#​8338](FreshRSS/FreshRSS#8338) - Fix handling of database unexpected null content (during migrations) [#​8319](FreshRSS/FreshRSS#8319), [#​8321](FreshRSS/FreshRSS#8321) - Fix drag & drop of user query losing information [#​8113](FreshRSS/FreshRSS#8113) - Fix DOM error while filtering retrieved full content [#​8132](FreshRSS/FreshRSS#8132), [#​8161](FreshRSS/FreshRSS#8161) - Fix `config.custom.php` during install [#​8033](FreshRSS/FreshRSS#8033) - Fix do not mark important feeds as read from category [#​8067](FreshRSS/FreshRSS#8067) - Fix regression of warnings in Web browser console due to lack of `window.bcrypt` object [#​8166](FreshRSS/FreshRSS#8166) - Fix chart resize regression due to `chart.js` v4 update [#​8298](FreshRSS/FreshRSS#8298) - Fix CLI user creation warning when language is not given [#​8283](FreshRSS/FreshRSS#8283) - Fix merging of custom HTTP headers [#​8251](FreshRSS/FreshRSS#8251) - Fix bug in the case of duplicated mark-as-read filters [#​8322](FreshRSS/FreshRSS#8322) - SimplePie - Fix support of HTTP trailer headers [#​7983](FreshRSS/FreshRSS#7983), [simplepie#943](simplepie/simplepie#943) - Apply HTTPS policy also on GUIDs and permalinks [#​8037](FreshRSS/FreshRSS#8037), [simplepie#951](simplepie/simplepie#951) - Fix `WordPress.com` HTTP duplicates with WebSub [Automattic/pushpress#16](Automattic/pushpress#16) - Implement HTML whitelist for SimplePie sanitizer [#​7924](FreshRSS/FreshRSS#7924), [simplepie#947](simplepie/simplepie#947) - Various upstream contributions [simplepie#940](simplepie/simplepie#940), [simplepie#944](simplepie/simplepie#944) - Deployment - Docker default image updated to Debian 13 Trixie with PHP 8.4.11 and Apache 2.4.65 [#​8032](FreshRSS/FreshRSS#8032) - Docker alternative image updated to Alpine 3.23 with PHP 8.4.15 and Apache 2.4.65 [#​8285](FreshRSS/FreshRSS#8285) - Fix Docker healthcheck `cli/health.php` compatibility with OpenID Connect [#​8040](FreshRSS/FreshRSS#8040) - Improve Docker for compatibility with other base images such as Arch Linux [#​8299](FreshRSS/FreshRSS#8299) - Improve `cli/access-permissions.sh` to detect the correct permission Web group such as `www-data`, `apache`, or `http` - Update PostgreSQL volume for Docker [#​8216](FreshRSS/FreshRSS#8216), [#​8224](FreshRSS/FreshRSS#8224) - Catch lack of `exec()` function for git update [#​8228](FreshRSS/FreshRSS#8228) - Work around `DOMDocument::saveHTML()` scrambling charset encoding in some versions of libxml2 [#​8296](FreshRSS/FreshRSS#8296) - Improve configuration checks for PHP extensions (in Web UI and CLI), including recommending e.g. `php-intl` [#​8334](FreshRSS/FreshRSS#8334) - UI - New button for toggling sidebar on desktop view [#​8201](FreshRSS/FreshRSS#8201), [#​8286](FreshRSS/FreshRSS#8286) - Better transitions between groups of articles [#​8174](FreshRSS/FreshRSS#8174) - New links in transitions and jump ⏭ to next transition [#​8294](FreshRSS/FreshRSS#8294) - More visible selected article [#​8230](FreshRSS/FreshRSS#8230) - Show the parsed search query instead of the original user input [#​8293](FreshRSS/FreshRSS#8293), [#​8306](FreshRSS/FreshRSS#8306), [#​8341](FreshRSS/FreshRSS#8341) - Show search query in the page title [#​8217](FreshRSS/FreshRSS#8217) - Scroll into filtered feed/category on page load in the sidebar [#​8281](FreshRSS/FreshRSS#8281), [#​8307](FreshRSS/FreshRSS#8307) - Fix autocomplete issues in change password form [#​7812](FreshRSS/FreshRSS#7812) - Fix navigating between read feeds using shortcut <kbd>shift</kbd>+<kbd>j</kbd>/<kbd>k</kbd> [#​8057](FreshRSS/FreshRSS#8057) - Dark background in Web app manifest to avoid white flash when opening [#​8140](FreshRSS/FreshRSS#8140) - Increase button visibility in UI to change theme [#​8149](FreshRSS/FreshRSS#8149) - Replace arrow navigation in theme switcher with `<select>` [#​8190](FreshRSS/FreshRSS#8190) - Improve scroll of article after load of user labels [#​7962](FreshRSS/FreshRSS#7962) - Keep scroll state of page when closing the slider [#​8295](FreshRSS/FreshRSS#8295), [#​8301](FreshRSS/FreshRSS#8301) - Scroll into filtered feed/category on page load [#​8281](FreshRSS/FreshRSS#8281) - Display sidebar dropdowns above if no space below [#​8335](FreshRSS/FreshRSS#8335), [#​8336](FreshRSS/FreshRSS#8336) - Use native CSS instead of SCSS [#​8200](FreshRSS/FreshRSS#8200), [#​8241](FreshRSS/FreshRSS#8241) - Using [CSS nesting](https://developer.mozilla.org/en-US/docs/Web/CSS/Guides/Nesting) and [relative colours](https://developer.mozilla.org/en-US/docs/Web/CSS/Guides/Colors/Using_relative_colors). - Various UI and style improvements: [#​8171](FreshRSS/FreshRSS#8171), [#​8185](FreshRSS/FreshRSS#8185), [#​8196](FreshRSS/FreshRSS#8196) - JavaScript finalise migration from `Promise` to `async`/`await`: [#​8182](FreshRSS/FreshRSS#8182) - API - API performance optimisation: streaming of large responses [#​8041](FreshRSS/FreshRSS#8041) - Fever API: Add `with_ids` parameter to mass-change read/unread/saved/unsaved on lists of articles [#​8312](FreshRSS/FreshRSS#8312) - Misc API: better REST error semantics [#​8232](FreshRSS/FreshRSS#8232) - Extensions - Add support for extension priority [#​8038](FreshRSS/FreshRSS#8038) - Add support for extension compatibility [#​8081](FreshRSS/FreshRSS#8081) - Improve PHP code with hook enums [#​8036](FreshRSS/FreshRSS#8036) - New hook `nav_entries` [#​8054](FreshRSS/FreshRSS#8054) - Rename [Extensions](https://github.com/FreshRSS/Extensions) default branch from *master* to *main* [#​8194](FreshRSS/FreshRSS#8194) - I18n - Translation status as text in README [#​7842](FreshRSS/FreshRSS#7842) - Add new translate CLI commands `move` [#​8214](FreshRSS/FreshRSS#8214) - Change some regional language codes to comply with RFC 5646 / IETF BCP 47 / ISO 3166 / ISO 639-1 [#​8065](FreshRSS/FreshRSS#8065) - Improve German [#​8028](FreshRSS/FreshRSS#8028) - Improve Greek [#​8146](FreshRSS/FreshRSS#8146) - Improve Finnish [#​8073](FreshRSS/FreshRSS#8073), [#​8092](FreshRSS/FreshRSS#8092) - Improve Hungarian [#​8244](FreshRSS/FreshRSS#8244) - Improve Italian [#​8115](FreshRSS/FreshRSS#8115), [#​8186](FreshRSS/FreshRSS#8186) - Improve Polish [#​8134](FreshRSS/FreshRSS#8134), [#​8135](FreshRSS/FreshRSS#8135) - Improve Russian [#​8155](FreshRSS/FreshRSS#8155), [#​8197](FreshRSS/FreshRSS#8197) - Improve Simplified Chinese [#​8308](FreshRSS/FreshRSS#8308), [#​8313](FreshRSS/FreshRSS#8313) - Misc. - Add code to modify a search expression [#​8293](FreshRSS/FreshRSS#8293) - Remove *Pocket* sharing service [#​8127](FreshRSS/FreshRSS#8127), [#​8128](FreshRSS/FreshRSS#8128) - Update to PHPMailer 7.0.1 [#​8048](FreshRSS/FreshRSS#8048), [#​8180](FreshRSS/FreshRSS#8180), [#​8272](FreshRSS/FreshRSS#8272) - 💥 Housekeeping of `lib_rss.php` with potential breaking changes for some extensions [#​8193](FreshRSS/FreshRSS#8193), - Use native PHP `#[Deprecated]` [#​8325](FreshRSS/FreshRSS#8325) - Improve PHP code [#​8156](FreshRSS/FreshRSS#8156), [#​8203](FreshRSS/FreshRSS#8203), [#​8284](FreshRSS/FreshRSS#8284), [#​8292](FreshRSS/FreshRSS#8292), [#​8297](FreshRSS/FreshRSS#8297) - GitHub Actions: `--no-progress` [#​8315](FreshRSS/FreshRSS#8315) - Update dev dependencies [#​8043](FreshRSS/FreshRSS#8043), [#​8044](FreshRSS/FreshRSS#8044), [#​8045](FreshRSS/FreshRSS#8045), [#​8046](FreshRSS/FreshRSS#8046), [#​8047](FreshRSS/FreshRSS#8047), [#​8052](FreshRSS/FreshRSS#8052), [#​8176](FreshRSS/FreshRSS#8176), [#​8177](FreshRSS/FreshRSS#8177), [#​8178](FreshRSS/FreshRSS#8178), [#​8179](FreshRSS/FreshRSS#8179), [#​8210](FreshRSS/FreshRSS#8210), [#​8270](FreshRSS/FreshRSS#8270), [#​8271](FreshRSS/FreshRSS#8271), [#​8273](FreshRSS/FreshRSS#8273), [#​8274](FreshRSS/FreshRSS#8274), [#​8275](FreshRSS/FreshRSS#8275), [#​8276](FreshRSS/FreshRSS#8276) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0Mi4zOS4xIiwidXBkYXRlZEluVmVyIjoiNDIuMzkuMSIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsiaW1hZ2UiXX0=--> Reviewed-on: https://gitea.alexlebens.dev/alexlebens/infrastructure/pulls/2851 Co-authored-by: Renovate Bot <renovate-bot@alexlebens.net> Co-committed-by: Renovate Bot <renovate-bot@alexlebens.net>
|
Help welcome to test with MariaDB the following index change (and whether we could remove the |
Issue
I have 700k+ entries in my
_entrytable and the performance is abysmal (10+ seconds) when querying the overview (front page) and having the "unread" filter set. I dug into the code and enabled the slow query log which pointed me at the\FreshRSS_EntryDAO::sqlListWheremethod taking an absurd amount of time.Cause
The query it's producing is
An
EXPLAINshows the issue:Its using the wrong index (on the PK), which means it's basically doing a full table scan.
So having exactly 737_141 entries in my table, the query takes (using PhpStorm)
Because MySQL is bad at picking the "right" index, this needs to be fixed.
Forcing the better suiting index manually, results in the query
which now isn't even taking a single second.
The
EXPLAINshows the resulting query plan:Bugfix
\FreshRSS_EntryDAO::sqlListWhereneeds an update which forces MySQL to pick the right indexResult
The front page with the "unread" filter is now instantly available.