Skip to content

Fix custom sort#1391

Merged
lucyleeow merged 25 commits intosphinx-gallery:masterfrom
drammock:fix-custom-sort
Nov 6, 2024
Merged

Fix custom sort#1391
lucyleeow merged 25 commits intosphinx-gallery:masterfrom
drammock:fix-custom-sort

Conversation

@drammock
Copy link
Copy Markdown
Member

closes #1389

I think no docs update is needed because now things work the way the docs describe them, but feel free to request changes / push commits if you think I'm wrong about that.

@larsoner larsoner added the bug label Oct 15, 2024
@larsoner
Copy link
Copy Markdown
Contributor

@lucyleeow I did not follow all discussion so I'll let you review, feel free to merge if you're happy. I can look deeper if it would help to have another set of eyes or you're busy though!

Copy link
Copy Markdown
Member Author

@drammock drammock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left a few comments to hopefully speed up reviewing


def _get_class(gallery_conf, key):
"""Get a class for the given conf key."""
from .sorting import FunctionSortKey
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nesting required to avoid circular import

Comment on lines +1646 to +1647
if not is_builtin_alias:
val = FunctionSortKey(val)
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just to spell out the logic of this function:

  • if it's a string alias of a builtin SortKey class, convert to FQN string
  • then, if it's a string, check for ".", split, and try to import
  • if that worked, wrap the imported func in FunctionSortKey (unless it's already a built-in SortKey class)



class FunctionSortKey:
def FunctionSortKey(func, r=None):
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

opted for CamelCase on the function name, to avoid changing the API (though if you hate this, it's probably no big deal to change it to snake_case, since it wasn't really working before anyway AFAICT).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we allow within_subsection_order to take a class instance, then I think this can be reverted and would work on within_subsection_order ?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep! done in 716b180

@drammock
Copy link
Copy Markdown
Member Author

one other important note: now that FunctionSortKey returns a type (rather than an instance) it will fail sphinx's is_serializable checks. So might be worth highlighting that in the docs / strongly encouraging the use of FQN strings as the preferred way to do custom sorting.

@lucyleeow
Copy link
Copy Markdown
Contributor

Half way through review I realised that I already figured out that subsection_order and minigallery_sort_order worked differently to within_subsection_order, when reviewing #1226. Thus I specifically removed within_subsection_order from the docs (see: https://sphinx-gallery.github.io/stable/configuration.html#custom-sort-keys and my last note: #1226 (comment)

(also which part of the docs were you reading? They can definitely be improved, I was not sure where to put the section on custom sort keys as you can use them for several 'unrelated' configs 😩 )

I do like that this now supports within_subsection_order BUT I think it would no longer work for subsection_order and minigallery_sort_order as these both use _get_callables, which can't be a class.

if inspect.isclass(what):
raise ConfigError(
f"Got class rather than callable instance for {readable}: {what}"
)

(obviously we need to improve our testing)

Ideally, I think I would love to get rid off _get_class, but this would involve:

  • internally make the convenience classes (e.g., ExampleTitleSortKey) functions instead of classes, but keep the names as camelcase (since we ask the user to pass as string anyway). I think we only had classes for the stable __repr__ but as we are passing a string now, this is not necessary. I think this would also aid the within_subsection_order and FunctionSortKey problem, as we would no longer be doing:

listdir, key=_get_class(gallery_conf, "within_subsection_order")(src_dir)

  • maybe getting rid of FunctionSortKey altogether (a quick search on github seems to show that it is not really used), it was introduced not long before the serialization stuff.

Sorry @larsoner, have to bring you in here, you'll be able to come up with a better solution than my tired brain.
Also apologies if I have made a mistake somewhere.

@drammock
Copy link
Copy Markdown
Member Author

Thus I specifically removed within_subsection_order from the docs (see: https://sphinx-gallery.github.io/stable/configuration.html#custom-sort-keys and my last note: #1226 (comment)

Ah yes, I somehow missed that within_subsection_order wasn't mentioned in the "Custom Sort Keys" section of the config docs! So my earlier statement of "the docs say you can X but it doesn't work" is not strictly correct.

maybe getting rid of FunctionSortKey altogether (a quick search on github seems to show that it is not really used)

-0.5 from me; this PR came about because I want to use it (on subsection orders) and couldn't get it to work. I'd be content though if passing an FQN string of a sorting function worked on subsection orders. Note that at present (in this PR) FunctionSortKey is getting used internally to convert the imported FQN into a callable class (to parallel the other SortKey classes), so even if removed from the public API, it might still be in the codebase... unless:

I would love to get rid off _get_class

...then maybe FunctionSortKey could go away entirely. I can look into that.

I do like that this now supports within_subsection_order BUT I think it would no longer work for subsection_order and minigallery_sort_order as these both use _get_callables, which can't be a class.

I'll add a test of whether FunctionSortKey still works on subsection_order and minigallery_sort_order, and report back.

@lucyleeow
Copy link
Copy Markdown
Contributor

lucyleeow commented Oct 16, 2024

I completely agree that ideally I would like all 3 configs to be able to take a custom sort key and for them to work similarly.

I think a function and string FQN should work on all 3 but we've just got to get the class thing working.

I thought of getting rid of _get_class because maybe we could get within_subsection_order to work with _get_callables, by changing the convenience classes (e.g., ExampleTitleSortKey) to be functions as they are now given as string. It would also be nice to reduce some complexity in the code. Hopefully I haven't missed something important though.

We can keep FunctionSortKey, I think we can make it work. But I think we'll want to recommend using functions.

Thanks for working on this!

@lucyleeow
Copy link
Copy Markdown
Contributor

Thinking about it more, maybe the convenience classes could be a 'callable' (i.e. class instance) instead of a class. This would allow us to use _get_callables and keep these CamelCased classes as classes, and not have to change them to functions? Instead of init-ing with the src_dir we can just pass it the full path?

The only other thing we would want to make sure works is any existing user's own classes used for within_section_order (as we know the FunctionSortKey does not work). They could have done this by subclassing _SortKey (a search on github for this class was not helpful as I don't think you can do a literal string search over all of github) or making their own class that works the same way. I don't know how common this is but I don't think we could get rid of _get_class without breaking existing users own within_subsection_order classes.

I guess the other option is to loosen, so a class instance or a class are both accepted? Not sure what the best solution is here. I would like to go for getting rid of _get_class but I don't want to break existing users' code/force them to change their code.

Apologies for patchiness, I am on holiday but I will try to keep on top of this.

Copy link
Copy Markdown
Member Author

@drammock drammock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK _get_class is now gone, replaced by a tweaked version of _get_callables. It was quite a game of whack-a-mole running tests locally (mostly because I didn't realize at first all the different ways _get_callables was being used).

Thinking about it more, maybe the convenience classes could be a 'callable' (i.e. class instance) instead of a class. This would allow us to use _get_callables and keep these CamelCased classes as classes, and not have to change them to functions? Instead of init-ing with the src_dir we can just pass it the full path?

What I ended up doing was detecting whether the "callable" was one of our built-in sorter classes, or the class returned by FunctionSortKey, and if so, instantiating them within _get_callables. I think this means that any user-created callable class instances should still work (and IIUC we don't expect users to pass in an actual class right? just instances?)

Comment on lines +53 to +56
with pytest.raises(ConfigError, match=r"Unknown string option.*when importing"):
_fill_gallery_conf_defaults(sphinx_gallery_conf)
sphinx_gallery_conf = dict(within_subsection_order=1.0)
with pytest.raises(ConfigError, match="a fully qualified.*got float"):
with pytest.raises(ConfigError, match="must be callable"):
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these changes reflect the change from using _get_class to using _get_callables (they had similar but not identical error messages)

sphinx_gallery_conf = {
"examples_dirs": "src",
"gallery_dirs": "ex",
"within_subsection_order": FunctionSortKey(custom_sorter),
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can add here

    "minigallery_sort_order": FunctionSortKey(custom_sorter),
    "subsection_order": FunctionSortKey(custom_sorter),

and the test will still pass, but I think that's not especially informative because (IIUC) the site being built by for this test doesn't have subsections / minigalleries? Could use some guidance on how/where to test that FunctionSortKey(my_func) and FQN strings work correctly for minigalleries and subsection ordering... maybe that's already tested here?:

def test_minigallery_directive(minigallery_tree, test, heading, sortkey):

...but I find that test/parametrization inscrutable, so I'd appreciate some guidance on whether those cases are already covered, and if not, how/where I should add/tweak a test to cover them.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we could do minigalleries here.

I added a add_rst pytest mark here, which will add a index.rst file in src with the contents of file arg. (This happens in the sphinx_wrapper_app in conftest). See this test for example:

@pytest.mark.add_conf(
content="""
sphinx_gallery_conf = {
'examples_dirs': 'src',
'gallery_dirs': 'ex',
}"""
)
@pytest.mark.add_rst(
file="""
Header
======
.. minigallery:: index.rst
"""
)
def test_minigallery_not_in_examples_dirs(sphinx_app_wrapper):

AFAICT we only test FQN string for subsection_order in tinybuild.
It may be a big ask, but what do you think about adding a second subsection to the sphinx_gallery/tests/testconfs/src gallery...?

@larsoner
Copy link
Copy Markdown
Contributor

and IIUC we don't expect users to pass in an actual class right? just instances?

It looks like at least mpl does:

https://github.com/matplotlib/matplotlib/blob/9489b938556b16633591c2e829462b1aa4e4a46e/doc/conf.py#L305

https://github.com/matplotlib/matplotlib/blob/9489b938556b16633591c2e829462b1aa4e4a46e/doc/conf.py#L187

https://github.com/matplotlib/matplotlib/blob/9489b938556b16633591c2e829462b1aa4e4a46e/doc/sphinxext/gallery_order.py#L125

Does that change things?

@drammock
Copy link
Copy Markdown
Member Author

and IIUC we don't expect users to pass in an actual class right? just instances?

It looks like at least mpl does [...] Does that change things?

ah, yes it would. thanks for the tip, I'll rethink this.

Comment on lines +1685 to +1686
# make sure built-in sorter classes get instantiated (so they become callable)
if is_custom_sorter or what in BUILTIN_SORTERS:
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we change this conditional to

Suggested change
# make sure built-in sorter classes get instantiated (so they become callable)
if is_custom_sorter or what in BUILTIN_SORTERS:
# make sure classes get instantiated (so they become callable)
if key == "within_subsection_order" and inspect.isclass(what):

then MPL's custom sorter class should work too, and the test suite still passes locally.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I am okay with this.

AFAICT custom within_subsection_order config values have to be a class as we previously always instantiated it, before passing it to sort.

This will however, also let you pass a class instance, like the other 2 sort configs, instead of a class that takes the srcdir on init.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done in 716b180

@larsoner
Copy link
Copy Markdown
Contributor

then MPL's custom sorter class should work too, and the test suite still passes locally.

Do you see an easy enough way to modify tinybuild to have mpl's use case? That way we can test the "class" case there but hopefully also test the "class instance" case in more targeted tests like I think you currently have in test_gen_gallery.py

@drammock
Copy link
Copy Markdown
Member Author

then MPL's custom sorter class should work too, and the test suite still passes locally.

Do you see an easy enough way to modify tinybuild to have mpl's use case?

Done in 2316166
We lose the fact that tinybuild is testing one of the built-in sorter classes though. Probably not a huge deal.

@lucyleeow
Copy link
Copy Markdown
Contributor

Sorry I got to this late. Would it work to alter sphinx_gallery/tests/testconfs/src to add a subgallery? Is it easy to 'fix' the tests that use sphinx_wrapper_app such that they pass even with a new subgallery?

I only suggest as it would be nice to have more unit tests, as tinygallery tests do get quite convoluted.

@drammock
Copy link
Copy Markdown
Member Author

Sorry I got to this late. Would it work to alter sphinx_gallery/tests/testconfs/src to add a subgallery? Is it easy to 'fix' the tests that use sphinx_wrapper_app such that they pass even with a new subgallery?

I only suggest as it would be nice to have more unit tests, as tinygallery tests do get quite convoluted.

I think I've achieved what you ask in d25c7d5 but please double-check it's what you expected, I'm still not sure I've wrapped my head around your test suite.

@lucyleeow
Copy link
Copy Markdown
Contributor

Yes thats exactly what I was thinking. Just seems like we'd need to update a few more tests, to account for the new subgallery but otherwise seems reasonable. Thank you.

@drammock
Copy link
Copy Markdown
Member Author

ok sorting tests are fixed @lucyleeow (sorry I missed those before, thought I ran full pytest before pushing but I must not have). All green now.

Copy link
Copy Markdown
Contributor

@lucyleeow lucyleeow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So sorry about the delay.

Some nits, mostly around testing (and its mostly our fault, e.g., I didn't realise we hardly test subsection_order), otherwise this is looking good.

I think some doc updates should go with this, but I can add them later in a separate PR as there are a few things I want to amend and move around.


# classes (not pickleable so need to resolve using fully qualified name)
_get_class(gallery_conf, "within_subsection_order") # make sure it works
# callable classes (not pickleable so need to resolve using fully qualified name)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am confused by this comment, maybe we could improve it.

AFAICT, within_subsection_order not does not need to be a class? If it is a class we instantiate it, but otherwise as long as it is a callable, it's fine.

The pickleable part; I think is talking about how we updated to using strings for the builtin sorters for new Sphinx (ref: #1289). I think we could move this to within _get_callables, above:

            if what in builtin_aliases:
                what = f"sphinx_gallery.sorting.{what}"

So maybe we can just amend this comment to:

Suggested change
# callable classes (not pickleable so need to resolve using fully qualified name)
# check `within_subsection_order`

def _check_order(sphinx_app, key, expected_order=None):
"""Iterates through sphx-glr-thumbcontainer divs and reads key from the tooltip.

Used to test that these keys (in index.rst) appear in a specific order.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a line here explaining what expected_order is doing?

"backreferences_dir": "gen_modules/backreferences",
"subsection_order": f"{util_root}.noop_key",
"within_subsection_order": "FileNameSortKey",
"within_subsection_order": CustomSortKey,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be nice to keep the string, to make sure this works. Is it possible check the custom class in test_gen_gallery.py now that we have added a subsection?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

somehow I missed this comment before, but I think it's now moot? The string-aliases for builtin sorters are now comprehensively tested in test_example_sorting (as are FQN strings of custom funcs, and actual callable functions) so AFAICT it's actually preferable to keep this as-is (since in the parametrized test we only test callable functions, not callable classes).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes totally agree, this is good as is.

sphinx_gallery_conf = {
"examples_dirs": "src",
"gallery_dirs": "ex",
"within_subsection_order": FunctionSortKey(custom_sorter),
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we could do minigalleries here.

I added a add_rst pytest mark here, which will add a index.rst file in src with the contents of file arg. (This happens in the sphinx_wrapper_app in conftest). See this test for example:

@pytest.mark.add_conf(
content="""
sphinx_gallery_conf = {
'examples_dirs': 'src',
'gallery_dirs': 'ex',
}"""
)
@pytest.mark.add_rst(
file="""
Header
======
.. minigallery:: index.rst
"""
)
def test_minigallery_not_in_examples_dirs(sphinx_app_wrapper):

AFAICT we only test FQN string for subsection_order in tinybuild.
It may be a big ask, but what do you think about adding a second subsection to the sphinx_gallery/tests/testconfs/src gallery...?

@lucyleeow
Copy link
Copy Markdown
Contributor

Let me know when you are done and I'll take another look. Thanks again 🙏

@drammock
Copy link
Copy Markdown
Member Author

drammock commented Nov 1, 2024

Let me know when you are done and I'll take another look. Thanks again 🙏

OK I think it's ready! I didn't touch the docs because you said you wanted to tackle that in a separate PR. I did a somewhat bold refactor of the example sorting tests --- there's now just one test, which is parametrized over the different ways of sorting. It's a bit wonky looking because part of the parametrization is itself in a pytest.mark (the add_conf mark) but the pytest.param IDs are really explicit so hopefully it's not too hard to figure out what's going on there.

On the plus side: one of the added tests revealed that it wasn't actually working to set subsection order with a plain list, so I've fixed that too (I checked by removing the ExplicitOrder() wrapper in MNE-Python's docs --- the doc build fails on SG master and succeeds on this PR).

Copy link
Copy Markdown
Contributor

@lucyleeow lucyleeow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for all your work on this. Especially for fixing the subsection order plain list!

I see a few minigallery directives in the new plot example files. Did you have plans for adding minigallery sort tests? I am very okay with you not doing this though, as you've already done so much work on this PR already.

Otherwise, just a question.

I'll merge in a few days in case @larsoner wants to take a look

Comment on lines +278 to +281
The regex below extracts info from that source-of-truth line to check that the
thumbnail ordering was correct --- unless `expected_order` is provided, in which
case an alternative regex extracts the digit-part of the filename from the crossref,
and compares the order of crossrefs to `expected_order`.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this. I actually did not realise the check was this complicated!

'gallery_dirs': 'ex',
'within_subsection_order': "FileSizeSortKey",
}"""
_subsection_explicit_order_list = pytest.mark.add_conf.with_args(
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was confused about the with_args so I tested it without (i.e. just pytest.mark.add_conf) and the test seemed to work fine. Is it safer/better to use with_args ?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nope, I just misunderstood the pytest docs. After re-reading, it seems you only need with_args if there's a single positional arg to the custom marker and that single positional arg is itself a callable. I'll push a commit removing the with_args part.

"backreferences_dir": "gen_modules/backreferences",
"subsection_order": f"{util_root}.noop_key",
"within_subsection_order": "FileNameSortKey",
"within_subsection_order": CustomSortKey,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes totally agree, this is good as is.

@drammock
Copy link
Copy Markdown
Member Author

drammock commented Nov 4, 2024

I see a few minigallery directives in the new plot example files. Did you have plans for adding minigallery sort tests?

nope, those were just copy-pasted from existing plot_1.py etc, I wasn't planning anything there.

Comment on lines +531 to +533
subsec_order = gallery_conf.get("subsection_order")
if isinstance(subsec_order, list):
gallery_conf["subsection_order"] = ExplicitOrder(subsec_order)
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the change that makes bare lists work for subsection_order (must be wrapped in ExplicitOrder before the call to _get_callables)

@lucyleeow
Copy link
Copy Markdown
Contributor

Thank you @drammock !

@lucyleeow lucyleeow merged commit 7e2adbc into sphinx-gallery:master Nov 6, 2024
@drammock drammock deleted the fix-custom-sort branch December 10, 2024 19:46
scott-huberty added a commit to scott-huberty/mne-bids that referenced this pull request Nov 3, 2025
in MNE-BIDS doc/conf.py we have:

```
sphinx_gallery_conf["within_subsection_order"] = "mne_bids.utils._example_sorter"
```

But AFAICT this functionality wasn't workign prior to sphinx-gallery/sphinx-gallery#1391 , which was included in the release of sphinx-agallery v0.19 (released Feb 12 2025).

https://sphinx-gallery.github.io/stable/changes.html#v0-19-0

Given that v0.19 was released less than 1 year ago, I think it is reasonable to assume that devs/contributors might have a sphinx-gallery version < 0.19 installed locally, in which case building the mne-bids docuemntation will fail, but the reason/solution won't be explicit.

So I think that it is best to be explicit about the minimum required sphinx-gallery version we require to be able to successfully build our docs.
sappelhoff pushed a commit to mne-tools/mne-bids that referenced this pull request Nov 4, 2025
* DOC: Add html-noplot recipe to doc/Makefile

And we also tweak `make view` so that the built documentation opens regardless of whether it was built with make html or make html-noplot

* FIX: Pin sphinx-gallery

in MNE-BIDS doc/conf.py we have:

```
sphinx_gallery_conf["within_subsection_order"] = "mne_bids.utils._example_sorter"
```

But AFAICT this functionality wasn't workign prior to sphinx-gallery/sphinx-gallery#1391 , which was included in the release of sphinx-agallery v0.19 (released Feb 12 2025).

https://sphinx-gallery.github.io/stable/changes.html#v0-19-0

Given that v0.19 was released less than 1 year ago, I think it is reasonable to assume that devs/contributors might have a sphinx-gallery version < 0.19 installed locally, in which case building the mne-bids docuemntation will fail, but the reason/solution won't be explicit.

So I think that it is best to be explicit about the minimum required sphinx-gallery version we require to be able to successfully build our docs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

custom within-subsection sorting doesn't work as documented

3 participants