Updating modeling to use new Compound Models and new Parameter behavior#8769
Conversation
| astropy.modeling | ||
| ^^^^^^^^^^^^^^^^ | ||
|
|
||
| - Major rework of modeling internals. See release notes for details. Eliminates |
There was a problem hiding this comment.
This needs to be moved to section 4.0 and added the PR number. Also, I'm not sure how the rest of the changes have ended up here, probably a rebase would resolve them.
|
@perrygreenfield - This will need a rebase as there are a few clearly unrelated merge commits left in the list, as well as some unrelated changelog entries. |
|
It would be best to merge this as soon as possible so there is time to deal with potential problems and build on top of it. I am pinging people who have worked on modeling before - please review if you have time and interest. @astrofrog @Cadair @cdeil @saimn and anyone else. |
|
@bsipocz rebasing this PR takes a long time, I'll do it again before merging. |
| ^^^^^^^^^^^^^^^^ | ||
|
|
||
| - Major rework of modeling internals. See release notes for details. Eliminates | ||
| support for compound classes (but not compound instances!) |
There was a problem hiding this comment.
| support for compound classes (but not compound instances!) | |
| support for compound classes (but not compound instances!) [#8769] |
pllim
left a comment
There was a problem hiding this comment.
It is quite impossible to eyeball the diff of this PR (at least for me). I'll take it for a spin with synphot soon.
Also need the html_docs job to pass, so we can look at the new rendered doc.
Are you sure all the change log diff is related to this PR?
| @@ -1,9 +1,9 @@ | |||
| .. _compound-models: | |||
| .. .. _compound-models: | |||
There was a problem hiding this comment.
Why not just delete the text instead of commenting it out?
There was a problem hiding this comment.
I agree, this page needs updating
|
To clarify my request - at this point it would be most useful if anyone can run the code in this PR and report problems. Also report any issues related to the general re-design of |
|
@nden @perrygreenfield - not that I expect much effect, but could you please send out an e-mail to astropy-dev about testing this branch? (including a line of how to pip install the branch). It would be helpful to get as much early feedback as possible. |
|
I think simplifying the implementation and limiting compound models to instances is a great change! @perrygreenfield and @nden - Thank you! The error message for I think this is the main feature removed and error some people might get when they used this feature. Can you change the implementation so that the error becomes something like this? I have to admit that I don't understand the old or the new implementation of parameters for compound models, but it sounds like the new one is simpler, so +1. In Gammapy we are not using Otherwise I have the same comment as @pllim above - the diff is so large and contains unrelated stuff (like a fix in a Moffat model) or commented out old docs that it's really hard to review and help polish this PR. Suggest you do the rebase and finish up the PR to the "ready for final review" state and put these changes in master quickly. |
|
So... turns out this broke The hacky stuff starts around https://github.com/spacetelescope/synphot_refactor/blob/373af480b4e4f897dab4ac555b171262e8ce3479/synphot/models.py#L652-L655 . |
|
OK, so it seems that there is some commit duplication that makes the branch rebase a bit tricky, but if duplicates are properly removed all the conflicts can be easily resolved. Here is the rebased version, commit numbers are back to 76 without doing any squashing: |
…responding change in test
|
|
||
| - Major rework of modeling internals. | ||
| See modeling documentation for details. | ||
| `<https://docs.astropy.org/en/v4.0/modeling/changes_for_4.html>`_ . [#8769] |
There was a problem hiding this comment.
this needs to be changed to an intersphinx link to the relevant what's new section, etc, but that can be done when that docs is written up before the release.
|
This is now looking so good, thank you both for all the work you put in it! |
astrofrog
left a comment
There was a problem hiding this comment.
This looks good, thanks for doing the rebase! 👍
The changes_for_4.rst document is empty at the moment, but I'm assuming this will be populated in a follow-up PR.
|
I'm going to go ahead and merge - but as noted above the documentation needs to be updated in a follow-up PR. Thanks @nden and @perrygreenfield! |
|
🎉 Awesome work everyone, sorry I wasn't really able to help. |
|
🎉 This only took slightly less than a year, over 100 commits (if you also count #7945) and as many comments. 😉 |
This is an updated PR which replaces #7945
[This may be more detail than some wish to see] Much of the original rationale was driven by improving the performance. At the time it seemed that compound models were using enormous amounts of memory for complex models (we are talking on the order of 90 component models and approximately 280 parameters). It appeared that parameter objects were being propagated at every node of the expression tree leading to a huge number of parameter object instances. But in doing the work, I don’t actually see how that happened since the previous implementation does go to pains to avoid such parameter creation. It may be that ASDF somehow triggered that regardless, or some other fix made since then has rectified the previous memory issue. Even so, the changes still appeared worthwhile for the reasons listed below.
Removing use of the metaclass from compound models. It is not believed that generating compound classes through expressions is useful enough to justify the complexity introduced by yet another metaclass (the current implementation has the CompoundModel use a metaclass and is also a subclass of Model, which has its own metaclass as well. Besides the complexity of this approach, there are very unexpected results when mixing classes and instances within an expression. The new implementation is more straightforward in approach, though not necessarily shorter in lines of code (though probably as well since calls to other libraries implementing tree handling and such are now eliminated). Essentially, the primary approach is to make CompoundModels intrinsically tree structures, i.e., they are themselves nodes in the expression tree, rather than resulting in a tree attribute that mirrors the implied tree structure.
Changing the semantics of parameters so that parameters are now not shared objects between models of the same class. This has many benefits since the previous behavior led to surprising results (e.g., corresponding parameters in compound models were disconnected from the value contained in the constituent models; change one and the other would not see the change). Making this change makes it simpler to clone models and change aspects of the parameters; something that is quite difficult to do now without resorting to interface violations or tedious construction of the arguments to the class constructor. Making changes to this effectively required a major reimplementation of compound models so these changes were coupled rather than implemented separately.
Changes in Behavior or Interface
This version of modeling has made substantial internal changes to the implementation of modeling and their associated parameters. Some of these changes impact previous behavior. This will summarize the most important changes.
Compound model classes are no longer supported. In previous versions it was possible to create a new model class (not instance!) by doing something like this:
from astropy.modeling.models import Gaussian1D, Const1D
NewCompoundClass = Gaussian1d + Const1D
That no longer is possible. It was removed to make the implementation simpler, as well as eliminate a number of odd results (particularly when combining classes with instances). As far as we know, there are not many uses of this capability, and most such uses have alternatives.
Parameters now store their values within the parameter instance. Previously, all parameter values were stored with the model the parameters were defined with. This leads to differences in behavior. In the past, if one defined a compound model, the parameter values were copied to the compound model instance. If the parameter value was changed in the compound model, that would not be reflected in the parameter value in constituent model that made up part of the compound model. In the current implementation the compound model uses the parameter values within the constituent model so changes to that parameter affect both. With the new implementation, it is possible for multiple models to share the same parameter and thus all be affected by a change in its value. Implementing this change while retaining the syntactic convenience of Descriptors requires some trickery. This was implemented by retaining the definition of parameters at the class level. But now when a model is instantiated, copies of the parameter instances are made and referenced at the model instance level; the existence of these parameter instances in the model’s instance dict effectively shields the class level instances (so long as the class level instance is not a descriptor).
When n_models > 1, previously the shape of the corresponding parameters removed the axis corresponding to the model_set_axis. Now the model_set_axis is part of the parameter shape. To illustrate:
Old:
In [1]: g = Gaussian1D([1,1], [1,2], [2,4], n_models=2)
In [2]: g.amplitude
Out[2]: Parameter('amplitude', value=[1. 1.])
In [3]: g.amplitude.shape
Out[3]: ()
New:
In [1]: g = Gaussian1D([1,1], [1,2], [2,4], n_models=2)
In [2]: g.amplitude
Out[2]: Parameter('amplitude', value=[1. 1.])
In [3]: g.amplitude.shape
Out[3]: (2,)
The implementation of the parameter validator method has changed though the behavior for most cases of use shouldn’t. But note, the current method stands double duty as a decorator and a method depending on the context. In the previous version the context was provided by whether the parameter had a link to a model. If not, it was interpreted as a decorator (it’s complicated; even though the use of the decorator is within a model, at the time the decorator is used, there is no link to the model—no, you don’t want to know the details). Since parameters no longer have links to the model, the method uses the type of the argument to distinguish the context. Callables imply decorator context; numbers imply method context. I recommend the method context be deprecated. A new method called “validate” is now available and recommended for that purpose.
[The following is conditional on whether the new compound model unit behavior is part of the 3.2 release; if not, it isn't needed]
The use of 'imputed' units, i.e., supplying input/output units to a compound model without them but where the component models support the _parameter_units_for_data_units() method is much more restricted in its applicability. That will only work when the compound expression uses the '+' or '-' operators. Past behavior led to sometimes arbitrary assignments of units, and sometimes incorrect units to the parameters.
Generally we expect the new implementation is faster, particularly for more complex models. If you find cases where it is significantly slower, please file an issue.
Other changes involve methods of compound model classes and parameter classes, mostly internal ones (i.e., ones that start with an underscore). The new compound model class no longer uses _CompoundModelMeta.
[by the way, a lot of the work in getting this PR ready was done by @nden]
Testing: this PR has been tested (and PRs filed if necessary) with:
[X] gwcs
[x] jwst
[x] photutils
[x] sunpy
[x] specutils