Skip to content

fix corner cases with slicing#10

Closed
nden wants to merge 76 commits into
embray:modeling/compoundfrom
nden:edge-cases-slicing
Closed

fix corner cases with slicing#10
nden wants to merge 76 commits into
embray:modeling/compoundfrom
nden:edge-cases-slicing

Conversation

@nden

@nden nden commented Dec 17, 2014

Copy link
Copy Markdown

@embray
This is a suggested PR to fix a few minor problems I found with slicing. The examples below did not work as expected:

model = Shift(1, name='shift_x') & Shift(2, name='shift_y') | Rotattion2D(12, name='rotation')
model[1:]
# index does not exist
model[1:6]
#name does not exist
model['shift_x' : 'final']

…gs the same code generation used by custom_model to the main Model class itself for __call__ methods. Now it's generally unnecessary for Model subclasses to redefine __call__ even if they want to update the method signature to reflect the names of their inputs. That is now automatic. Also the Model base class no longer makes any assumptions about the number of a model's inputs or outputs. It assumes no inputs/outputs until otherwise specified in a subclass.
…t came up, interestingly, when trying to call help() on a model class.
…h are no longer needed due to the automatic code generation in the _ModelMeta metaclass.
… generating __init__ methods as well. This allows still more boilerplate to be removed from most Model subclasses (though it still allows individual subclasses to define their own __init__ when needed--see for example the polynomial models).
…orator, since that part is now already handled (and somewhat better) by _ModelMeta.__new__
…onger needs to include its own copy of the make_func_with_sig utility. I also renamed make_func_with_sig to make_function_with_signature as suggested here: astropy#2835 (comment)  Previously I thought the unabbreviated name was a little on the long side.  But on second thought I prefer the clarity here.
… a True-like value, so ensure that we actually get a float when a float result is expected.
…creating new model classes with custom names.
…om for improvement here but this is a start.
…y expression. This requires one input which is a mapping of operators to a numerical precedence so that the correct order of operations can be determined. This ended up looking a lot like ExpressionTree.evaluate, which might be able to be used to implement this with a little refactoring.
…afs. This method still needs more work though :/
…s. In retrospect it should have been blatantly obvious that was a bad idea, and unnecessary anyways. Turns out Python creates a new method object every time you access a method on an object anyways (this is still reasonably efficient, as it uses a cache of already malloc'd PyMethodObjects and just updates the im_func and im_self pointers on those objects).
…il the enhancements to the tree display are complete.
… a new instance is returned, rather than the class itself. If the class is for some reason needed it can still be gotten at, but in most cases it seems to be clearer if we just return an instance.
…om so that it's out of the way while I do a little more reorg.
… process I actually rewrote traverse_postorder() to be, I think, a little clearer as to what's going on. But it still had the same bug (worse even because now it would go into an infinite loop). But simply ensuring that one node does not have two subtrees that are the same *object* seems good enough, I think.
embray and others added 15 commits November 12, 2014 17:21
…not that it really matters I just think tuples look better for cases like this
…es, instances of compound models should be treated as single entities when being used in an expression--their tree should not be added to the tree for the expression they're being used in.
Rotation2D model.

- Rotation2D was using a pre-computed rotation matrix that was computed
  when the model is instantiated (regardless if it's ever evaluated).
  This matrix was never *updated* though, so if the user sets a new
  angle parameter it would still compute the rotation with the old
  rotation matrix.

  In my initial fix for this I thought of using an LRU cache
  (specificlly the functools.lru_cache from Python 3.3+ which I included
  a backport of), which would work for single angles, but which does not
  work for Numpy arrays (and hence arrays of angles, which are
  technically allowed).  I may reintroduce the cache idea again later on
  (perhaps we could cache only single-valued parameters and just not
  array parameters which is fine for most cases) but for now it's an
  unnecessary complication.

- I added a workaround for the problem where parameter values stored in
  radians were being converted from radians to degrees back to radians
  again in order to perform the evaluation.  This is just an ugly
  workaround for now until I've refactored how parameter values are
  stored.
…y wound up revealing a number of seemingly unrelated bugs which are now fixed. I also modified the ExpressionTree.evaluate method so that the 'getter' callable now takes two arguments, the first of which is the node's leaf index, which proves to be useful to have available.
…ccept names yet (only integers) but that will be next. Compound model instances can't be sliced yet, but that is also coming soon.
… exactly motivated that, but I don't think it's necessary anymore (undoing it doesn't break any existing tests at least). In any case this is one small necessary step toward the reworked handling of model instances in compound models.
…nything. Fixed that, and then fixed another bug in the subexpression evaluation algorithm that the now fixed tests revealed.
…d in question is defined on some parent class.
…del classes *and* instances, by name, by index, or some combination thereof. Added a few basic tests of the capability.
… list) rather than a lambda--turns out lambdas will create problems for pickling)
…his converts the values in Model._param_metrics to a dict rather than a tuple for easier understanding of the code (maybe at some point a namedtuple?) and merged the _param_broadcast_shape attribute into _param_metrics (no reason they should be separate)
…ce--ensure that regardless of whether the compound model's template used model classes or instances a *new* model instance is returned whose parameters are linked to the compound model's parameters. This was done for now by changing Model._from_existing to a sharedmethod.
@nden

nden commented Dec 17, 2014

Copy link
Copy Markdown
Author

examples updated

@embray

embray commented Dec 17, 2014

Copy link
Copy Markdown
Owner

Thanks--incidentally I just found some other issues with slicing that only fail on Python 2, but not on Python 3 (where most of my testing is done anymore).

Could you please add a regression test based on this example? Also it looks like this fix might still need some tweaking. Although it did fix the model[1:] case, the other two use cases are still broken:

>>> model[1:6]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "astropy/modeling/core.py", line 1834, in __getitem__
    index = self.__class__._normalize_index(index)
  File "astropy/modeling/core.py", line 1780, in _normalize_index
    stop = check_index(stop)
  File "astropy/modeling/core.py", line 1758, in check_index
    raise IndexError("Model index {0} out of range.".format(index))
IndexError: Model index 6 out of range.
>>> model['shift_x':'rotation']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "astropy/modeling/core.py", line 1834, in __getitem__
    index = self.__class__._normalize_index(index)
  File "astropy/modeling/core.py", line 1780, in _normalize_index
    stop = check_index(stop)
  File "astropy/modeling/core.py", line 1758, in check_index
    raise IndexError("Model index {0} out of range.".format(index))
IndexError: Model index 3 out of range.

In the first case, although 6 is out of range, it should work like Python lists in that an out of range end-point should be equivalent to "take until the end". The second case obviously should also work (it should include 'shift_x' through 'rotation' inclusive).

@embray

embray commented Dec 17, 2014

Copy link
Copy Markdown
Owner

Ack, sorry, I just rebased my compound modeling branch too, so you'll have to rebase this PR as well.

@nden

nden commented Dec 17, 2014

Copy link
Copy Markdown
Author

Well, my intention was to raise an error if the name is not in the model.submodel_names, not to return the model until the end (like lists). The same for indexing. With the WCS use case in mind I consider these a user error. If a user does: model['shift_x' : 'shift'] when they meant shift_y I think this should raise an error instead of returning the wrong model. What is the use case for returning a model[index1:] if the second index is not correct?

@embray

embray commented Dec 17, 2014

Copy link
Copy Markdown
Owner

For a non-existent name, sure. I'm just talking about when using integers, I was trying to keep in the interface consistent with the expectations for integer-based slicing. Using names is different.

@nden

nden commented Dec 17, 2014

Copy link
Copy Markdown
Author

It's too much hassle to go through all conflicts.
Either put in these changes manually or I'll do a new PR when you are done
with your work.

On Wed, Dec 17, 2014 at 1:38 PM, Erik Bray notifications@github.com wrote:

For a non-existent name, sure. I'm just talking about when using integers,
I was trying to keep in the interface consistent with the expectations for
integer-based slicing. Using names is different.


Reply to this email directly or view it on GitHub
#10 (comment).

@embray

embray commented Dec 17, 2014

Copy link
Copy Markdown
Owner

Ah, sorry. Since this was based on an older version of my branch which was since rebased, then a simple rebase doesn't work. Since this is just one commit though the easiest thing to do is reset your branch and then cherry-pick the last commit:

$ git reset --hard embray/modeling/compound
$ git cherry-pick bb74d9c6fb7b879568661dfca8fda77da4c56cbb
$ git push -f origin edge-cases-slicing

I didn't realize when pushing a rebase of my branch that it would screw up existing PRs (it should not be hard for GH to do this automatically). Sorry.

@nden

nden commented Dec 17, 2014

Copy link
Copy Markdown
Author

No problem about the rebase, I'll make a new PR against the updated branch.
Slicing fails on the updated branch in a strange way (without my chamges). It may have had the same problem before, may be I didn't notice. I finally figured out the problem is not with the sliced model but with its repr - regardless of the slice it always prints parameters based on indexing the initial model. I'll leave this out of this PR.

One more question about slicing using integers vs slicing using labels - with labels it includes the stop label in the sliced models, not so with integer inidices. I wonder if this will be confusing.

How shoudl I write the regression tests given that model comparison is not implemented yet? I can compare sliced_model.submodel_names and/or sliced_model._submodels ? Is there a better way?

@embray

embray commented Dec 17, 2014

Copy link
Copy Markdown
Owner

@nden Okay, no problem. A new PR when you're ready to is fine.

I actually noticed that one or two of the slicing-related tests were failing on Python 2 but not on Python 3 (where I'm doing most of my testing now) so I definitely need to look into that. That might be the same problem, but I'm not sure.

One more question about slicing using integers vs slicing using labels - with labels it includes the stop label in the sliced models, not so with integer inidices. I wonder if this will be confusing.

I think in a way it's a little confusing, but there's also precedent for it. The fact that normal (with integers) slicing does not include the end point makes sense for doing index arithmetic and zero-based indexing. But none of that logic applies when using strings. Other libraries that have some objects that can be sliced by string labels take the same approach (see for example, Pandas). It's less confusing to have integer slices work just like slicing lists, but for labels the end-point should be inclusive (otherwise there's no way to make a slice that explicitly includes the end-point via labels).

How shoudl I write the regression tests given that model comparison is not implemented yet? I can compare sliced_model.submodel_names and/or sliced_model._submodels ? Is there a better way?

Looking at submodel_names is a good idea. Some of the other tests like this one also provide a good example. Most of the tests so far just test evaluating a slice and ensuring that it unambiguously returns the correct result. However, I also like the idea of adding tests that explicitly make sure that the correct sub-model is returned with transformations in the correct order.

There isn't otherwise a great API right now to loop over or introspect the models making up a compound model except be looking at ._submodels or at ._tree. I'm definitely open to ideas for this--it's something I'd like to improve (printing a compound model will tell you all about it, which is fine for interactive use, but not as useful for introspection by code).

@nden nden closed this Dec 18, 2014
embray pushed a commit that referenced this pull request Jan 23, 2015
Make sure that astropy_helpers/src gets installed
@nden nden deleted the edge-cases-slicing branch April 30, 2018 12:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants