Skip to content

AutoPropertyObject: support abstract and class properties#8393

Merged
feerrenrut merged 15 commits into
nvaccess:masterfrom
BabbageCom:abstract
Dec 13, 2018
Merged

AutoPropertyObject: support abstract and class properties#8393
feerrenrut merged 15 commits into
nvaccess:masterfrom
BabbageCom:abstract

Conversation

@LeonarddeR

@LeonarddeR LeonarddeR commented Jun 13, 2018

Copy link
Copy Markdown
Collaborator

Link to issue number:

Closes #8294
fixes #8652
closes #8658

Summary of the issue:

There are several classes in NVDA that are quite abstract, i.e. in the contentRecog, mathPres, NVDAObjects and textInfos modules. These classes can't be used by themselves, but they are building blocks that need to be the base classes of other objects. These base classes define what methods and properties have to be implemented on a sub-class. In the current situation, this can only be found out either by experience, reading doc strings or finding raise statements for NotImplementedError.

Description of how this pull request fixes the issue:

This pr starts using the python abc module. For more information, read the Python docs. Using abstract methods and properties eases debugging, because as long as a subclass of an abstract class hasn't implemented all the required methods and properties, it can't be instantiated. For example, you can no longer create a SettingsDialog or SettingsPanel that doesn't have an overridden makeSettings method.

  1. Some of the methods and properties I could think of that are in the current source code are now made abstract. As pointed out in Use pythonic abstract classes instead of just raising NotImplementedError for unimplemented methods #8294, it wasn't my intention to go over the whole source code. Abstract methods or properties that aren't marked as such and that are not a part of this pr can be made abstract in future pull requests if desired.

  2. AutoPropertyObject is a special case. We can't simply treat auto properties that need to be abstract as abstract properties, it requires some extra magic and changes to baseObject.Getter. I decided to implement abstract support on top of baseObject.Getter instead of creating baseObject.AbstractGetter, since this would also require baseObject.AbstractCacheGetter and therefore would make the code less readable.

To make an auto property abstract, you would use a strategy that is similar to making cached getters. E.g.:

class Test(baseObject.AutoPropertyObject):
    _abstract_x = True
    def _get_x(self):
        return True

My initial idea was to decorate all get* and set* methods, but that would cause an inconsistency between NVDA's core and the abc module. More importantly, it would fail if you'd make a get* method abstract and then override it by setting a completely new value on a subclass.

  1. As a bonus, I made baseObject.Getter support class methods, i.e.
    @classmethod
    def _get_classProperty(cls):
        return True

This is only supported for getters, not for setters. In this case, the decoration occurs on a per method basis. This is currently not used somewhere in the core, but I've always wanted to use it. :)

  1. This pr also introduces the use of six.with_metaclass to assign meta classes, in order to ensure python 3 compatibility. Though I initially preferred the add_metaclass decorator (Python3: incompatible metaclass syntax #8652) this does not work for the NVDAObject class, probably because the decorator adds the meta class after the construction of the class. IN the NVDAObjects module, I eliminated the import of new.instancemethod, which wasn't used anywhere within this file and therefore was an unneeded import.

Testing performed:

Unit tests

Known issues with pull request:

None

Change log entry:

@LeonarddeR LeonarddeR requested review from feerrenrut and josephsl July 3, 2018 18:50
Comment thread source/gui/guiHelper.py
self.dialogDismissButtonsAdded = True
return buttons

class SIPABCMeta(wx.siplib.wrappertype, ABCMeta):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this need to be used with wx subclasses?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WX Python 3 used swig to wrap WX C++ classes, WX Python Phoenix uses sip.

>>> type(wx.Dialog)
<type 'sip.wrappertype'>

If we don't create SIPABCMeta and just assign ABCMeta as meta class to SettingsDialog or SettingsPanel, NVDA will fail to start due to a meta class conflict, since ABCMeta and wx.siplib.wrappertype are unrelated to each other. You can't assign multiple meta classes to a class.

def _get_x(self):
return True

def _set_x(self, value):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about abstract set? If set and get need different patterns, perhaps _abstract_get_x and _abstract_set_x

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the current implementation, you mark an entire property as abstract. Note that not the methods, but the property itself is considered abstract, which is also the case when using abc.abstractproperty.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is clearer to me now thanks to your other explanation. However, I'm concerned that its not so obvious from the syntax used to mark the property as abstract. Essentially what _abstract_x is saying is "use magic to declare that this class must have a property called x but its implementation is left to subclasses". It makes me wonder though.. is there any point in declaring _get_x and _set_x methods. This makes x look very much NOT abstract.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks odd, but is behavior defined by Python. It might be helpful to read the docs for the Abstract module

In short: an abstract method or or property doesn't mean that the method or property should be left unimplemented, since a sub class can call the method or get the property of the abstract base class.

Also note from the Python docs:

C++ programmers should note that Python’s virtual base class concept is not the same as C++’s.

Does this help?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In short: an abstract method or or property doesn't mean that the method or property should be left unimplemented, since a sub-class can call the method or get the property of the abstract base class.

But it doesn't have to be implemented in the base class, right? Except there is the note in the AutoPropertyObject about set overriding the property.

C++ programmers should note that Python’s virtual base class concept is not the same as C++’s.

Hahaha this would help a lot more if the differences were described.

I think I must be missing the point when it comes to properties. I assume the goal is: allow a base class to declare that a certain property must exist in the construction of a subclass, without necessarily providing the implementation for that property in the base class?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@feerrenrut commented on 7 aug. 2018 10:57 CEST:

Note that within AutoPropertyObject, there is no such thing as a read only property. When someone only implements _get_x, setting x will override the property completely.

It would be nice if we could find a way to catch the situation where _get_x is provided but not _set_x. Is there is a situation in which this behaviour is desirable?

There are actually quite a lot of cases where _get_x is declared and _set_x is not. See displayModel.DisplayModelTextInfo, _get_backgroundSelectionColor and _get_foregroundSelectionColor. These cases perfectly explain the use of this functionality, namely fetching a value the first time the property is retrieved.

What do you actually want to catch exactly? I'd rather catch cases where set is implemented and get isn't, because those make less sense to me.

Either decorating _get_x or _set_x with abstractmethod is dangerous as I noted above

But if it's not abstract, then the base class should be providing it:

class Test(AutoPropertyObject):
...

Your example looks nice, but I"m afraid that it doesn't work as you expect it to work, or I"m just not getting what you're trying to point out.

class Test(AutoPropertyObject):

	def _get_x(self):
		return 42

	def _set_x(self, val):
		print("Magic setter")

class Subtest(Test):
	x = 84

test = Test()
print(test.x)
42
test.x = "random"
"Magic setter"
print(test.x)
42

subtest = Subtest()
# Note that setting x on the subclass wil override x completely, so it will abandon both the getter and the setter. Therefore, overriding x when it is marked as abstract should make it no longer abstract?.
print(subtest.x)
84
# Note that this does no longer print anything.
subtest.x = "random"
print(subtest.x)
84

I think I can live with it if we decide to do the abstraction at the method level, it might look easier to understand.

Thinking about the example I gave above, I guess that doing abstraction at the method level will also cause confusion anyway..

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you actually want to catch exactly?

In the case of displayModel.DisplayModelTextInfo, someone accidentally sets backgroundSelectionColor overriding the property entirely, meaning that the implementation for _get_backgroundSelectionColor no longer gets called. Perhaps this is intentional, but it means that there is a disconnect between the set value and the underlying object.

I'm just not getting what you're trying to point out.

I just had a play with this in the python console. I wasn't aware that the autoProperty would not fall back to the base classes definition (say of _set_x). Eg given:

class base(AutoProperty):
    def _get_x(self):
        return 42

    def _set_x(self, val):
        print("in base _set_x")

class sub(base):
    def _get_x(self):
        return 84

I expected:

b = base()
b.x # returns 42
b.x = "hello" # prints "in base _set_x"
b.x # returns 42
s = sub()
s.x # returns 84
s.x = "hello" # expected it to print "in base _set_x", actually does not.
s.x # expected to return 84, but returns "hello"

Are you aware of a reason for this behaviour? It seems to break the polymorphic behaviour that I would expect. I suppose it could be that I'm tying the interface of the property to the interface of the class, and instead the interface of the property (and it's type) should not be considered part of the class that holds it, merely the existence of a property called x

@LeonarddeR LeonarddeR Aug 8, 2018

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Woops, I pressed reply too quickly.

@feerrenrut commented on 8 aug. 2018 04:20 CEST:

Are you aware of a reason for this behaviour?

Yes, see this code snipped from baseObject.

			if not g:
				# There's a setter or deleter, but no getter.
				# This means it could be in one of the base classes.
				for base in bases:
					g = getattr(base,'_get_%s'%x,None)
					if g:
						break

I agree that it is confusing that AutoPropertyType digs in base classes to find an already set getter, but it does not do this for setters and deleters. I didn't care much, as it has been this way since ages, at least since 2009. However, it is certainly a bit unclear as of what the initial ittentions were when @jcsteh wrote this.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@feerrenrut: Do you want me to change anything in this behaviour, or do you have any other concerns?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I think it's ok for now.

raise NotImplementedError

_abstract_bookmark = True
def _get_bookmark(self):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why can't this also just use @abstractmethod?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because that would mark _get_bookmark as an abstract method. Imagine the following subclass:

class Test(textInfos.TextInfo):
    Bookmark = None

Here, bookmark is overridden at the class level and therefore should no longer be abstract. However, when we mark _get_bookmark as an abstract method, the subclass wouldn't instantiate because _get_bookmark is still abstract. To summarize, using abstractmethod for magic get and set methods will make it impossible to override a property at the subclass by means of the method explained above.

I hope this clears up things a little bit.

@LeonarddeR LeonarddeR requested a review from feerrenrut July 30, 2018 18:31

class SubclassedAutoPropertyObjectWithImplementedProperty(AutoPropertyObjectWithAbstractProperty):

def _get_x(self):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is part of what threw me off. The property is considered implemented if either of _get_x or _set_x are implemented? Another concern, is how does the base class declare that the intended interface is "read-only" for the property.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The property is considered implemented if either of _get_x or _set_x are implemented?

Yes, either set, get or del.

This would be the pure pythonic way

class foo(object):
    __metaclass__ = ABCMeta

    @abstractproperty
    def bar(self):
        return 42

class foo2(foo):

    @property
    def bar(self):
        return super(foo2, self).bar*2

baseObject will behave like abstractproperty if _abstract_x is True, else it behaves like property.

Another concern, is how does the base class declare that the intended interface is "read-only" for the property.

Could you elaborate? I assume you mean somehting like this:

class foo(object):
	@property
	def bar(self):
		return 42

inst=foo()
inst.bar=42*2

Traceback (most recent call last):
  File "<console>", line 1, in <module>
AttributeError: can't set attribute

From the doc string of AutoPropertyObject:

If there is a _get_x but no _set_x then setting x will override the property completely.

So indeed, this differs from how python properties behave normally.

Comment thread source/baseObject.py Outdated
cache=cacheByDefault

abstract=dict.get('_abstract_%s'%x,False)
if g and not s and not d:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would applying deorgan's law make this more clear to a human?
if g and (not s and not d)
turns into
if g and not (s or d)
or in other words, if it's a getter, and neither a setter or deleter

Comment thread source/baseObject.py

if newAbstractProps or oldAbstractProps:
# The __abstractmethods__ set is frozen, therefore we ought to override it.
self.__abstractmethods__=(self.__abstractmethods__|newAbstractProps)-oldAbstractProps

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be equivelent to instead do a set intersection with newAbstratProps & abstract eleminating oldAbstractprops?

@LeonarddeR LeonarddeR Aug 1, 2018

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean, just self.__abstractmethods__ = (self.__abstractmethods__&newAbstractProps) . That would also eliminate abstract methods that aren't abstract properties at all.

Comment thread source/baseObject.py
if abstract:
newAbstractProps.add(x)
elif x in self.__abstractmethods__:
oldAbstractProps.add(x)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the case where this occurs only within subclasses?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, as it deals with removing the abstract properties from _-abstractmethods that are no longer abstract.

feerrenrut
feerrenrut previously approved these changes Aug 22, 2018
@feerrenrut

Copy link
Copy Markdown
Contributor

Are you planning on updating this as per #8652 (comment)

If not, then you can go ahead and merge this.

@LeonarddeR LeonarddeR removed the request for review from josephsl December 5, 2018 13:17
@LeonarddeR

Copy link
Copy Markdown
Collaborator Author

@feerrenrut wrote:

Are you planning on updating this as per #8652 (comment)

If not, then you can go ahead and merge this.

Jup, this has been updated to cover #8652

@feerrenrut feerrenrut merged commit 2d8f08a into nvaccess:master Dec 13, 2018
@nvaccessAuto nvaccessAuto added this to the 2018.4 milestone Dec 13, 2018
@Adriani90

Copy link
Copy Markdown
Collaborator

wow thank you for this huge work guys. This is indeed a big milestone for future developments!

@LeonarddeR LeonarddeR deleted the abstract branch February 7, 2019 15:49
feerrenrut pushed a commit that referenced this pull request Aug 28, 2019
…ed (PR #10102)

When we introduced abstract auto (class) properties in baseObject (#8393), we introduced an exception for abstract class properties, as they aren't supported by Python 2. In Python 3, abstract class properties are supported.
JulienCochuyt added a commit to accessolutions/nvda that referenced this pull request Nov 17, 2020
feerrenrut pushed a commit that referenced this pull request Nov 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

5 participants