Skip to content

Improve handling of attribute access on Proxy class #517

@gpauloski

Description

@gpauloski

Describe the Request

The Proxy class redefines common Python attributes (e.g., __module__) as properties such that their access can be forwarded to the proxy's target object.

This causes problems when external code needs to access those attributes on the Proxy class itself (i.e., not an instance of a Proxy). I don't yet have an idea in mind for fixing this, but long-term this would be a very useful thing to fix.

We could potentially try changing from the slots Proxy implementation to the lazy_object_proxy default (cext with fallback to simple).

This has caused some of the following errors:

  1. Dask fails to serialized Proxy types:
    Traceback (most recent call last):
     File "/home/jgpaul/workspace/proxystore-extensions/example.py", line 15, in <module>
        y = client.submit(sum, p)
            ^^^^^^^^^^^^^^^^^^^^^
      File "/home/jgpaul/workspace/proxystore-extensions/venv/lib/python3.11/site-packages/distributed/client.py", line 1943, in submit
        key = funcname(func) + "-" + tokenize(func, kwargs, *args)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/jgpaul/workspace/proxystore-extensions/venv/lib/python3.11/site-packages/dask/base.py", line 964, in tokenize
        hasher = _md5(str(tuple(map(normalize_token, args))).encode())
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/jgpaul/workspace/proxystore-extensions/venv/lib/python3.11/site-packages/dask/utils.py", line 641, in __call__
        meth = self.dispatch(type(arg))
               ^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/jgpaul/workspace/proxystore-extensions/venv/lib/python3.11/site-packages/dask/utils.py", line 626, in dispatch
        toplevel, _, _ = cls2.__module__.partition(".")
                         ^^^^^^^^^^^^^^^^^^^^^^^^^
    AttributeError: 'property' object has no attribute 'partition'
    
  2. Calling help() on a module which imports Proxy fails:
    python -c "import proxystore.proxy as m; help(m)"
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "<frozen _sitebuiltins>", line 103, in __call__
      File "/usr/lib/python3.11/pydoc.py", line 2018, in __call__
        self.help(request)
      File "/usr/lib/python3.11/pydoc.py", line 2079, in help
        else: doc(request, 'Help on %s:', output=self._output, is_cli=is_cli)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/pydoc.py", line 1792, in doc
        pager(render_doc(thing, title, forceload))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/pydoc.py", line 1785, in render_doc
        return title % desc + '\n\n' + renderer.document(object, name)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/pydoc.py", line 483, in document
        if inspect.ismodule(object): return self.docmodule(*args)
                                            ^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/pydoc.py", line 1272, in docmodule
        inspect.getclasstree(classlist, 1), name)]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/inspect.py", line 1300, in getclasstree
        return walktree(roots, children, None)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/inspect.py", line 1269, in walktree
        classes.sort(key=attrgetter('__module__', '__name__'))
    TypeError: '<' not supported between instances of 'str' and 'property'
    

Sample Code

No response

Metadata

Metadata

Assignees

Labels

enhancementNew features or improvements to existing functionality

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions