GH-46145: make `PyStructSequence` compatible with `namedtuple` by FFY00 · Pull Request #108648 · python/cpython

FFY00 · 2023-08-29T16:28:39Z

Issue: Increase PyStructSequence compatibility with collections.namedtuple #108647

Issue: Enhance Object/structseq.c to match namedtuple and tuple api #46145

Signed-off-by: Filipe Laíns <lains@riseup.net>

FFY00 · 2023-08-29T16:46:39Z

Sorry for the force-push mess, I should have taken some more time preparing the PR locally.

Signed-off-by: Filipe Laíns <lains@riseup.net>

…seq_dict Signed-off-by: Filipe Laíns <lains@riseup.net>

sobolevn · 2023-08-29T17:58:20Z

+
+    // Set _fields_defaults to an empty dir, as we don't support defaults
+    defaults = PyDict_New();
+    if (!defaults)


Suggested change

if (!defaults)

if (!defaults) {

nit

Signed-off-by: Filipe Laíns <lains@riseup.net>

FFY00 · 2024-11-17T04:50:38Z

It has been a while and nobody has complained about this, so I am tempted to merge it, but I'd like someone else to also sign off first.

@serhiy-storchaka maybe you could have a look?

Signed-off-by: Filipe Laíns <lains@riseup.net>

serhiy-storchaka

I do not think that it needs _replace after adding more general __replace__. _replace in namedtuples may be deprecated in future.

_make is also redundant. There is a great obstacle in unifying namedtuples and structseqs -- their constructors have incompatible interfaces (this is why implementing __replace__ was important). Namedtuple's _make is like a structseq constructor -- it takes field values as a tuple.

_asdict can easily be implemented as a general external function if we have a set of field names. It does not need a method.

Unfortunately, fields are represented differently in dataclasses and namedtuples. We should first provide an general way to get field names for dataclasses and namedtuples, and then we can add support for structseq.

BTW, I'm planning to implement different mechanism to create struct-like types. They will have fields of different types, so a dataclass-like protocol can be bore useful.

serhiy-storchaka · 2024-11-18T10:28:55Z

+    def test_asdict_unnamed(self):
+        t = _testcapi.MixedNamedTuple((0, 1))
+        self.assertRaises(ValueError, t._asdict)
    def test_copy_replace_all_fields_visible(self):


Empty line.

XuehaiPan

Will need to update dataclasses.asdict(...) and dataclass.astuple(...) and corresponding tests to handle namedtuple and structseq.

namedtule types are determined by heuristic:

def is_namedtuple(obj):
    return isinstance(obj, tuple) and hasattr(obj, '_fields')

cpython/Lib/dataclasses.py

Lines 1448 to 1468 in 9949091

    
           elif issubclass(obj_type, tuple): 
        
               if hasattr(obj, '_fields'): 
        
                   # obj is a namedtuple.  Recurse into it, but the returned 
        
                   # object is another namedtuple of the same type.  This is 
        
                   # similar to how other list- or tuple-derived classes are 
        
                   # treated (see below), but we just need to create them 
        
                   # differently because a namedtuple's __init__ needs to be 
        
                   # called differently (see bpo-34363). 
        
                   # I'm not using namedtuple's _asdict() 
        
                   # method, because: 
        
                   # - it does not recurse in to the namedtuple fields and 
        
                   #   convert them to dicts (using dict_factory). 
        
                   # - I don't actually want to return a dict here.  The main 
        
                   #   use case here is json.dumps, and it handles converting 
        
                   #   namedtuples to lists.  Admittedly we're losing some 
        
                   #   information here when we produce a json list instead of a 
        
                   #   dict.  Note that if we returned dicts here instead of 
        
                   #   namedtuples, we could no longer call asdict() on a data 
        
                   #   structure where a namedtuple was used as a dict key. 
        
                   return obj_type(*[_asdict_inner(v, dict_factory) for v in obj])

cpython/Lib/dataclasses.py

Lines 1522 to 1529 in 9949091

    
           elif isinstance(obj, tuple) and hasattr(obj, '_fields'): 
        
               # obj is a namedtuple.  Recurse into it, but the returned 
        
               # object is another namedtuple of the same type.  This is 
        
               # similar to how other list- or tuple-derived classes are 
        
               # treated (see below), but we just need to create them 
        
               # differently because a namedtuple's __init__ needs to be 
        
               # called differently (see bpo-34363). 
        
               return type(obj)(*[_astuple_inner(v, tuple_factory) for v in obj])

github-actions · 2026-04-13T18:24:38Z

This PR is stale because it has been open for 30 days with no activity.

add _fields to PyStructSequence

668b9e5

Signed-off-by: Filipe Laíns <lains@riseup.net>

bedevere-bot mentioned this pull request Aug 29, 2023

Increase PyStructSequence compatibility with collections.namedtuple #108647

Closed

bedevere-bot added the awaiting core review label Aug 29, 2023

add _asdict to PyStructSequence

6ed9a86

Signed-off-by: Filipe Laíns <lains@riseup.net>

FFY00 force-pushed the gh-108647 branch from fb6383d to 2f38b17 Compare August 29, 2023 16:29

add news

995a2d9

Signed-off-by: Filipe Laíns <lains@riseup.net>

sobolevn reviewed Aug 29, 2023

View reviewed changes

Comment thread Objects/structseq.c Outdated

FFY00 force-pushed the gh-108647 branch from 2f38b17 to 2218ba8 Compare August 29, 2023 16:42

FFY00 changed the title ~~GH-108647: add _fields and _asdict to PyStructSequence~~ GH-108647: add _fields, _field_defaults, and _asdict to PyStructSequence Aug 29, 2023

add _field_defaults to PyStructSequence

c7cf78b

Signed-off-by: Filipe Laíns <lains@riseup.net>

FFY00 force-pushed the gh-108647 branch from 2218ba8 to c7cf78b Compare August 29, 2023 16:44

(from review) use Py_DECREF instead of Py_XDECREF

76c0581

Signed-off-by: Filipe Laíns <lains@riseup.net>

FFY00 added 2 commits August 29, 2023 17:58

add tests

5c3e4a5

Signed-off-by: Filipe Laíns <lains@riseup.net>

use Py_DECREF instead of Py_XDECREF for defaults in initialize_struct…

48b5c2a

…seq_dict Signed-off-by: Filipe Laíns <lains@riseup.net>

sobolevn reviewed Aug 29, 2023

View reviewed changes

don't support namedtuple helpers when unnamed fields are present

173546c

Signed-off-by: Filipe Laíns <lains@riseup.net>

FFY00 changed the title ~~GH-108647: add _fields, _field_defaults, and _asdict to PyStructSequence~~ GH-46145: add _fields, _field_defaults, and _asdict to PyStructSequence Sep 1, 2023

bedevere-bot mentioned this pull request Sep 1, 2023

Enhance Object/structseq.c to match namedtuple and tuple api #46145

Open

This was referenced Dec 4, 2023

[pytree] support PyStructSequence types for Python pytree pytorch/pytorch#113258

Open

refactor: reorder cases for namedtuple and PyStructSequence types metaopt/optree#111

Merged

FFY00 added 8 commits November 17, 2024 04:05

Merge branch 'main' into pythongh-108647

1a4d1f3

Fix linting

c99456f

Signed-off-by: Filipe Laíns <lains@riseup.net>

Fix inconsistent style

8881c96

Signed-off-by: Filipe Laíns <lains@riseup.net>

Fix comment typo

0482b08

Signed-off-by: Filipe Laíns <lains@riseup.net>

Join code that the comment is referring to

df1bd45

Signed-off-by: Filipe Laíns <lains@riseup.net>

Simplify error description

7ec1eb6

Signed-off-by: Filipe Laíns <lains@riseup.net>

Put _asdict after dunder methods in structseq_methods

d658233

Signed-off-by: Filipe Laíns <lains@riseup.net>

Fix comment typo

81a07f2

Signed-off-by: Filipe Laíns <lains@riseup.net>

FFY00 added 5 commits November 17, 2024 08:51

Fix news doc references

de300a6

Signed-off-by: Filipe Laíns <lains@riseup.net>

Sync structseq_asdict error message

c11f544

Signed-off-by: Filipe Laíns <lains@riseup.net>

Add extra checks to _asdict

00ae6a8

Signed-off-by: Filipe Laíns <lains@riseup.net>

Add _replace as an alias to __replace__

7fc1885

Signed-off-by: Filipe Laíns <lains@riseup.net>

Add structseq_make

684d2b2

Signed-off-by: Filipe Laíns <lains@riseup.net>

FFY00 changed the title ~~GH-46145: add _fields, _field_defaults, and _asdict to PyStructSequence~~ GH-46145: make PyStructSequence compatible with namedtuple Nov 17, 2024

serhiy-storchaka reviewed Nov 18, 2024

View reviewed changes

XuehaiPan suggested changes Nov 29, 2024

View reviewed changes

github-actions Bot added the stale Stale PR or inactive for long period of time. label Apr 13, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

GH-46145: make `PyStructSequence` compatible with `namedtuple`#108648

GH-46145: make `PyStructSequence` compatible with `namedtuple`#108648
FFY00 wants to merge 21 commits intopython:mainfrom
FFY00:gh-108647

FFY00 commented Aug 29, 2023 •

edited by bedevere-bot

Loading

Uh oh!

Uh oh!

FFY00 commented Aug 29, 2023

Uh oh!

sobolevn Aug 29, 2023

Uh oh!

FFY00 commented Nov 17, 2024

Uh oh!

serhiy-storchaka left a comment

Uh oh!

serhiy-storchaka Nov 18, 2024

Uh oh!

XuehaiPan left a comment

Uh oh!

github-actions Bot commented Apr 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

	elif issubclass(obj_type, tuple):
	if hasattr(obj, '_fields'):
	# obj is a namedtuple. Recurse into it, but the returned
	# object is another namedtuple of the same type. This is
	# similar to how other list- or tuple-derived classes are
	# treated (see below), but we just need to create them
	# differently because a namedtuple's __init__ needs to be
	# called differently (see bpo-34363).

	# I'm not using namedtuple's _asdict()
	# method, because:
	# - it does not recurse in to the namedtuple fields and
	# convert them to dicts (using dict_factory).
	# - I don't actually want to return a dict here. The main
	# use case here is json.dumps, and it handles converting
	# namedtuples to lists. Admittedly we're losing some
	# information here when we produce a json list instead of a
	# dict. Note that if we returned dicts here instead of
	# namedtuples, we could no longer call asdict() on a data
	# structure where a namedtuple was used as a dict key.
	return obj_type(*[_asdict_inner(v, dict_factory) for v in obj])

	elif isinstance(obj, tuple) and hasattr(obj, '_fields'):
	# obj is a namedtuple. Recurse into it, but the returned
	# object is another namedtuple of the same type. This is
	# similar to how other list- or tuple-derived classes are
	# treated (see below), but we just need to create them
	# differently because a namedtuple's __init__ needs to be
	# called differently (see bpo-34363).
	return type(obj)(*[_astuple_inner(v, tuple_factory) for v in obj])

Uh oh!

Conversation

FFY00 commented Aug 29, 2023 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

FFY00 commented Aug 29, 2023

Uh oh!

sobolevn Aug 29, 2023

Choose a reason for hiding this comment

Uh oh!

FFY00 commented Nov 17, 2024

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

serhiy-storchaka Nov 18, 2024

Choose a reason for hiding this comment

Uh oh!

XuehaiPan left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

FFY00 commented Aug 29, 2023 •

edited by bedevere-bot

Loading