astropy tables with mixin columns cannot be pickled#4098
astropy tables with mixin columns cannot be pickled#4098embray merged 2 commits intoastropy:masterfrom
Conversation
…n this branch also fix that issue.
|
Changed from "Affects-release" to "Affects-dev" since the problem specifically with pickling multi-dimensional arrays is only in dev (though the issues fixed by #4099 affects v1.0.x). |
…n this branch also fix that issue.
|
So I just looked into this again and it seems like the case I mentioned above is now fixed in astropy-dev, but now certain types of tables can't be pickled (or actually loaded). Here is what happens if you pickle astropy times in two different ways: In [1]: import numpy as np
In [2]: import cPickle as pickle
In [3]: from astropy.table import Table
In [4]: from astropy.time import Time
In [5]: # This one didn't work before...
In [6]: a = [[0,1],[1,2],[2,3]]
In [7]: t = Table([a], names=('a'))
In [8]: pickle.dump(t, open("table.pickle", "wb"))
In [9]: pickle.load(open("table.pickle", "r"))
Out[9]:
<Table length=3>
a [2]
int64
------
0 .. 1
1 .. 2
2 .. 3
In [10]: # Here is an array of Time objects
In [11]: a = np.asarray([Time(55000.0, format='mjd', scale='utc'), \
....: Time(55001.0, format='mjd', scale='utc'), \
....: Time(55002.0, format='mjd', scale='utc')])
In [12]: t = Table([a], names=('a'))
In [13]: pickle.dump(t, open("table.pickle", "wb"))
In [14]: pickle.load(open("table.pickle", "r"))
Out[14]:
<Table length=3>
a
object
-------
55000.0
55001.0
55002.0
In [15]: # Here is a Time-object array
In [16]: a = Time(55000.0+np.arange(3.0), format='mjd', scale='utc')
In [17]: t = Table([a], names=('a'))
In [18]: pickle.dump(t, open("table.pickle", "wb"))
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-18-563cc76ba7b1> in <module>()
----> 1 pickle.dump(t, open("table.pickle", "wb"))
/usr/lib/python2.7/copy_reg.pyc in _reduce_ex(self, proto)
68 else:
69 if base is self.__class__:
---> 70 raise TypeError, "can't pickle %s objects" % base.__name__
71 state = base(self)
72 args = (self.__class__, base, state)
TypeError: can't pickle weakref objects
In [19]: pickle.load(open("table.pickle", "r"))
---------------------------------------------------------------------------
EOFError Traceback (most recent call last)
<ipython-input-19-f5f0b110d204> in <module>()
----> 1 pickle.load(open("table.pickle", "r"))I'm also seeing a "ValueError: non-string names in Numpy dtype unpickling" from another Table that I have, but I haven't figured out a simple way to trigger that error.... |
|
@scottransom - just so you know this is on my radar, just didn't get a chance to look at it yet. |
|
@scottransom @mhvk - I found a bug and hopefully fixed it. Code attached. |
|
@scottransom - if you can reproduce your other problem "ValueError: non-string names in Numpy dtype unpickling" please put that in a new issue. |
|
@taldcroft - this looks good to me. Since this now fixes a bug in 1.0, I set the milestone to 1.0.7. It should still have a changelog entry... |
|
@embray - I've updated the changelog for 1.0.7, but this should definitely get into 1.1 as well. This is ready for merge but I'll let you push the button to make sure the changelog is OK. |
astropy tables with mixin columns cannot be pickled
astropy tables with mixin columns cannot be pickled Conflicts: astropy/table/table.py
astropy tables with mixin columns cannot be pickled
|
Merged and ported. |
…n this branch also fix that issue.
…n this branch also fix that issue.
[EDIT by @taldcroft] - the problem described below is already fixed, but later another issue pickling a table with mixin columns is described. This issue now relates to that problem.
It seems if the shape of a column is multidimensional, then pickling does not work. Here is a toy example given version 1.1.dev13184