Skip to content

astropy tables with mixin columns cannot be pickled#4098

Merged
embray merged 2 commits intoastropy:masterfrom
taldcroft:table-pickle-mixins
Nov 12, 2015
Merged

astropy tables with mixin columns cannot be pickled#4098
embray merged 2 commits intoastropy:masterfrom
taldcroft:table-pickle-mixins

Conversation

@taldcroft
Copy link
Copy Markdown
Member

[EDIT by @taldcroft] - the problem described below is already fixed, but later another issue pickling a table with mixin columns is described. This issue now relates to that problem.

It seems if the shape of a column is multidimensional, then pickling does not work. Here is a toy example given version 1.1.dev13184

In [1]: from astropy.table import Table

In [2]: import cPickle as pickle

In [3]: a = zeros(3)

In [4]: t = Table([a], names=('a'))

In [5]: pickle.dump(t, open("table.pickle", "wb"))

In [6]: print t['a']
 a 

---
0.0
0.0
0.0

In [7]: a = zeros((3,2))

In [8]: t = Table([a], names=('a'))

In [9]: pickle.dump(t, open("table.pickle", "wb"))
---------------------------------------------------------------------------
PicklingError                             Traceback (most recent call last)
<ipython-input-9-563cc76ba7b1> in <module>()
----> 1 pickle.dump(t, open("table.pickle", "wb"))

PicklingError: Can't pickle <class 'astropy.table.column.Column'>: it's not the same object as astropy.table.column.Column

In [10]: print t['a']
  a [2]   
----------
0.0 .. 0.0
0.0 .. 0.0
0.0 .. 0.0

In [11]: a = [[0,1],[1,2],[2,3]]

In [12]: t = Table([a], names=('a'))

In [13]: pickle.dump(t, open("table.pickle", "wb"))
---------------------------------------------------------------------------
PicklingError                             Traceback (most recent call last)
<ipython-input-13-563cc76ba7b1> in <module>()
----> 1 pickle.dump(t, open("table.pickle", "wb"))

PicklingError: Can't pickle <class 'astropy.table.column.Column'>: it's not the same object as astropy.table.column.Column

In [14]: t['a']
Out[14]: 
<Column name='a' dtype='int64' shape=(2,) length=3>
0 .. 1
1 .. 2
2 .. 3

@taldcroft
Copy link
Copy Markdown
Member

No object should suffer the shame of not pickling. I'll have a look, but my first guess is that it comes the class trickery that @embray introduced in #3930.

@taldcroft
Copy link
Copy Markdown
Member

I confirm that #3930 introduces this regression. Along the way I discovered a different bug that is fixed in #4099.

@embray
Copy link
Copy Markdown
Member

embray commented Aug 27, 2015

This should be fixed by #4075 which reverts #3930 and replaces it with a different solution which shouldn't have this problem. I'll double-check though.

embray added a commit to embray/astropy that referenced this pull request Aug 27, 2015
@embray embray added Affects-dev PRs and issues that do not impact an existing Astropy release and removed Affects-release labels Aug 27, 2015
@embray
Copy link
Copy Markdown
Member

embray commented Aug 27, 2015

Changed from "Affects-release" to "Affects-dev" since the problem specifically with pickling multi-dimensional arrays is only in dev (though the issues fixed by #4099 affects v1.0.x).

embray added a commit to embray/astropy that referenced this pull request Sep 30, 2015
@scottransom
Copy link
Copy Markdown
Author

So I just looked into this again and it seems like the case I mentioned above is now fixed in astropy-dev, but now certain types of tables can't be pickled (or actually loaded). Here is what happens if you pickle astropy times in two different ways:

In [1]: import numpy as np

In [2]: import cPickle as pickle

In [3]: from astropy.table import Table

In [4]: from astropy.time import Time

In [5]: # This one didn't work before...

In [6]: a = [[0,1],[1,2],[2,3]]

In [7]: t = Table([a], names=('a'))

In [8]: pickle.dump(t, open("table.pickle", "wb"))

In [9]: pickle.load(open("table.pickle", "r"))
Out[9]: 
<Table length=3>
a [2] 
int64 
------
0 .. 1
1 .. 2
2 .. 3

In [10]: # Here is an array of Time objects

In [11]: a = np.asarray([Time(55000.0, format='mjd', scale='utc'), \
   ....:                 Time(55001.0, format='mjd', scale='utc'), \
   ....:                 Time(55002.0, format='mjd', scale='utc')])

In [12]: t = Table([a], names=('a'))

In [13]: pickle.dump(t, open("table.pickle", "wb"))

In [14]: pickle.load(open("table.pickle", "r"))
Out[14]: 
<Table length=3>
   a   
 object
-------
55000.0
55001.0
55002.0

In [15]: # Here is a Time-object array

In [16]: a = Time(55000.0+np.arange(3.0), format='mjd', scale='utc')

In [17]: t = Table([a], names=('a'))

In [18]: pickle.dump(t, open("table.pickle", "wb"))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-18-563cc76ba7b1> in <module>()
----> 1 pickle.dump(t, open("table.pickle", "wb"))

/usr/lib/python2.7/copy_reg.pyc in _reduce_ex(self, proto)
     68     else:
     69         if base is self.__class__:
---> 70             raise TypeError, "can't pickle %s objects" % base.__name__
     71         state = base(self)
     72     args = (self.__class__, base, state)

TypeError: can't pickle weakref objects

In [19]: pickle.load(open("table.pickle", "r"))
---------------------------------------------------------------------------
EOFError                                  Traceback (most recent call last)
<ipython-input-19-f5f0b110d204> in <module>()
----> 1 pickle.load(open("table.pickle", "r"))

I'm also seeing a "ValueError: non-string names in Numpy dtype unpickling" from another Table that I have, but I haven't figured out a simple way to trigger that error....

@taldcroft
Copy link
Copy Markdown
Member

@scottransom - just so you know this is on my radar, just didn't get a chance to look at it yet.

@taldcroft taldcroft changed the title astropy tables with multidimensional columns cannot be pickled astropy tables with mixin columns cannot be pickled Nov 8, 2015
@taldcroft
Copy link
Copy Markdown
Member

@scottransom @mhvk - I found a bug and hopefully fixed it. Code attached.

@taldcroft taldcroft added Affects-release and removed Affects-dev PRs and issues that do not impact an existing Astropy release labels Nov 8, 2015
@taldcroft
Copy link
Copy Markdown
Member

@scottransom - if you can reproduce your other problem "ValueError: non-string names in Numpy dtype unpickling" please put that in a new issue.

@mhvk
Copy link
Copy Markdown
Contributor

mhvk commented Nov 10, 2015

@taldcroft - this looks good to me. Since this now fixes a bug in 1.0, I set the milestone to 1.0.7. It should still have a changelog entry...

@taldcroft
Copy link
Copy Markdown
Member

@embray - I've updated the changelog for 1.0.7, but this should definitely get into 1.1 as well. This is ready for merge but I'll let you push the button to make sure the changelog is OK.

embray added a commit that referenced this pull request Nov 12, 2015
astropy tables with mixin columns cannot be pickled
@embray embray merged commit c3b3f70 into astropy:master Nov 12, 2015
embray added a commit that referenced this pull request Nov 12, 2015
astropy tables with mixin columns cannot be pickled
Conflicts:
	astropy/table/table.py
embray added a commit that referenced this pull request Nov 12, 2015
astropy tables with mixin columns cannot be pickled
@embray
Copy link
Copy Markdown
Member

embray commented Nov 12, 2015

Merged and ported.

@taldcroft taldcroft deleted the table-pickle-mixins branch November 16, 2015 14:01
dhomeier pushed a commit to dhomeier/astropy that referenced this pull request Dec 17, 2015
dhomeier pushed a commit to dhomeier/astropy that referenced this pull request Jun 12, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants