Skip to content

Fix problem in to_pandas() introduced in pandas 1.3.0#11921

Merged
taldcroft merged 1 commit intoastropy:mainfrom
taldcroft:table-pandas-1.3-index-name
Jul 4, 2021
Merged

Fix problem in to_pandas() introduced in pandas 1.3.0#11921
taldcroft merged 1 commit intoastropy:mainfrom
taldcroft:table-pandas-1.3-index-name

Conversation

@taldcroft
Copy link
Copy Markdown
Member

@taldcroft taldcroft commented Jul 3, 2021 •
edited by mhvk
Loading

Description

Fix an issue in Table.to_pandas(index=<colname>) where the index column name was not being set properly for the DataFrame index. This was introduced by an API change in pandas version 1.3.0. Previously when creating a DataFrame with the index set to an astropy Column, the DataFrame index name was automatically set to the column name.

Fixes #11918

Checklist for package maintainer(s)

This checklist is meant to remind the package maintainer(s) who will review this pull request of some common things to look for. This list is not exhaustive.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see "When to rebase and squash commits".
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the Extra CI label.
  • Is a change log needed? If yes, did the change log check pass? If no, add the no-changelog-entry-needed label.
  • Is a milestone set? Milestone must be set but astropy-bot check might be missing; do not let the green checkmark fool you.
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate backport-X.Y.x label(s) before merge.

@taldcroft taldcroft added this to the v4.3.1 milestone Jul 3, 2021
@taldcroft taldcroft requested a review from mhvk July 3, 2021 21:40
@taldcroft taldcroft mentioned this pull request Jul 3, 2021
Closed
mhvk
mhvk approved these changes Jul 4, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@mhvk mhvk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change looks good!

Out of scope here perhaps, but if pandas has the concept of names, should all names be set?

Also, might it be that the change in pandas is inadvertent? Perhaps still worth reporting, even with the proviso that perhaps it was happenstance and we're fine with being explicit?

@taldcroft
Copy link
Copy Markdown
Member Author

taldcroft commented Jul 4, 2021

Out of scope here perhaps, but if pandas has the concept of names, should all names be set?

I'm not a pandas expert, but my guess was that the index column is special and having a name is treated differently for the index. Well, I just tried something and I don't understand pandas names, but I am not sure I want to get into it...

In [2]: t = simple_table()

In [3]: tp = t.to_pandas()

In [4]: tp['a'].name
Out[4]: 'a'

In [5]: tp['a'].name = 'b'

In [6]: tp
Out[6]: 
   a    b  c
0  1  1.0  c
1  2  2.0  d
2  3  3.0  e

In [7]: tp['a'].name
Out[7]: 'b'  # ???

I'll open a pandas issue to report the astropy regression.

@taldcroft taldcroft merged commit d576219 into astropy:main Jul 4, 2021
@taldcroft taldcroft deleted the table-pandas-1.3-index-name branch July 4, 2021 02:09
@taldcroft taldcroft mentioned this pull request Jul 6, 2021
Merged
9 tasks
@eteq
Copy link
Copy Markdown
Member

eteq commented Jul 23, 2021

(turns out this didn't backport to 4.3.x correctly so I'm doing it manually in 4e58cf4)

eteq pushed a commit that referenced this pull request Jul 23, 2021
@taldcroft @eteq
Merge pull request #11921 from taldcroft/table-pandas-1.3-index-name
4e58cf4 
Fix problem in to_pandas() introduced in pandas 1.3.0
@eteq eteq modified the milestones: v4.3.1, v4.3 Jul 23, 2021
@eteq eteq mentioned this pull request Jul 23, 2021
Merged
astrofrog pushed a commit that referenced this pull request Aug 17, 2021
@taldcroft @astrofrog
Merge pull request #11921 from taldcroft/table-pandas-1.3-index-name
a6a3121 
Fix problem in to_pandas() introduced in pandas 1.3.0
@astrofrog astrofrog modified the milestones: v4.3, v4.0.6 Aug 19, 2021
@astrofrog
Copy link
Copy Markdown
Member

astrofrog commented Aug 19, 2021

I had to backport this to LTS to fix failures there so updating the milestone

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mhvk mhvk mhvk approved these changes

Assignees

No one assigned

Labels

Bug table

Projects

None yet

Milestone

v4.0.6

Development

Successfully merging this pull request may close these issues.

TST: Test failures with pandas 1.3.0

4 participants

@taldcroft @eteq @astrofrog @mhvk