Skip to content

Use gufunc fast parser in Time (alternative)#11768

Merged
mhvk merged 6 commits intoastropy:mainfrom
mhvk:time-fast-parser-gufunc2
May 28, 2021
Merged

Use gufunc fast parser in Time (alternative)#11768
mhvk merged 6 commits intoastropy:mainfrom
mhvk:time-fast-parser-gufunc2

Conversation

@mhvk
Copy link
Copy Markdown
Contributor

@mhvk mhvk commented May 21, 2021

Alternative to #11718. @taldcroft - I think this will be closer to what you had in mind.

@mhvk mhvk added this to the v5.0 milestone May 21, 2021
@mhvk mhvk requested a review from taldcroft May 21, 2021 18:33
@github-actions github-actions bot added the time label May 21, 2021
@taldcroft
Copy link
Copy Markdown
Member

@mhvk - thanks, I'll have a look!

It would certainly help if you could put in a doc string for parser_loop similar to the previous one. Also, comments which define in words all of the variables defined up top would help out a lot. Like the first time I see tm_ptr = args[1], I'd rather not have to guess what's happening or read further in the code to know what that is supposed to be.

@mhvk
Copy link
Copy Markdown
Contributor Author

mhvk commented May 21, 2021

@taldcroft - ok, uploaded a version with a lot more comments. Used skip ci to avoid running everything again (tests passed before). I suggest looking at the full parse_times.c rather than the diff...

@taldcroft
Copy link
Copy Markdown
Member

@mhvk - Thanks for bearing with me. This looks basically good to me now, with just a question about testing. Does this need a test of inputing a strided array? Beyond that, are there features now enabled by doing this via gufunc that didn't work previously?

"Returns\n" \
"-------\n" \
"parser : `~numpy.ufunc`\n" \
" Will parse bytes or unicode."
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you clarify what's meant by unicode here? I guess this means numpy UTF4, or ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, that is really a leftover of what I wanted to happen. Right now, it would be possible to extent to UTF4, but little point until unicode is more directly supported for ufuncs. I replaced this text with 'Suitable for use by ~astropy.time.TimeString formats.'

@taldcroft
Copy link
Copy Markdown
Member

Since this code appears to managing Python references, what assurance do you have that there are no leaks for the various cases (including different kinds of failures).

@taldcroft
Copy link
Copy Markdown
Member

One other thing is that I did build this on my Windows 10 VM using Python 3.7 and confirmed that tests passed. I know that those compiler flags were put in there in response to things failing otherwise, but nice to see that they appear to be no longer needed!

@mhvk
Copy link
Copy Markdown
Contributor Author

mhvk commented May 27, 2021

@taldcroft - oops, the mistake you pointed out before came back in with the reversion... I'll fix that tomorrow.

On the reference counting: I think I got it right, but note that it is strictly for the ufunc and its associated data with information on start/stop, etc., so this is allocated once and not touched again. The actual call to the parser all goes through numpy's ufunc machinery.

On strided arrays, good point, might as well put in some tests with specifically odd strides.

p.s. And happy to hear windows worked. I was hopeful since the tests passed. I remain unsure how numpy avoids the problems the ctypes implementation had.

@mhvk
Copy link
Copy Markdown
Contributor Author

mhvk commented May 27, 2021

OK, addressed the inline comments and added a test with a transposed array to check ordering is as expected.

Copy link
Copy Markdown
Member

@taldcroft taldcroft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks @mhvk!

@mhvk mhvk merged commit 74b54c0 into astropy:main May 28, 2021
@mhvk mhvk deleted the time-fast-parser-gufunc2 branch May 28, 2021 17:59
@nstarman
Copy link
Copy Markdown
Member

nstarman commented Jun 1, 2021

This might break pytest collection.

I'm getting a pytest error when doing >>> pytest astropy/.

This is the error:
ImportError: dynamic module does not define module export function (PyInit__parse_times)

On further examination, this is only appears when astropy is installed with python setup.py develop.

@taldcroft
Copy link
Copy Markdown
Member

@nstarman - what platform was this on? I do all my testing with pytest like that and didn't run into a problem on Mac and Windows, though I use pip install -e . (which I think is basically the same as setup.py develop but not 100% sure).

@nstarman
Copy link
Copy Markdown
Member

nstarman commented Jun 2, 2021

I'm actually having trouble reproducing this. I reinstalled astropy from source and it's gone. I'm hazarding a guess that the problem is related to installing astropy in develop mode and then switching to a branch where there's new compiled code.

@mhvk
Copy link
Copy Markdown
Contributor Author

mhvk commented Jun 2, 2021

Yes, that seems likely - the same would have happened with an update to libwcs or so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants