Conversation
|
This looks interesting. Could you make a post on the numpy-discussion mailing list proposing this enhancement? |
|
@kgabor was this discussed on the mailing list? |
|
There was only a single reply mentioning that |
|
Needs rebase. |
To implement the incremential writing of binary .npy files, support for pre-defined header space is added here.
dcb9f61 to
b734481
Compare
|
Rebased. |
| return d | ||
|
|
||
| def _write_array_header(fp, d, version=None): | ||
| def _write_array_header(fp, d, version=None,fixedheaderlen=0,extrapad=0): |
There was a problem hiding this comment.
pep8-compliant spacing please :-)
@njsmith - Gabor wrote this routine for our pipeline. Recently I spent some time to clean up parts of it and port them to py3, and thus run into the question whether this ever made it to upstream. |
|
I'm happy to resurrect this PR in this release cycle. |
|
@bsipocz yes that would be nice |
|
@bsipocz ping. This missed the 1.18 cutoff |
|
Yes, sadly there is always more on that plate than time. Maybe during the holidays I'll have more time for passion projects (and frankly I need to warm up some old projects anyway and this was part of a pipeline in one such project). |
|
We don't actively use the pipeline any more that relied on this (and thus a patched numpy), but doing some contributions to numpy is very much on my wishlist. So I indeed plan to come back unless this is considered feature creep. |
|
@bsipocz thanks for the reply! I havent taken a closer look, but from other comments, looks like it would be nice to have it in. |
|
OK, let's put a deadline on this then, e.g. if I don't come back and wrap this up by the Scipy sprint this summer, then it's probably time to face the bitter truth and give up on it. |
|
Considering the age of this PR and the fact that it needs to be rebased, I am going to close it. We should probably discuss the API again, but for anyone interested in this work: Please feel free to rebase and open a new PR based on it. |
|
@seberg - 💯 . This was on my mind for a very long time, yet didn't find the time, or the deep breath to finish it off. And I'm the greatest advocate of closing off stale PRs. In my experience, stale closes have the effect of helping let go or give a new boost to dust of old things to finish. |
I would like to add a class for writing one (possibly big) .npy file saving multiple (same dtype, compatible shape) arrays. My use case was saving slowly accumulating data regularly for a long time.
This is a first implementation, opening an existing file for append and reading back parts from a very big .npy file would be straightforward next steps. Please comment this idea.