Implement Spitzer Heritage Archive query#102
Conversation
|
Oops, Looks like I added the |
|
Nice. I'll test this out before merging, but I think LAMDA is good to go so hopefully that will make the diff a little cleaner; otherwise I'll ask you to rebase. A development note: when you make a new feature, you should make a new "feature branch" from upstream master. i.e., you would do: to make two different branches, both "branched from" master (there may be other ways to accomplish the same thing). Also, on SHA specifically, there needs to be a way to get access to the image files as well. Presumably that's part of the metadata? |
|
To get rid of the LAMDA commits, you can do: (assuming you have 'upstream' defined - if not, do then remove any lines that are from LAMDA commits and not from SHA commits. After a rebase, you'll need to |
|
Thanks for the help! Git is still confusing for me. I've cleaned up the history with rebase like you suggested and added a function I haven't rigorously tested the routine, but I compared this file URL: >>> t = sha.query(pid=30080)
>>> url = t['accessUrl'][16]
>>> sha.get_img(url)with a version downloaded from |
There was a problem hiding this comment.
I think you could replace this block with:
iofile = io.BytesIO(response.content)
fitsfile = fits.open(iofile)
(just import io and import astropy.io.fits as fits)
see vizier/core.py for an example
|
Good work on the rebase; it looks clean now. In terms of reading binary files, originally we were using but since we're trying to move to requests, I suggest using the other approach I pointed out. At some point, we should probably unify all of this to a single |
|
They do have the metadata available on the API help page. But it actually may be easier in the SHA's case because the queries return a full URL for each file along with all the image/table information. The method I found takes the with open(path, 'wb') as f:
for block in response.iter_content(1024):
f.write(block)The only thing you will need to know is what file-extension to stick on the end of the path name, or else it's not clear that the system will know how to open it. (The 1024 is only to speed up the number of bytes written for each chunk). |
|
Ah, alright, if it's a zip file we can't return the open file. But my first example was to show how to parse a FITS file into a FITS object, rather than write it to disk. We'll need to refactor a lot of the file-getting stuff in the long term, but for now this is fine. One last important thing to do before I merge this: Can you add a document Good stuff, I'm looking forward to using this one.... |
|
I see what you mean with the opening the fits file like |
|
In general, yes. But since we haven't really implemented that for |
|
Sorry, I'm asynchronous with the comments, and yes, I can update those! |
|
Are the |
|
The latter, they're |
|
Yes... this actually raises an important point, though. For |
|
Gotcha, well for the moment, I think it may be most transparent to have two functions a |
|
Could you rebase this against master? They must have diverged in a different PR. |
|
I rebased and did a forced update, did that fix things? |
|
Still says I can't merge. There appears to be a conflict in the If that's what you already did, let me know and we can try something else... |
|
When rebasing I |
|
OK, when you hit that point, that probably means you've hit a conflict. This time, it's pretty easy to fix (almost certainly just one line), but at this point type stuff marking the conflicting area. There are specific tools for dealing with this - try |
|
Sorry to be somewhat lost on this, gvimdiff is my default mergetool, and I can see that |
|
No worries, this is by far the most confusing part of version control. As a vim user, you might be interested in this tool: to set it up in your |
Changed from dictionary dependence to parameters in requests get. Using the standard libs struct parser to parse the fixed width lines.
The query function works with appropriate data types and tested for PID=30080 as on the API help page.
Added a method to get images and spectra from the URLs supplied by the SHA query function. Issues could occur if there are different character encodings or file types, but utf-8 and fits have covered all the tested cases so far.
Commented out the actual get_img calls because they would create a directoy and download files if run, potentially in parts of the file system where the user/process does not have the right permissions.
|
Ok! I think that should have solved it, probably took a bit more googling that it should have. |
|
OK, looks good, thanks! |
Implement Spitzer Heritage Archive query
|
Great, thanks for the help! |
updated docs Makefile to reflect astropy Move the py.test conftest.py file to the astropy directory so it is installed. Changed the import of plugins accordingly. Also updated the .gitignore file so it ignores the Mac OS X .DS_Store file. fixed conf.py to not insert source dir into sys.path typo fixes in conf.py comment Added CSS file from Astropy core package Updated setup.cfg to match Astropy core package Copy bootstrap-astropy theme from astropy remove default.css import from astropy.css - now not needed update conf.py to use bootstrap-astropy theme, mirror astropy's version use installed astropy theme Added templates that inherit from core package Added `_templates` to list of patterns to exclude added comment directing to base of jinja template Update testing to only import pytest when tests are run Added `coverage` option that was added in the Astropy core Removed comments added kwarg options to test runner removed line setting unused _test_runner variable Updated setup.py to bring it in line with the Astropy core Update distribute_setup.py to 0.6.28 Remove unnecessary instructions from README.rst tweak to instructions from @astrofrog's suggestion Changed check in setup.py to syntax in core package to avoid issues when Sphinx is not installed. Update setup.py to reflect re-factored setup code for 0.2 release If necessary, move configuration file to the right place on first import Added open_files option for test() and implemented fix for caught exception in __init__.py Fix scripts/README.rst bug added commented-out edit_on_github section to conf.py and related instructions in README line wrap fixes Add .travis.yml file Add instructions for travis-ci to README _PACKAGE_SETUP_ should be _ASTROPY_SETUP_ (see astropy/astropy#950) Test with different Numpy versions, only test Sphinx with Python 2.7. Ensure that optional dependencies for Sphinx are installed Updated distribute_setup.py to fix compatibility with setuptools 0.7 README typo fixes Switch to using ez_setup.py, and simplify setup.py, following recent changes in Astropy core. Updates to __init__.py to match Astropy core Minor fix Added backward compatibility with Astropy 0.2 parallel option can only be used with Astropy 0.3 or later Re-organize the setup.py so that Astropy is not required for egg_info Revert "Re-organize the setup.py so that Astropy is not required for egg_info" This reverts commit 9aa5339. Replace distribute_setup.py with ez_setup.py and ``setuptools_bootstrap.py`` Support `setup.py test --coverage` in affiliated packages. Simplify package-template usage and updating by requiring fewer files to be edited. Add information about pulling changes to package template Store stuff in "metadata" section, not "affiliated" section. Insert space before comment. Fix documentation build Elaborate on the need to search for .c files in a comment. Change ``clean`` rule to remove generated docs Made necessary by astropy/astropy@b94fd38 Update to newer ez_setup.py Add the current version of ah_bootstrap--for the tests themselves we will always copy over the latest version of ah_bootstrap (so that we can test ah_bootstrap itself), but there should still be a default copy of it in the package-template repo as well Configure the package-template to use astropy_helpers. By default includes astropy_helpers as a submodule. Yes, astropy_helpers is now a submodule of package_template which is a submodule of astropy_helpers update the astropy_helpers version to include fix from c3f841b9e099f506a751b8a6971b15e7e3d6590a Update the astropy_helpers version used in the package_template Update the astropy_helpers version again; use the auto_use feature of ah_bootstrap. Switch to using astropy_helpers for docs build. Updated astropy_helpers version (included fix for Python 3) Update the git submodule path to use http instead--hopefully this will work then on Travis Updating to the actual astropy-helpers Updated astropy-helpers to latest version Fixing docs links in package test function Added example API docs Updated astropy-helpers to latest version. This version includes fixes for the coverage testing in affiliated packages. Use wheels for travis testing Care is taken to use a numpy wheel whose version matches the scipy/astropy wheels. Previous commit messages: Use development astropy for test --coverage Install coverage when needed Add cython to the travis build environment, controlled by an environmental variable Also added a few more words of advice to the travis part of the README Restore -w option to build_sphinx Change URL for numpy-specific wheels and make scipy an optional install Fix latest stable numpy version Several tweaks to improve performance and reduce unnecessary tests Remove debugging statements and cython install in egg_only Don't override package data after it's been determined from setup_package.py files Include astropy_helpers in the affilated package's tarball. Fix astropy#69: Affiliated packages can opt-in to deprecations-as-exceptions removed setuptools_bootstrap.py as it doesn't exist any more in the new package-template added ah_bootstrap and setup.cfg Update astropy_helpers Use MiniConda for affiliated packages Make sure conda’s astropy is removed before pip installing the latest developer version Added more comments to .travis.yml Pin Numpy version Don’t use --parallel with Python 3 More tidying up of .travis.yml file Move astropy dev builds earlier in .travis.yml Since they take longer, it is better to start them sooner for improved load balancing. Import astropy_helpers from inside the docs/ directory Fix a warning that "Could not determine version of package ccdproc" from configuration system (since the __version__ member doesn't yet exist for the package when the configuration file is copied). More detail in example __init__.py Add numpy import to exmaple_mod This ensures that egg_info builds will break if __init__.py is not set up properly Add coveralls --rcfile option Update README for coveralls rcfile Use the version module correctly in edit_on_github Closes astropy#55. Added files to ignore in affiliated package coverage tests Clarify purpose of cextern directory. Updated astropy-helpers to v0.4 Update ah_bootstrap.py from astropy-helpers Replace development build with stable now that astropy 0.4 is out Closes astropy#65 Add info about a specific Sphinx version check. explain readthedocs add released versions of astropy-helper and astropy to rtd-requirements Add a few entries to .gitignore Remove affiliated package instructions from README page update readme with a bit more info Typo fix pointed out by @cdeil Updated astropy-helpers and ah_bootstrap.py to v0.4.1 Added TEMPLATE_CHANGES.md. Fixes astropy#102. MANIFEST.in should exclude pyc *after* including astropy-helpers prune docs/api gitignore manifest template changes udpate Updated astropy-helpers to v0.4.3 Update TEMPLATE_CHANGES.md Fix typo Tagging v0.4.1 Updated astropy-helpers to patched v0.4.3 Update dependency information for open-files option Add note about adding/removing packages to/from pytest header Ensure that conftest.py change works with all versions of Astropy Make sure to update apt-get listings Add jinja2 in .travis.yml Drop python 3.2 testing Drop numpy 1.5 testing and use numpy 1.9 as baseline Use setuptools entry_points for command line scripts Update MANIFEST.in [skip ci] Update TEMPLATE_CHANGES [skip ci] add powered by astropy badge
I noticed that #45 mentioned querying the SHA through the SIAP protocol that they have setup. I've implemented the query to return a astropy.table.Table instance with the right dtypes using the requests library.
I'm not sure if this is redundant with the IRSA sub-module, but the SHA is mostly for querying meta-data and download links for image files, so it seems like it could be of use.