ingest hyp3 by forrestfwilliams · Pull Request #542 · insarlab/MintPy

forrestfwilliams · 2021-03-24T22:22:27Z

Description of proposed changes

The ASF HyP3 on-demand InSAR service is a great option for creating interferograms, but MintPy is not currently set up to ingest these products. This pull request creates a prep_hyp3.py workflow that would allow for the ingestion of these products.

Changes

Creates prep_hyp3.py for the creation of rsc metadata files for HyP3 products
Adds hyp3 to load_data.py processor list, and sets up reference to prep_hyp3 when hyp3 is the processor.
Has readfile.read process HyP3 products as GDAL tifs

Resolves Issue: #540

HyP3 is an on-demand service for producing Sentinel-1 interferograms. prep_hyp3.py provides a utility to produce MintPy-readable rsc files so that HyP3 data can be loaded in MintPy. See Issue: insarlab#540

Adds hyp3 to load_data.py processor list, and sets up reference to prep_hyp3 when hyp3 is the processor. See Issue: insarlab#540

See Issue: insarlab#540

Also removed commented import statements

mintpy/prep_hyp3.py

yunjunz · 2021-03-25T04:20:22Z

This is awesome @forrestfwilliams! I only have a few minor suggestions to complete this support:

add HyP3 with link to the docs/README
add hyp3 to the comment of mintpy.load.processor in mintpy/defaults/smallbaselineApp.cfg
could you add an example directory structure and its template options to docs/dir_structure.md? That will be a good place to add some descriptions and/or links on the tutorials as well if available.
if it's not big trouble, could you please send me a test dataset for me to try it as well?

forrestfwilliams · 2021-03-25T19:02:46Z

Hi @yunjunz I'll get started on these changes! What is the best way to get the test dataset to you?

yunjunz · 2021-03-25T20:10:55Z

Hi @forrestfwilliams, I just sent you a link to my Box; you could also upload it to Google Drive if you have enough space there.

forrestfwilliams · 2021-03-25T20:56:29Z

Hi @yunjunz, I've uploaded test data to the box drive, it should match the directory structure I've described in docs/dir_structure.md. I've only included two interferograms to decrease the file size, but let me know if you would like more.

yunjunz · 2021-03-25T21:31:30Z

Thank you @forrestfwilliams, the doc looks great.

For the test data, could you send me a few more to have some redundacy? You have sent me date1_date3, date2_date3, maybe add date1_date2?

The LAT/LON_REF1/2/3/4 attributes are in a different order for ascending and descending geometries. This commit reworks the scirpt so that this difference can be accounted for. The dem file origionally did not have an external metadata file, so adding in this functionally created a large changes in the script. Also updated docs/dir_structure.md to reflect these changes. See Pull Request: insarlab#542

forrestfwilliams · 2021-03-26T02:11:53Z

@yunjunz I've made all the changes we've discussed and the updated example data is in the box folder!

yunjunz · 2021-03-26T03:17:11Z

@forrestfwilliams In the newly added 1st interferogram, I did not find the *_clip.tif, could you make sure?

We need incidence angle and slant range distance for tropospheric delay correction and DEM error correction. The ideal case would be the pixel-wised 2D matrix. If they are not available, a mean value also works, in this case, could you add the following two attributes for it? They will be used by geometryDict during load_data.py to write the mean 2D matrix. This is optional. Let me know if you prefer to deal with it in another PR please.

SLANT_RANGE_DISTANCE =   # mean slant range distance between the satellite and the ground
INCIDENCE_ANGLE =   # mean incidence angle in degree

yunjunz

This PR itself works fine without errors, so I think it's ready to be merged. Thank you @forrestfwilliams!!!

There are a few more things to consider, to better support the full functionalities of MintPy. Maybe for future PRs, unless you want to do it in this PR? @forrestfwilliams Let me know please.

1. In the newly added 1st interferogram, I did not find the *_clip.tif, could you make sure @forrestfwilliams?
2. We need incidence angle and slant range distance for tropospheric delay correction and DEM error correction. The ideal case would be the pixel-wised 2D matrix. If they are not available, a mean value also works, in this case, could you add the following two attributes for it? They will be used by geometryDict during load_data.py to write the mean 2D matrix.

SLANT_RANGE_DISTANCE =   # mean slant range distance between the satellite and the ground
INCIDENCE_ANGLE =   # mean incidence angle in degree

3. The "meter" unit is not properly displayed in view.py currently, which requires some changes (I will probably try it sometime this weekend or next week). For now, one could run view.py --coord radar to force the display in radar coordinate. (#548)

forrestfwilliams · 2021-03-27T00:00:55Z

@yunjunz 1. was my mistake, sorry. I've uploaded the correct version of the example data to the box folder. As for 2. and 3. let's fix them in a future PR. incidence angle and slant range distance are not available in the current version of HyP3, but the project is still in Beta, so they'll likely be open to adding this data as an optional output. Let's go ahead and merge!

+ create `prep_hyp3.py` to prepare rsc metadata files for ASF HyP3 data, which is an on-demand service for producing Sentinel-1 interferograms. + load_data.py: add hyp3 to the processor list, and sets up a reference to prep_hyp3 when hyp3 is the processor. + utils.readfile: read HyP3 products as GDAL TIFF files + docs/dir_structure: add example directory structure and template options for HyP3 + prep_hyp3/aria: fix bug of LAT/LON_REF1/2/3/4, considering their differences between asc /desc geometries.

forrestfwilliams added 5 commits March 25, 2021 11:06

Create prep_hyp3.py to prepare rsc metadata files for hyp3 data

cac7e7b

HyP3 is an on-demand service for producing Sentinel-1 interferograms. prep_hyp3.py provides a utility to produce MintPy-readable rsc files so that HyP3 data can be loaded in MintPy. See Issue: insarlab#540

Add hyp3 as a processor to load_data.py

d9082ab

Adds hyp3 to load_data.py processor list, and sets up reference to prep_hyp3 when hyp3 is the processor. See Issue: insarlab#540

Have readfile.read process HyP3 products as GDAL tifs

be238f3

See Issue: insarlab#540

Removed trailing whitespace from prep_hyp3.py

a3e1862

Removed trailing whitespace from prep_hyp3.py, second attempt

e804839

Also removed commented import statements

yunjunz self-requested a review March 25, 2021 03:28

yunjunz added 2 commits March 24, 2021 21:00

load_data: add hyp3 to read_inps_dict2geometry_dict_obj()

41bd5ac

prep_hyp3: try/except import gdal

b1fc8bf

yunjunz reviewed Mar 25, 2021

View reviewed changes

mintpy/prep_hyp3.py Outdated Show resolved Hide resolved

yunjunz reviewed Mar 25, 2021

View reviewed changes

mintpy/prep_hyp3.py Outdated Show resolved Hide resolved

Added HyP3 readme and directory structure docs

949e2d0

forrestfwilliams added 5 commits March 26, 2021 13:54

Add in deletion of GDAL object once completed

6f00ba4

prep_aria: change LAT/LON_REF1/2/3/4 for ascending geometries

230e2ac

Removed trailing whitespace

7877dbe

Fix dir_structure.md typo

0d064d0

yunjunz added 3 commits March 25, 2021 19:16

Update prep_aria.py

97ab6b3

Update dir_structure.md

bc83813

Update prep_hyp3.py

543d8be

yunjunz approved these changes Mar 26, 2021

View reviewed changes

yunjunz merged commit 4d08d5a into insarlab:main Mar 27, 2021

yunjunz mentioned this pull request Apr 17, 2021

(ts)view: miscellaneous updates #548

Merged

4 tasks

forrestfwilliams deleted the ingest_hyp3 branch June 14, 2021 01:28

This was referenced Jan 17, 2022

Remove reference to GUNW products in overview of HyP3 directory struc… yunjunz/MintPy#6

Closed

Remove GUNW reference from HyP3 directory structure #724

Merged

yunjunz mentioned this pull request Jul 5, 2023

fix bug in gdal -> mintpy corner coordinate convention #1034

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ingest hyp3#542

ingest hyp3#542
yunjunz merged 16 commits intoinsarlab:mainfrom
forrestfwilliams:ingest_hyp3

forrestfwilliams commented Mar 24, 2021

Uh oh!

Uh oh!

Uh oh!

yunjunz commented Mar 25, 2021

Uh oh!

forrestfwilliams commented Mar 25, 2021

Uh oh!

yunjunz commented Mar 25, 2021

Uh oh!

forrestfwilliams commented Mar 25, 2021

Uh oh!

yunjunz commented Mar 25, 2021

Uh oh!

forrestfwilliams commented Mar 26, 2021

Uh oh!

yunjunz commented Mar 26, 2021 •

edited

Loading

Uh oh!

yunjunz left a comment •

edited

Loading

Uh oh!

forrestfwilliams commented Mar 27, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

forrestfwilliams commented Mar 24, 2021

Uh oh!

Uh oh!

Uh oh!

yunjunz commented Mar 25, 2021

Uh oh!

forrestfwilliams commented Mar 25, 2021

Uh oh!

yunjunz commented Mar 25, 2021

Uh oh!

forrestfwilliams commented Mar 25, 2021

Uh oh!

yunjunz commented Mar 25, 2021

Uh oh!

forrestfwilliams commented Mar 26, 2021

Uh oh!

yunjunz commented Mar 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yunjunz left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

forrestfwilliams commented Mar 27, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yunjunz commented Mar 26, 2021 •

edited

Loading

yunjunz left a comment •

edited

Loading