Add support for RAiDER tropo layers by sssangha · Pull Request #335 · aria-tools/ARIA-tools

sssangha · 2022-12-21T09:48:07Z

Addresses Issue #334

Implemented end-to-end support for RAiDER tropo layers in a version 1c GUNW product. This includes setting and passing the appropriate dictionary and product layer keys, as well as interpolating and intersecting the tropo layers with a DEM.

Note a couple of caveats:

Unlike the geometry layers, height level information is not readily transparent through the metadata. So for the time-being, I've made some assumptions based off of this RAiDER discussion: Defining GUNW cubes dbekaert/RAiDER#444. See lines 721-731 in extractProduct.py. Again, I understand this is not completely appropriate, so I'd like to touch bases with the team on this to set the correct values. Furthermore, the regular grid interpolator used here does not allow for descending height values, so I have to flip the input array vertically before interpolation.
The program runs end-to-end now without issue and generates interpolated/intersected outputs with the expected dimensions and spatial extents. However, the outputs appear strange -- looks like clearly something went wrong with the interpolation. I assume it may be related to the above assumptions that were made.
Below from left to right: amplitude, troposphereWet,troposphereHydrostatic

Replicate test:

Using a test product from @bbuzz31 , the following was my commandline input:
ariaExtract.py -l 'amplitude,troposphereWet,troposphereHydrostatic' -nt 10 -f "products/S1*.nc" -d Download -of ISCE -w test

sssangha · 2022-12-22T02:04:23Z

Following our convos, I've located the appropriate height levels variable within the netcdf and made changes accordingly: https://github.com/aria-tools/ARIA-tools/blob/sss_raidersup/tools/ARIAtools/extractProduct.py#L629-#L645

Note from this height field, I realized the heights were already in ascending order, so I reverted the tweak I made to flip the tropo layers.

tools/ARIAtools/extractProduct.py

bbuzzanga

Incorporates #329

…able

update GACOS function to agree with MintPy's

add tropoTotal

sssangha · 2023-01-24T00:54:48Z

@dbekaert regarding your note on another PR, looks like the build is failing because it's running out of the allotted 4Gb RAM in the container under the free plan. This explains why we were unable to replicate it locally.

Aside from finding the root of this bottleneck, the only other two options is to either subscribe to the "Performance Plan" or switch from a docker container to a machine executor by tweaking the config file.

Do not assume tropo-layers exist in all products.

But keep in separate stack VRT files through ariaTSsetup

Originally, latter was getting overwritten

source: https://circleci.com/docs/executor-intro/#using-machine

sssangha · 2023-01-27T07:40:30Z

Change to organization and structure of total troposphere (plus other components when specified).

Specifically, now the weather model itself is tracked such that tropo products are extracted into respective subdirectories, e.g.:

(ARIA-tools) [ssangha@trappist test_output_raider]$ ls
azimuthAngle    coherence            DEM             lookAngle          productBoundingBox  troposphereHydrostatic  troposphereWet  user_bbox.json
bPerpendicular  connectedComponents  incidenceAngle  metadatalyr_plots  stack               troposphereTotal        unwrappedPhase
(ARIA-tools) [ssangha@trappist test_output_raider]$ ls tropo*/*
troposphereHydrostatic/HRES:
20200130_20200112  20200130_20200112.aux.xml  20200130_20200112.hdr  20200130_20200112.vrt  20200130_20200118  20200130_20200118.aux.xml  20200130_20200118.hdr  20200130_20200118.vrt

troposphereTotal/HRES:
20200130_20200112  20200130_20200112.aux.xml  20200130_20200112.hdr  20200130_20200112.vrt  20200130_20200118  20200130_20200118.aux.xml  20200130_20200118.hdr  20200130_20200118.vrt

troposphereWet/HRES:
20200130_20200112  20200130_20200112.aux.xml  20200130_20200112.hdr  20200130_20200112.vrt  20200130_20200118  20200130_20200118.aux.xml  20200130_20200118.hdr  20200130_20200118.vrt

For TS analysis or stack visualization, separate VRTs reflecting this structure are deposited within the stack subdirectory as so e.g.:

(ARIA-tools) [ssangha@trappist test_output_raider]$ cd stack/
(ARIA-tools) [ssangha@trappist stack]$ ls
cohStack.vrt  connCompStack.vrt  troposphereHydrostatic  troposphereTotal  troposphereWet  unwrapStack.vrt
(ARIA-tools) [ssangha@trappist stack]$ ls tropo*/*
troposphereHydrostatic/HRESStack.vrt  troposphereTotal/HRESStack.vrt  troposphereWet/HRESStack.vrt
(ARIA-tools) [ssangha@trappist stack]$ head tropo*/*
==> troposphereHydrostatic/HRESStack.vrt <==
<VRTDataset rasterXSize="3625" rasterYSize="2359">
        <SRS>GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]</SRS>
        <GeoTransform>-119.2725,0.000833333333333238,-0.0,34.945,-0.0,-0.000833333333333362</GeoTransform>

          <VRTRasterBand dataType="Float32" band="1">
        <SimpleSource>
            <SourceFilename relativeToVRT="1">../../troposphereHydrostatic/HRES/20200130_20200112.vrt</SourceFilename>
            <SourceBand>1</SourceBand>
            <SourceProperties RasterXSize="3625" RasterYSize="2359"
                DataType="Float32"/>

==> troposphereTotal/HRESStack.vrt <==
<VRTDataset rasterXSize="3625" rasterYSize="2359">
        <SRS>GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]</SRS>
        <GeoTransform>-119.2725,0.000833333333333238,-0.0,34.945,-0.0,-0.000833333333333362</GeoTransform>

          <VRTRasterBand dataType="Float32" band="1">
        <SimpleSource>
            <SourceFilename relativeToVRT="1">../../troposphereTotal/HRES/20200130_20200112.vrt</SourceFilename>
            <SourceBand>1</SourceBand>
            <SourceProperties RasterXSize="3625" RasterYSize="2359"
                DataType="Float32"/>

==> troposphereWet/HRESStack.vrt <==
<VRTDataset rasterXSize="3625" rasterYSize="2359">
        <SRS>GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]</SRS>
        <GeoTransform>-119.2725,0.000833333333333238,-0.0,34.945,-0.0,-0.000833333333333362</GeoTransform>

          <VRTRasterBand dataType="Float32" band="1">
        <SimpleSource>
            <SourceFilename relativeToVRT="1">../../troposphereWet/HRES/20200130_20200112.vrt</SourceFilename>
            <SourceBand>1</SourceBand>
            <SourceProperties RasterXSize="3625" RasterYSize="2359"
                DataType="Float32"/>

Also @mgovorcin , note that I reverted some of the logic you introduced in generate_stack because we expect/force there to be a one-to-one match between layer keys and output subdirectories such that there shouldn't be a need to query with re tools or so. If I'm mistaken by your intention here though, then please let me know and we'll iterate. Thanks!

Add support for RAiDER tropo layers

2ceceeb

sssangha requested review from bbuzzanga and dbekaert December 21, 2022 09:48

track heights properly

65d81e7

bbuzzanga requested changes Dec 22, 2022

View reviewed changes

tools/ARIAtools/extractProduct.py Outdated Show resolved Hide resolved

tools/ARIAtools/extractProduct.py Outdated Show resolved Hide resolved

bbuzzanga and others added 4 commits December 22, 2022 14:34

replace height dim hardcoding

36f8395

single-frame bugfix

996f4e3

Add more robust layer check

a25aa33

explicitly set log level

1c4b34a

bbuzzanga reviewed Dec 23, 2022

View reviewed changes

bbuzzanga and others added 19 commits December 23, 2022 08:47

update environment file

fbf591a

logging bug fix in tropo

a2c7582

install 'rise' package through conda

915fa7e

remove libgdal dependency

34fed8c

add glibc dependency

03fd3ec

codacy patch in download

cbd13ae

glib fix attempt 1

72d96fa

remove trailing whitespace

25638be

circleci fix attempt 2

6a0cc96

circleci fix attempt 3

4da30cd

circleci fix attempt 4

c6636d2

circleci fix attempt 5

cce4f3b

circleci fix attempt 6

7115f7d

env fix for circleci

4a9cfbd

circleci fix attempt 7

9070a9b

python dependency update

e347eb6

circleci fix attempt 8

2101c52

gdal dependency update

91cfddb

gdal dependency update attempt 2

69804b9

sssangha and others added 8 commits January 20, 2023 21:31

Fix introduced typo in gacos

4296a74

Introduced fixes to make inline with ariaExtract changes

f06c0dc

Tweaks in logic to extract totalTropo by default for TS

0bdc1d7

update GACOS function to agree with MintPy's

08bf802

Change vestigial comment

6766a48

Revert nc_version specification to not assume latest version is avail…

3f4ebba

…able

Merge pull request #340 from bbuzz31/sss_raidersup_BB-gacos

3178e35

update GACOS function to agree with MintPy's

Merge pull request #337 from mgovorcin/sss_raidersup

e4c1ea3

add tropoTotal

sssangha and others added 5 commits January 23, 2023 20:01

Add pip dependency

c9cf303

Fix bug in vrtmanager

4f53d48

Do not assume tropo-layers exist in all products.

If applicable, pass both gacos and raider-derived tropo delay

01bbad5

But keep in separate stack VRT files through ariaTSsetup

Separate GACOS vrt from unwrap vrt

dee64f1

Originally, latter was getting overwritten

extract_product call cleanup

9d7e8c4

sssangha mentioned this pull request Jan 24, 2023

Five-km misregistration for some scenes ACCESS-Cloud-Based-InSAR/DockerizedTopsApp#90

Closed

sssangha added 4 commits January 24, 2023 16:11

attempt to circumvent docker

ce276f4

source: https://circleci.com/docs/executor-intro/#using-machine

config patch attempt #2

54c7e35

Updated config.yml

38801f0

revert config to docker

9bcd058

bbuzzanga mentioned this pull request Jan 26, 2023

PyAPS ndarray is not C-contiguous error with scipy.interpolate insarlab/MintPy#949

Closed

Simran S Sangha added 2 commits January 26, 2023 23:02

Seperate stack VRTs for difference models

26832c3

organize tropo components

1ec9a9d

Simran S Sangha added 4 commits January 31, 2023 00:26

do not overwrite unw phase stack reference

1d33f8b

Pass nodata values to stack

8d2860f

Ensure backwards compatibility with 1b

5747335

Fix some codacy issues

83f6707

bbuzzanga approved these changes Feb 2, 2023

View reviewed changes

bbuzzanga merged commit a858597 into dev Feb 2, 2023

bbuzzanga deleted the sss_raidersup branch February 2, 2023 00:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for RAiDER tropo layers#335

Add support for RAiDER tropo layers#335
bbuzzanga merged 96 commits intodevfrom
sss_raidersup

sssangha commented Dec 21, 2022

Uh oh!

sssangha commented Dec 22, 2022

Uh oh!

Uh oh!

Uh oh!

bbuzzanga left a comment

Uh oh!

sssangha commented Jan 24, 2023

Uh oh!

sssangha commented Jan 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

sssangha commented Dec 21, 2022

Uh oh!

sssangha commented Dec 22, 2022

Uh oh!

Uh oh!

Uh oh!

bbuzzanga left a comment

Choose a reason for hiding this comment

Uh oh!

sssangha commented Jan 24, 2023

Uh oh!

sssangha commented Jan 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants