Skip to content

Create Coseismic_Requirement_Validation-Approach2.ipynb#26

Merged
dbekaert merged 8 commits intonisar-solid:mainfrom
EJFielding:coseismic_revised
May 18, 2022
Merged

Create Coseismic_Requirement_Validation-Approach2.ipynb#26
dbekaert merged 8 commits intonisar-solid:mainfrom
EJFielding:coseismic_revised

Conversation

@EJFielding
Copy link
Collaborator

New version of Coseismic notebook with approach 1 and 2 code added, along with other changes.

New version of Coseismic notebook with approach 1 and 2 code added, along with other changes.
@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@dbekaert
Copy link
Contributor

dbekaert commented May 2, 2022

@EJFielding Suggest not to change names of the notebook versions as then we loose the ability to track versions using github or compare with the notebook tool. If a name change is needed please use git mv capabilities.

@review-notebook-app
Copy link

review-notebook-app bot commented May 2, 2022

View / edit / reply to this conversation on ReviewNB

dbekaert commented on 2022-05-02T03:12:14Z
----------------------------------------------------------------

Noticed this traceback error. Does not look like a failure, but any thoughts as to why this occurred?


EJFielding commented on 2022-05-02T03:15:55Z
----------------------------------------------------------------

Yes, because I used the staged data, the ARIA-tools files do not exist. The MintPy script tries to look for them to check if they changed and did not find them. I am not sure how to fix that. Maybe @yunjunz could suggest a way to avoid this?

yunjunz commented on 2022-05-02T03:32:14Z
----------------------------------------------------------------

I just tried to copy over the printout custom template and tested on the San Francisco Bay example dataset, and could not re-produce the error. I can't think of causes on top of my head.

EJFielding commented on 2022-05-02T04:23:16Z
----------------------------------------------------------------

I think it is because we are using mintpy.load.azAngleFile: ../azimuthAngle/\*.vrt in the MintPy configuration file, but the SF Bay example uses a specific file name. Since we have been changing the intervals downloaded, it is much easier to use the *.vrt name instead of having to change the file name to whatever the new name is.

EJFielding commented on 2022-05-02T04:24:46Z
----------------------------------------------------------------

Does everybody else always modify their MintPy template file every time they run ariaTSsetup.py to match whatever new filename is created?

mgovorcin commented on 2022-05-02T16:26:08Z
----------------------------------------------------------------

I notice this trackback error before, and believe it is associated with staged ifgramStack.h5 and "mintpy.network.excludeIfgIndex" flag. Initially, the stack is staged with excluded 6 ifgs, so when we change it to use all pairs, I guess MintPy cannot find them and raise the error. Should we leave it like this or staged the data again with all included pairs?

EJFielding commented on 2022-05-02T16:30:03Z
----------------------------------------------------------------

Hi Marin, I believe the ifgramStack.h5 file contains all the interferograms, but 6 were marked to be excluded. This is easy for MintPy to update as it is only a flag on those layers. The error message is that MintPy is trying to check the ARIA-tools stack products to see if it needs to update the interferograms in the loading step.

mgovorcin commented on 2022-05-02T16:38:41Z
----------------------------------------------------------------

I just reproduced the error. Believe it is associated with the update of smallbaseline config file as it is different than the staged one (having also different timestamp). So, guess MintPy is trying to reload the data but it cannot find ARIA stack folder and raise the error.

yunjunz commented on 2022-05-02T17:14:32Z
----------------------------------------------------------------

@mgovorcin could you show here how to re-produce the error? preferably on laptop for me to easily test.

EJFielding commented on 2022-05-02T17:29:21Z
----------------------------------------------------------------

@yunjunz I got the same error on my Mac laptop. It happens when we download the staged data that does not include the ARIA-tools products and run the script a second time in the same directory.

mgovorcin commented on 2022-05-02T17:41:49Z
----------------------------------------------------------------

@yunjun try to run the notebook with these changes:

  1. in Prep A.: set sites['RidgecrestA64']['ifgExcludeList'] = 'no'
  2. run cells Prep A, Prep B, 2.1. 'Set Up MintPy Configuration file', and then 2.2 'Load data into Mintpy'

EJFielding commented on 2022-05-02T18:04:23Z
----------------------------------------------------------------

@yunjunz Looking at it again, it can also be due to MintPy parsing. It thinks that it needs to change the configuration because of the backslash that is added as an escape for the asterisk.

mintpy.load.azAngleFile: ../azimuthAngle/\*.vrt --> ../azimuthAngle/*.vrt

yunjunz commented on 2022-05-02T20:32:30Z
----------------------------------------------------------------

Thank you @EJFielding and @mgovorin, I was able to re-produce the error. The issue is in the template updating code: the staged default template option "mintpy.network.excludeIfgIndex = [123,124,125]" has "[]" included, these are special characters for re expression syntax, causing issues during the searching and updating of template files. It's now fixed in https://github.com/insarlab/MintPy/pull/768. Thank you for founding this issue!

EJFielding commented on 2022-05-02T23:03:10Z
----------------------------------------------------------------

Thanks, @yunjunz! I pulled the new version of MintPy to my Mac and run the notebook again, and it is working well without the errors. It is re-running the network inversion without the excluded scenes.

@review-notebook-app
Copy link

review-notebook-app bot commented May 2, 2022

View / edit / reply to this conversation on ReviewNB

dbekaert commented on 2022-05-02T03:12:14Z
----------------------------------------------------------------

Same in this cell output.


EJFielding commented on 2022-05-02T03:16:25Z
----------------------------------------------------------------

Yes, the problem affects all the smallbaselineApp.py calls.

yunjunz commented on 2022-05-02T03:33:12Z
----------------------------------------------------------------

This means all the smallbaselineApp calls in this notebook are not actually run.

EJFielding commented on 2022-05-02T23:03:30Z
----------------------------------------------------------------

Now this is fixed.

@review-notebook-app
Copy link

review-notebook-app bot commented May 2, 2022

View / edit / reply to this conversation on ReviewNB

dbekaert commented on 2022-05-02T03:12:15Z
----------------------------------------------------------------

Seems to be impacted here by the coseismic. Was not the case before.


EJFielding commented on 2022-05-02T03:18:06Z
----------------------------------------------------------------

This depends on whether the velocity fit was done with the coseismic step or not. If the coseismic step is not estimated, then the linear velocity absorbs the coseismic.

EJFielding commented on 2022-05-02T03:22:59Z
----------------------------------------------------------------

The staged velocity.h5 file was the fit with the linear velocity plus coseismic step function at time of Ridgecrest earthquake. I it is better to use the timeseries2velocity.py call for this first fit so it runs even if the other MintPy steps are skipped., but that still shows the large coseismic signal in the velocity, so I added an explanation. I will update PR when it completes the run.

@EJFielding
Copy link
Collaborator Author

@dbekaert OK, I removed the old version and moved the new version to the same name.

@dbekaert
Copy link
Contributor

dbekaert commented May 2, 2022

hi @EJFielding Were you running this from already existing run? I noticed some error messages in the mintpy output which seems not to have stopped from continue the running. The version is slightly different from what was used on OSL.

One plot i noticed for secular or mean velocity showed now some coseismic signal.

Stats look good!

@dbekaert
Copy link
Contributor

dbekaert commented May 2, 2022

@dbekaert OK, I removed the old version and moved the new version to the same name.

Worked well, thanks for doing that!

Copy link
Collaborator Author

Yes, because I used the staged data, the ARIA-tools files do not exist. The MintPy script tries to look for them to check if they changed and did not find them. I am not sure how to fix that. Maybe @yunjunz could suggest a way to avoid this?


View entire conversation on ReviewNB

Copy link
Collaborator Author

Yes, the problem affects all the smallbaselineApp.py calls.


View entire conversation on ReviewNB

Copy link
Collaborator Author

This depends on whether the velocity fit was done with the coseismic step or not. If the coseismic step is not estimated, then the linear velocity absorbs the coseismic.


View entire conversation on ReviewNB

Copy link
Collaborator Author

The staged velocity.h5 file was the fit with the linear velocity plus coseismic step function at time of Ridgecrest earthquake.


View entire conversation on ReviewNB

Copy link
Contributor

yunjunz commented May 2, 2022

I just tried to copy over the printout custom template and tested on the San Francisco Bay example dataset, and could not re-produce the error. I can't think of causes on top of my head.


View entire conversation on ReviewNB

Copy link
Contributor

yunjunz commented May 2, 2022

This means all the smallbaselineApp calls in this notebook are not actually run.


View entire conversation on ReviewNB

@dbekaert
Copy link
Contributor

dbekaert commented May 2, 2022

@mgovorcin did you experience this on OSL? Is something missing from the clean-up?

Copy link
Collaborator Author

I think it is because we are using mintpy.load.azAngleFile: ../azimuthAngle/\*.vrt in the MintPy configuration file, but the SF Bay example uses a specific file name. Since we have been changing the intervals downloaded, it is much easier to use the *.vrt name instead of having to change the file name to whatever the new name is.


View entire conversation on ReviewNB

Copy link
Collaborator Author

Does everybody else always modify their MintPy template file every time they run ariaTSsetup.py to match whatever new filename is created?


View entire conversation on ReviewNB

Copy link
Contributor

I notice this trackback error before, and believe it is associated with staged ifgramStack.h5 and "mintpy.network.excludeIfgIndex" flag. Initially, the stack is staged with excluded 6 ifgs, so when we change it to use all pairs, I guess MintPy cannot find them and raise the error. Should we leave it like this or staged the data again with all included pairs?


View entire conversation on ReviewNB

Copy link
Collaborator Author

Hi Marin, I believe the ifgramStack.h5 file contains all the interferograms, but 6 were marked to be excluded. This is easy for MintPy to update as it is only a flag on those layers. The error message is that MintPy is trying to check the ARIA-tools stack products to see if it needs to update the interferograms in the loading step.


View entire conversation on ReviewNB

Copy link
Contributor

I just reproduced the error. Believe it is associated with the update of smallbaseline config file as it is different than the staged one (having also different timestamp). So, guess MintPy is trying to reload the data but it cannot find ARIA stack folder and raise the error.


View entire conversation on ReviewNB

Copy link
Contributor

yunjunz commented May 2, 2022

@mgovorcin could you show here how to re-produce the error? preferably on laptop for me to easily test.


View entire conversation on ReviewNB

Copy link
Collaborator Author

@yunjunz I got the same error on my Mac laptop. It happens when we download the staged data that does not include the ARIA-tools products and run the script a second time in the same directory.


View entire conversation on ReviewNB

Copy link
Contributor

@yunjun try to run the notebook with these changes:

  1. in Prep A.: set sites['RidgecrestA64']['ifgExcludeList'] = 'no'
  2. run cells Prep A, Prep B, 2.1. 'Set Up MintPy Configuration file', and then 2.2 'Load data into Mintpy'

View entire conversation on ReviewNB

Copy link
Collaborator Author

@yunjunz Looking at it again, it can also be due to MintPy parsing. It thinks that it needs to change the configuration because of the backslash that is added as an escape for the asterisk.

mintpy.load.azAngleFile: ../azimuthAngle/\*.vrt --> ../azimuthAngle/*.vrt

View entire conversation on ReviewNB

Copy link
Contributor

yunjunz commented May 2, 2022

Thank you @EJFielding and @mgovorcin, I was able to re-produce the error. The issue is in the template updating code: the staged default template option "mintpy.network.excludeIfgIndex = [123,124,125]" has "[]" included, these are special characters for re expression syntax, causing issues during the searching and updating of template files. It's now fixed in https://github.com/insarlab/MintPy/pull/768. Thank you for finding this issue!


View entire conversation on ReviewNB

Copy link
Collaborator Author

Thanks, @yunjunz! I pulled the new version of MintPy to my Mac and run the notebook again, and it is working well without the errors. It is re-running the network inversion without the excluded scenes.


View entire conversation on ReviewNB

1 similar comment
Copy link
Collaborator Author

Thanks, @yunjunz! I pulled the new version of MintPy to my Mac and run the notebook again, and it is working well without the errors. It is re-running the network inversion without the excluded scenes.


View entire conversation on ReviewNB

Copy link
Collaborator Author

Now this is fixed.


View entire conversation on ReviewNB

Added alert boxes to mark results. Other cleanup to reduce number of cells and improve descriptions.
@review-notebook-app
Copy link

View / edit / reply to this conversation on ReviewNB

EJFielding commented on 2022-05-05T21:36:06Z
----------------------------------------------------------------

All the GNSS fits change by a tiny amount because a few new dates are included in the time-series.


Fixed bug where not all Method 2 uses of `dist` were changed to `M2dist`.
Copy link
Contributor

@dbekaert dbekaert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good with all the fixes made.

@dbekaert dbekaert merged commit fd7f1bd into nisar-solid:main May 18, 2022
@EJFielding EJFielding deleted the coseismic_revised branch May 18, 2022 15:04
@EJFielding EJFielding restored the coseismic_revised branch May 19, 2022 21:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants