Skip to content

PySAR v1.2#4

Merged
hfattahi merged 153 commits intoinsarlab:masterfrom
yunjunz:master
Jun 3, 2017
Merged

PySAR v1.2#4
hfattahi merged 153 commits intoinsarlab:masterfrom
yunjunz:master

Conversation

@yunjunz
Copy link
Member

@yunjunz yunjunz commented May 23, 2017

  1. consistent naming style for all data files
  2. more generalized load-data
  3. introduce coherence-based network modification
  4. referencing data in space and reading modified network while inverting network to timeseries
  5. introduce auto referencing in time
  6. introduce auto turbulence detection using timeseries residual standard deviation
  7. full template list for pysarApp.py - pysarApp_template.txt
  8. new progress bar
  9. new time-series viewer python script
  10. updated scripts for time-series web viewer
  11. bug fixed for UNAVCO InSAR Archive output
  12. centralized code for parallel computing check
  13. bugs fixed
  14. improved plotting scripts

yunjunz and others added 30 commits March 12, 2017 19:22
1. move code of drawing lat/lon label from plot_matrix() to independent
function draw_lalo_label() inside class Basemap2

2. minor adjustment to Basemap2.drawscale()
rename —footprint option to —tight option, and setup output filename to
file*_tight.ext, instead of a prefix of ‘subset_’
1. bug fixed converting ref_y/x to ref_lat/lon while geocoding
2. bug fixed while convert radar2geo coord with one point, instead of multiply
write full attribute into rsc file before calling geocode.pl from ROIPAC for geocoding, so less warning for missing attributes
update Y/XMIN/MAX in attributes for output file
add —coastline option to plot coastline, along with —resolution option
to specify the boundary database resolution used in coastline plotting.

Add map_group in input argument parsing, to group —coastline,
—resolution, —scalebar and —noscalebar options.
add qr code file for pysar github page and um insarmaps
Short URL: q-r.to/pysar
Target URL: yunjunz.github.io/PySAR/

Short URL: q-r.to/insarmaps
Target URL: insarmaps.miami.edu
1. add auto figure size in view.py and save_kml.py; by default, it
calculate a figure size proportional to data shape, and constrain its
shorter dimension to 8 inches.

2. update auto_row_col_num() in view.py with better handling of
calculation residual for multiple display
…rmaps; add functionality in json_mbtiles2insarmaps to upload mbtiles and json files
… scripts; began work on add_plot_attribute function; remove unnecessary check for table name in add_attribute
stackTom and others added 26 commits May 10, 2017 00:26
__init__.py: add figsize_single_max variable for max size

_datetime.py:
1. move progress_bar class from _pysar_utilities.py to _datetime.py;
and adjust corresponding scripts
2. return start and end date in auto_adjust_xaxis_date()

_network.py: add ymin/max argument in auto_adjust_yaxis()

_pysar_utilities.py
1. add function timeseries_std() and timeseries_coherence() to
calculate standard deviation and phase coherence in space for each
epoch of timeseries
2. add function get_residual_std() to calculate deramped phase residual
standard deviation and write it to text file
3. check is inFile exists in update_file()

dem_error.py: output phase inversion residual to a new timeseries file

pysarApp.py:
1. constrain input subset box in radar coord within the data coverage
2. add new pysar options in the template
pysar.residualStd.maskFile  - mask file used for residual std
calculation
pysar.residualStd.ramp - ramp type
pysar.residualStd.minStd - threshold for turbulence detection

pysar.velocity.excludeDate - customized text file for excluded dates in
linear velocity estimation

3. plot all command line even they are not ran

4. use tropospheric delay file if exists instead of call pyaps again
5. calculate linear velocity from tropospheric delay file

6. add new step: calculate timeseries residual standard deviation

7. add new step: automatic referencing in time using min residual std

reference_epoch.py
1. use argparse to replace get opt for input argument parsing
2. add —ref-date auto option to find reference date automatically based
on residual standard deviation

new script - timeseries_std.py to calculate standard deviation of
deramped phase residual for each epoch of timeseries

 timeseries2velocity.py
1. rename -m/M option to —start-date and —end-date option, add
—template option to use pysar.velocity.excludeDate option
2. remove obsolete function yyyymmdd2years(), call pysar._datetime
module instead
3. add function read_template2inps() to read template options into
Namespace object
4. add get_exclude_date() to read all date related options
5. add get_velocity_filename() to get output velocity filename based on
input initial name, and with/without exclude date
tropcor_pyaps.py: add support to read bl_list.txt file to get list of
date for weather re-analysis data downloading

use —exclude by default in select_network.py and view.py
Introduce a default, full template file - pysarApp_template.txt - to
pysarApp.py. With parameter setting for all steps run within
pysarApp.py.

For the first time running, pysarApp.py will generate this txt file
with all default settings unless it already exists.

If custom template, like KujuAlosAT422F650.template is input, script
will read the value from it and update it to the default template file.

use pysarApp.py -g to generate/check default template and exit, similar
to run -g for Doris.

_pysar_utilities.py
1. add function update_template_file() to write template new value into
an organized existing template file
2. add function get_spatial_average() to get spatial average list of
input file. And change output file extension from .list to .txt

_readfile.py: modify read_template() to not read option with empty value

load_data.py: adjust script for full template list and the auto value

modify_network.py:
1. adjust script for full template list and auto value
2. add —reset option to select all group within input file -
drop_ifgram=no
3. support maskAoi.yx/lalo to change area of pixels for average
coherence calculation

pysarApp.py
add full template list with -g option to stop after template reading

reference_epoch.py: do not write date to reference_date.txt file and
support full template list with auto value

seed_data.py: support full template list
_pysar_utilities.py: add multiple input files - list - support for
update_file(), and adjust corresponding function usage in
_pysar_utilities.py and pysarApp.py

diff.py: do not exit if all groups of file1 exists within file2 for
timeseries file type

multilook.py:
1. fix bug while multi looking mapping transformation file
2. add print_message option for multilook_attribute(), so that no
message print while multi looking multi_group files
_network.py
1. add coherence_matrix() to calculate 2D coherence matrix given
date12_list and coherence_list
2. add threshold_coherence_based_mst() to return a minimum spanning
tree of network of interferograms with inverse of coherence as weight,
given date12_list and coherence_list
3. add plot_coherence_matrix() and plot_coherence_history() for plotting
4. modify plot_network() to find color jump based on input date12_list
to keep and drop, so that all dropped interferograms shown as red dash
line and all kept interferograms shown as solid blue line; change min
coherence to display from 0.4 to 0.2

modify_network.py: add MST constrain if using coherence-based network
modification, so that network will not become non-fully connected,
because of coherence thresholding.

plot_network.py: plot two new figures - CoherenceMatrix.pdf and
CoherenceHistory.pdf

pysarApp.py: add comment about coherence-based network with MST
constrain.

timeseries_std.py: adjust bar width using common mode of sampling
length in time
1. file permission adjustment for _gmt.py and insarmaps_query.py

2. do not plot CoherenceMatrix and CoherenceHistory if no coherence info input

3. tropcor_pyaps.py: get ride of --timeseries option name for timeseries input
…or correction performance

1. rename timeseries_std.py to timeseries_rms.py and calculate timeseries_ECMWF_demErrInvResid_quadratic.h5's RMS instead of STD. Test shows that both of them are the same for deramped timeseries, but RMS are more sentative than STD if timeseries is not deramped, and RMS is more clear in definition.

Code is updated in pysarApp.py, _pysar_utilities.py, timeseries_rms.py

2. bug fixed for seed_data.py while using pysar.reference.lalo option with template input

3. use default maskTempCoh.h5 mask in view.py for velocity, timeseries, rmse and interferograms.
add pysarApp.py --subset-data option to create a subset dataset of all required files easily, so that the newly created SUBSET_X*Y* folder could be shared and re-processed.

add ROIPAC file support for add_attribute.py, and if new attribute's value is None, drop this item from attribute dictionary

reverse back the multilooking for geomap*.trans file, do not divide its value with number of multilooks
add list_ifgram2date12() in _datetime.py to convert ifgram list into date12_list, and adjust corresponding script, mainly the one use this info to shown suffix in progress bar

consider difference reference date for timeseries file input of diff.py

check corrupted hdf5 file in load_data.py

more robust tight croping in pysarApp.py

add --exclude option in tsviewer.py to show excluded date as gray dot

bug fixed for view.py when input exclude date for timeseries file
fix bug in load_data.py introduced by the full template txt file in pysarApp.py

do not tight subset DEM in geo coord if it's not exists in pysarApp.py
1. add pysar.topoError.stepFuncDate option to specify the date of step jump, i.e. date of earthquake or volcanic eruption, to the temporal deformation model; more robust for complicated deformation dataset

2. add pysar.topoError.excludeDate option to exclude date(s) for error estimation, which usually are streamely strong ionosphere

3. add --template option to dem_error.py for input argument, and adjust corresponding code in pysarApp.py
_network.py:
1. fix bug with MST date12 list exchanged master/slave date
2. add coh_thres to specify color jump/cut in network plot with coherence
3. bug fixed with coloring in plot_network()
4. add disp_title option

diff.py: support differencing of two timeseries files with different reference date and reference pixel

plot_network.py: enable --figsize option

save_kml.py: do not read 'X/Y_UNIT', make it more robust
test for falk
1. add new script prep_gamma.py to prepare a .rsc file for Gamma product. It contains the following functions:

a. def attribute_gamma2roipac() to convert exsiting GAMMA attribute to ROIPAC attribute

b. def get_perp_baseline() to calculate perpendicular baseline based on master/slave par file and interferogram off file, using Gamma command base_orbit and base_perp

c. def get_lalo_ref() to calculate lat/lon of interferogram's four corners based on master par file, using Gamma command SLC_corners

d. def extract_attribute() to extract the whole attribute based on interferogram file name

2. add new template optioin pysar.insarProcessor into full template list, for pysarApp.py and load_data.py

3. _readfile.py:
add byteorder option into read_real_float32() to support Gamma product

4. load_data.py: adjust to be able to load Gamma unw and cor files into PySAR
@hfattahi hfattahi merged commit ea7fc46 into insarlab:master Jun 3, 2017
hfattahi pushed a commit that referenced this pull request May 10, 2018
add script for tropospheric delay correction from GACOS at U New Castle
yunjunz pushed a commit that referenced this pull request Sep 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants