limit NUM_THREADS=1 for numpy/scipy.linalg to save CPU usage#620
Merged
yunjunz merged 2 commits intoinsarlab:mainfrom Jul 29, 2021
Merged
limit NUM_THREADS=1 for numpy/scipy.linalg to save CPU usage#620yunjunz merged 2 commits intoinsarlab:mainfrom
yunjunz merged 2 commits intoinsarlab:mainfrom
Conversation
+ utils.readfile.read_gdal_vrt(): support reading metadata from file without SRS info by setting defalt X/Y_UNIT in degrees + defaults.smallbaselineApp.cfg: add comments on SNAPHU as the only unwrapper that provides connected components info. + dask: use explicit cluster name instead of non-local to avoid confusion in the smallbaselineApp.cfg and dask.md files. + view: printout multilook-num even if not calculated from auto_*(); plot water/shadowMask in gray by default
Testing shows that numpy/scipy with OMP_NUM_THREADS > 1 does not help much on the computing time but uses significantly more CPU. The true fast way is "Dask with multiple workers + OMP_NUM_THREADS = 1". Thus, I am setting all the relevant NUM_THREADS env variables to 1, in ifgram_inversion.py and dem_error.py, before running the big matrix inversion and roll back to the original values afterwards. + objects/cluster: - add set_num_threads() to save original num of threads info and set all relevant env var to the given value - add roll_back_num_threads() to roll back to the original setting + ifgram_inversion/dem_error: call the above two functions in the begining and the end of the process, to ignore the *_NUM_THREADS setting during the numpy/scipy matrix computation.
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of proposed changes
Testing shows that numpy/scipy with OMP_NUM_THREADS > 1 does not help much on the computing time but uses significantly more CPU. The true fast way is Dask with multiple workers + OMP_NUM_THREADS = 1. Thus, I am setting all the relevant NUM_THREADS env variables to 1, in ifgram_inversion.py and dem_error.py, before running the big matrix inversion and roll back to the original values afterward.
Here is the test on
ifgram_inversion.py inputs/ifgramStack.h5 -t smallbaselineApp.cfgon my laptop:@falkamelung, @Ovec8hkin could you help to confirm: if this PR does give a slight speedup and significant CPU save on HPC?
Update: I am merging this now, but it would still be useful if you could confirm on HPC.
Reminders