Skip to content

Dockerfile: rewrite using Micromamba Image#685

Merged
forrestfwilliams merged 11 commits intoinsarlab:mainfrom
forrestfwilliams:docker
Nov 4, 2021
Merged

Dockerfile: rewrite using Micromamba Image#685
forrestfwilliams merged 11 commits intoinsarlab:mainfrom
forrestfwilliams:docker

Conversation

@forrestfwilliams
Copy link
Collaborator

@forrestfwilliams forrestfwilliams commented Nov 1, 2021

References issue #123
A recent build can be found at https://hub.docker.com/repository/docker/forrestwilliams/mintpy

  • Rewrites the MintPy dockerfile using the mambaorg/micromamba image
  • Shorter build times and slightly smaller image!
  • Forces Python=3.6 for faster image build
  • Sets the user as micromamba, not root
  • Optionally allows the image to start with a Jupyter Lab instance

Reminders

  • Fix #xxxx
  • Pass Codacy code review (green)
  • Pass Circle CI test (green)
  • Make sure that your code follows our style. Use the other functions/files as a basis.
  • If modifying functionality, describe changes to function behavior and arguments in a comment below the function declaration.
  • If adding new functionality, add a detailed description to the documentation and/or an example.

- Rewrites the MintPy dockerfile using the mambaorg/micromamba image
- Forces Python=3.6 for faster image build
- Sets the user as micromamba, not root
- Optionally allows the image to start with a Jupyter Lab instance
- tag micromamba version
- remove apt-get lists
Copy link
Collaborator

@jhkennedy jhkennedy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks pretty good! And should be a big build improvement

Co-authored-by: Joseph H Kennedy <me@jhkennedy.org>
@yunjunz yunjunz self-requested a review November 2, 2021 19:38
@forrestfwilliams
Copy link
Collaborator Author

Up to date version of the image can be found at forrestwilliams/mintpy:1.3.1

@jhkennedy
Copy link
Collaborator

@forrestfwilliams this is great!

I was able to build this image and run the tests inside it just fine, like:


docker run -it forrestwilliams/mintpy:1.3.1
cd tools/MintPy/tests
micromamba install -c conda-forge wget  # needed for test
python test_smallbaselineApp.py

So in terms of the dockerfile, I'd call this g2g! (unless it's worth adding wget to the apt-get install as well)

@yunjunz, I assume you'd like to do the final approval?

@jhkennedy
Copy link
Collaborator

jhkennedy commented Nov 3, 2021

Other thoughts for you @forrestfwilliams and @yunjunz:

Edit: Moved 2nd point to #123.

@yunjunz
Copy link
Member

yunjunz commented Nov 3, 2021

Good point @jhkennedy. Let's update the docker image's link in the two *.md files to forrestwilliams/mintpy, could you please do that @forrestfwilliams?

@yunjunz
Copy link
Member

yunjunz commented Nov 3, 2021

I tried to test the docker image following the same command as @jhkennedy wrote above. It passed the first two datasets for (ISCE and ARIA) and failed at generating KMZ file for the GAMMA dataset, as shown in the error message below. Manually running the save_kmz.py command works fine actually. Any idea why?

******************** step - google_earth ********************
creating Google Earth KMZ file for geocoded velocity file: ...

save_kmz.py /home/micromamba/data/test/WellsEnvD2T399/mintpy/geo/geo_velocity.h5 -o /home/micromamba/data/test/WellsEnvD2T399/mintpy/geo/geo_velocity.kmz
data   coverage in y/x: (0, 0, 753, 537)
subset coverage in y/x: (0, 0, 753, 537)
update LENGTH, WIDTH, Y/XMAX
update/add SUBSET_XMIN/YMIN/XMAX/YMAX: 90/12/843/549
update Y/X_FIRST
update REF_Y/X
read mask from file: geo_maskTempCoh.h5
masking out pixels with zero value in file: None
colormap: jet
plotting data ...
figure size : [16.83, 12.00]
show reference point
writing /home/micromamba/data/test/WellsEnvD2T399/mintpy/geo/geo_velocity.png with dpi=600
Killed
Traceback (most recent call last):
  File "test_smallbaselineApp.py", line 209, in <module>
    main(sys.argv[1:])
  File "test_smallbaselineApp.py", line 190, in main
    test_isce=inps.test_isce)
  File "test_smallbaselineApp.py", line 153, in test_dataset
    raise RuntimeError('Test failed for example dataset {}'.format(dset_name))
RuntimeError: Test failed for example dataset WellsEnvD2T399

@jhkennedy
Copy link
Collaborator

@yunjunz the Killed in the line makes me suspect a memory issue (hitting a docker imposed limit?) -- I was able to run all of them through on my linux box.

@yunjunz
Copy link
Member

yunjunz commented Nov 3, 2021

@jhkennedy you are right. After increasing the resource limit for Docker from the default values, all tests passed for me as well.

@forrestfwilliams
Copy link
Collaborator Author

Morning! I'll go ahead and make these changes:

  • add wget
  • update installation.md and docker.md

@yunjunz, for PYAPS_HOME and MINTPY_HOME are you saying the user doesn't need to access them once MintPy is built? If so, I might go ahead and hard-code them.

@yunjunz
Copy link
Member

yunjunz commented Nov 3, 2021

@forrestfwilliams yes, users do not need to access the two variables.

- hard code MINTPY_HOME and PYAPS_HOME
- install wget
- update installation.md and docker.md
@forrestfwilliams
Copy link
Collaborator Author

I've updated the dockerfile/docs. I also ran the tests in a container built from the new image and all tests passed! I'm re-uploading the image to docker and I'll let you know when it's done.

@forrestfwilliams
Copy link
Collaborator Author

New build is available at forrestwilliams/mintpy:1.3.1 !

@forrestfwilliams
Copy link
Collaborator Author

Oops sorry I missed these

@yunjunz
Copy link
Member

yunjunz commented Nov 3, 2021

No worries at all.

I am not a regular user of docker. @forrestfwilliams could you confirm if my modification of using ARG MINTPY_HOME and PYAPS_HOME in the DockerFile is reasonable?

Copy link
Member

@yunjunz yunjunz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks all good to me.

Thank you @forrestfwilliams for updating this and @jhkennedy for the constructive review!

@forrestfwilliams
Copy link
Collaborator Author

@yunjunz, I'd like to get @jhkennedy's opinion on this but my understanding is that ARGs should be reserved for variables that we want to be modifiable by someone building the image (see here). Your modifications improve readability, but I'm not sure if we want the MINTPY_HOME and PYAPS_HOME locations to be modifiable. Thoughts?

Copy link
Collaborator

@jhkennedy jhkennedy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@forrestfwilliams @yunjunz ARG is something that could be overridden at build time, but is also a convenient and often used way to have constants/variables in the docker file, so I don't see any issues using it for that.

I only see one small problem is someone did specify different values at build time (the directory to clone into might not exist), but added a suggestion that would solve that.

yunjunz and others added 2 commits November 3, 2021 16:10
Co-authored-by: Joseph H Kennedy <me@jhkennedy.org>
Co-authored-by: Joseph H Kennedy <me@jhkennedy.org>
@yunjunz
Copy link
Member

yunjunz commented Nov 3, 2021

The PR is ready to merge in my opinion. @forrestfwilliams please feel free to merge it if you think it's ready.

@forrestfwilliams
Copy link
Collaborator Author

Thanks @yunjunz ! I think we're ready to merge as well but I'm not sure I have the permissions to do so.

@yunjunz
Copy link
Member

yunjunz commented Nov 4, 2021

I just sent you an invite. Could you try again after acceptance?

@forrestfwilliams forrestfwilliams merged commit 8472709 into insarlab:main Nov 4, 2021
@forrestfwilliams
Copy link
Collaborator Author

Merged! Thanks for inviting me to join insarlab!

@forrestfwilliams forrestfwilliams deleted the docker branch November 4, 2021 01:29
@yunjunz
Copy link
Member

yunjunz commented Nov 4, 2021

Great to have you here!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants