Sentinel2 client module to ingest Sentinel-2 imagery from Theia and AWS providers#1326
Conversation
|
Travis is failing the tests for the jp2 files. perhaps we need to add we also cache large files in travis so we don't need to reach out and download it every run, for example with landsat8: https://github.com/locationtech/geowave/blob/master/.travis.yml#L30 |
|
ahh - sorry, I didn't dig in enough to realize that problem it does sound like it could be troublesome, I'll try to find some time to investigate more as well. Unfortunately in the end, we can't include tests that can't pass travis-ci so we'll have to think of some alternative. |
|
Thanks a lot! I don't know how to resolve this (my apologies for the inconvenience) without copy my own gdal_ECW_JP2ECW file (dll or so). I don't have found any public repository from where to download them. This is my problematic line :-( |
0faeec9 to
287ca42
Compare
|
After several attempts, I am not be able to test if ECWJP2 plugin is properly setup in order to use or to notify to user that AWS provider is not available. It works in Windows, but I get a not-catchable native error in Travis that stops the checks. Update: |
|
I will try to take a closer look when I have a chance, but in the meantime don't hesitate to keep giving it a shot. I looked around and so there may be some instability with the ECW decoding through java bindings such as the issue you highlighted in your update. Is this a public repo where it could be used for travis CI? I carefully read the EULA, and we should be ok to use it in travis CI's testing within the context of an open source project packaging the GDAL ECW decoder purely for testing a community contribution (not for profit). |
|
Thanks @rfecher !, I am reviewing this module from SNAP - ESA's SentiNel Application Platform (https://github.com/senbox-org/s2tbx/tree/master/lib-gdal). It provides GDAL 2.1.3 for Windows including native JP2ECW plugin, and GDAL 2.2.0 for Linux. I think this module is used by ESA Tools to read JP2 Files. UPDATE: |
95fdbb6 to
78d28fc
Compare
|
the module you linked looks promising, but one caveat that we will have to be careful with, we can't add it as a direct dependency (I think we can use it with maven "test" scope though which may work out ok) |
3c362d8 to
58b249c
Compare
|
Hi @rfecher, I have fixed the sentinel2 module in Travis (Linux with GDAL 1.9.2). I found several options, but I think I have chosen the least intrusive one in order to modify the global behavior as minimum as possible. I was testing:
I have chosen the last one. |
|
Where did you download the test JP2 file thats included in this PR? Also, to pass IP scrutiny, we need to make sure any data included in the baseline comes with an appropriately open license. If we're not sure about the license, it ends up being better to have the software download it as part of the test rather than include it in the baseline (for this we don't have to worry as much about the licensing because we are not distributing the data). |
|
I downloaded it from AWS (I think is the "T30TXN-2017-10-16" scene), I was searching a very little image. But I could confirm it if you want. |
| public static final String THEIA_AUTHENTICATION_FILE = "auth_theia.txt"; | ||
|
|
||
| // URL of raster to validate JP2 support in GDAL. | ||
| private final static String JP2ECW_RASTER_TEST_URL = "http://sentinel-s2-l1c.s3.amazonaws.com/tiles/30/T/WN/2017/10/16/0/preview/B01.jp2"; |
There was a problem hiding this comment.
it looks like the tests are already setup to download the file, can we just remove the file from the baseline and have it automatically downloaded? It would be easier for IP review, and seems to already be handled here.
There was a problem hiding this comment.
Thanks a lot for your advices and for reviewing. I have added a new JP2 in data folder to check when testing the JP2ECW plugin. I have verified that no every JP2 files from sentinel2 get to crash the GDAL 1.9.2 redist for Linux. This file is little and breaks GDAL. My apologies I don't remember whether I downloaded it from AWS or PEPS (LEVEL1C mirror at France). Is there any problem?
2781e91 to
5b10b31
Compare
It will be replaced by a new more generic Sentinel2 client module
5b10b31 to
76e09e5
Compare
|
Thanks @rfecher for reviewing and merging! |
|
of course, sorry for the delays |
|
You are welcome! |
This pull replaces the current Theia client module with a more generic Sentinel2 client module. It defines the concept of Sentinel2 providers in order to easily implement one for the repository you want to ingest.
Now, the module provides two Sentinel2 providers:
All commans take care of new
--providerargument in order to select what repository to ingest.Issues:
AWS provides the band raster files in JP2 format, the module uses JP2ECWReader component to load them. It requires the ECW plugin for GDAL available in the GDAL_DRIVER_PATH environment variable. It would be desirable to remove this dependency using by example the JP2K Plugin of GeoTools, but it doesn't work for the AWS imagery tested.