Skip to content

Sentinel2 client module to ingest Sentinel-2 imagery from Theia and AWS providers#1326

Merged
rfecher merged 3 commits intolocationtech:masterfrom
ahuarte47:master-cli-sentinel2
May 29, 2018
Merged

Sentinel2 client module to ingest Sentinel-2 imagery from Theia and AWS providers#1326
rfecher merged 3 commits intolocationtech:masterfrom
ahuarte47:master-cli-sentinel2

Conversation

@ahuarte47
Copy link
Copy Markdown
Contributor

@ahuarte47 ahuarte47 commented Apr 24, 2018

This pull replaces the current Theia client module with a more generic Sentinel2 client module. It defines the concept of Sentinel2 providers in order to easily implement one for the repository you want to ingest.

Now, the module provides two Sentinel2 providers:

  • THEIA: It provides LEVEL2A products processed with MAJA and with this geographic coverage. This provider is a simple refactoring of the version one in the old Theia module.
  • Amazon Web Services - AWS: At the moment, this provider only reads LEVEL1C products.

All commans take care of new --provider argument in order to select what repository to ingest.

Issues:
AWS provides the band raster files in JP2 format, the module uses JP2ECWReader component to load them. It requires the ECW plugin for GDAL available in the GDAL_DRIVER_PATH environment variable. It would be desirable to remove this dependency using by example the JP2K Plugin of GeoTools, but it doesn't work for the AWS imagery tested.

@rfecher
Copy link
Copy Markdown
Contributor

rfecher commented Apr 25, 2018

Travis is failing the tests for the jp2 files.

Running mil.nga.giat.geowave.format.sentinel2.RasterIngestRunnerTest
Downloading scene 'S2A_MSIL1C_20180104T110431_N0206_R094_T30TXN_20180104T130839'
Wait please... 
Downloading File 'http://sentinel-s2-l1c.s3-website.eu-central-1.amazonaws.com/tiles/30/T/XN/2018/1/4/0/productInfo.json' (1 KB) -> ok: (1 KB)
Downloading File 'http://sentinel-s2-l1c.s3-website.eu-central-1.amazonaws.com/tiles/30/T/XN/2018/1/4/0/preview.jpg' (12 KB) -> ok: (146 KB)
Downloading File 'http://sentinel-s2-l1c.s3-website.eu-central-1.amazonaws.com/tiles/30/T/XN/2018/1/4/0/B04.jp2' (16 KB) -> ok: (113 MB)
Apr 25, 2018 8:06:56 AM it.geosolutions.imageio.gdalframework.GDALUtilities loadGDAL
INFO: GDAL Native Library loaded (version: 1.9.2)
terminate called after throwing an instance of 'std::length_error'
  what():  basic_string::_S_create
Aborted (core dumped)

perhaps we need to add GDAL_DRIVER_PATH as an environment variable to surefire, similar to what we add to failsafe here: https://github.com/locationtech/geowave/blob/master/test/pom.xml#L412-L413

we also cache large files in travis so we don't need to reach out and download it every run, for example with landsat8: https://github.com/locationtech/geowave/blob/master/.travis.yml#L30

@ahuarte47
Copy link
Copy Markdown
Contributor Author

Thanks @rfecher, but I have a problem, the module is using the native ECW plugin for GDAL (Using the gdal_ECW_JP2ECW.dll in Windows) that I think it doesn't exist in the GDAL-1.9.2 zip redist downloaded from here, I have to study about this issue.

@rfecher
Copy link
Copy Markdown
Contributor

rfecher commented Apr 25, 2018

ahh - sorry, I didn't dig in enough to realize that problem

it does sound like it could be troublesome, I'll try to find some time to investigate more as well. Unfortunately in the end, we can't include tests that can't pass travis-ci so we'll have to think of some alternative.

@ahuarte47
Copy link
Copy Markdown
Contributor Author

ahuarte47 commented Apr 25, 2018

Thanks a lot! I don't know how to resolve this (my apologies for the inconvenience) without copy my own gdal_ECW_JP2ECW file (dll or so). I don't have found any public repository from where to download them.

This is my problematic line :-(

@ahuarte47 ahuarte47 force-pushed the master-cli-sentinel2 branch 5 times, most recently from 0faeec9 to 287ca42 Compare April 27, 2018 08:38
@ahuarte47
Copy link
Copy Markdown
Contributor Author

ahuarte47 commented Apr 27, 2018

After several attempts, I am not be able to test if ECWJP2 plugin is properly setup in order to use or to notify to user that AWS provider is not available. It works in Windows, but I get a not-catchable native error in Travis that stops the checks.

Update:
Could this to be the cause?

@rfecher
Copy link
Copy Markdown
Contributor

rfecher commented Apr 30, 2018

I will try to take a closer look when I have a chance, but in the meantime don't hesitate to keep giving it a shot. I looked around and so there may be some instability with the ECW decoding through java bindings such as the issue you highlighted in your update. Is this a public repo where it could be used for travis CI? I carefully read the EULA, and we should be ok to use it in travis CI's testing within the context of an open source project packaging the GDAL ECW decoder purely for testing a community contribution (not for profit).

@ahuarte47
Copy link
Copy Markdown
Contributor Author

ahuarte47 commented Apr 30, 2018

Thanks @rfecher !, I am reviewing this module from SNAP - ESA's SentiNel Application Platform (https://github.com/senbox-org/s2tbx/tree/master/lib-gdal).

It provides GDAL 2.1.3 for Windows including native JP2ECW plugin, and GDAL 2.2.0 for Linux.

I think this module is used by ESA Tools to read JP2 Files.

UPDATE:
May be this module has the key, It reads the content of Sentinel2 products

@ahuarte47 ahuarte47 force-pushed the master-cli-sentinel2 branch 3 times, most recently from 95fdbb6 to 78d28fc Compare May 1, 2018 07:42
@rfecher
Copy link
Copy Markdown
Contributor

rfecher commented May 1, 2018

the module you linked looks promising, but one caveat that we will have to be careful with, we can't add it as a direct dependency (I think we can use it with maven "test" scope though which may work out ok)
...
the license is GPL, which is incompatible with EPL (Eclipse Product License) or Apache 2.0
to be IP-compliant with the Eclipse Foundation's standards we can use it in the sense that it "works with" our software, but we cannot "depend on" it. I believe running tests only with it included as a maven module but not including it anywhere else, will be sufficient for the "works with" definition.

@ahuarte47 ahuarte47 force-pushed the master-cli-sentinel2 branch 6 times, most recently from 3c362d8 to 58b249c Compare May 3, 2018 07:59
@ahuarte47
Copy link
Copy Markdown
Contributor Author

ahuarte47 commented May 3, 2018

Hi @rfecher, I have fixed the sentinel2 module in Travis (Linux with GDAL 1.9.2). I found several options, but I think I have chosen the least intrusive one in order to modify the global behavior as minimum as possible.

I was testing:

  • Change the GDAL 1.9.2 redist used to test with a fixed version of the JP2ECW core in Linux platforms. It requires to download it from an unknown public resource (this doesn't work and the ESA module has GPL license as you said).
  • Clear the NCS_USER_PREFS environment variable for testing to avoid the JP2ECW plugin loading.

I have chosen the last one.

@rfecher
Copy link
Copy Markdown
Contributor

rfecher commented May 3, 2018

Where did you download the test JP2 file thats included in this PR? Also, to pass IP scrutiny, we need to make sure any data included in the baseline comes with an appropriately open license. If we're not sure about the license, it ends up being better to have the software download it as part of the test rather than include it in the baseline (for this we don't have to worry as much about the licensing because we are not distributing the data).

@ahuarte47
Copy link
Copy Markdown
Contributor Author

ahuarte47 commented May 3, 2018

I downloaded it from AWS (I think is the "T30TXN-2017-10-16" scene), I was searching a very little image. But I could confirm it if you want.

public static final String THEIA_AUTHENTICATION_FILE = "auth_theia.txt";

// URL of raster to validate JP2 support in GDAL.
private final static String JP2ECW_RASTER_TEST_URL = "http://sentinel-s2-l1c.s3.amazonaws.com/tiles/30/T/WN/2017/10/16/0/preview/B01.jp2";
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it looks like the tests are already setup to download the file, can we just remove the file from the baseline and have it automatically downloaded? It would be easier for IP review, and seems to already be handled here.

Copy link
Copy Markdown
Contributor Author

@ahuarte47 ahuarte47 May 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for your advices and for reviewing. I have added a new JP2 in data folder to check when testing the JP2ECW plugin. I have verified that no every JP2 files from sentinel2 get to crash the GDAL 1.9.2 redist for Linux. This file is little and breaks GDAL. My apologies I don't remember whether I downloaded it from AWS or PEPS (LEVEL1C mirror at France). Is there any problem?

@ahuarte47 ahuarte47 force-pushed the master-cli-sentinel2 branch 3 times, most recently from 2781e91 to 5b10b31 Compare May 8, 2018 07:38
It will be replaced by a new more generic Sentinel2 client module
@ahuarte47 ahuarte47 force-pushed the master-cli-sentinel2 branch from 5b10b31 to 76e09e5 Compare May 19, 2018 16:09
@rfecher rfecher merged commit 4b897c4 into locationtech:master May 29, 2018
@ahuarte47
Copy link
Copy Markdown
Contributor Author

Thanks @rfecher for reviewing and merging!

@rfecher
Copy link
Copy Markdown
Contributor

rfecher commented May 29, 2018

of course, sorry for the delays

@ahuarte47
Copy link
Copy Markdown
Contributor Author

You are welcome!

@ahuarte47 ahuarte47 deleted the master-cli-sentinel2 branch May 29, 2018 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants