Merged
Conversation
Contributor
Author
Contributor
Author
|
Manual Test: S3 Download Manual test is completed successfully. Restart the sandbox. Clear the items from cache. Conduct load now accepts S3 URL as input. S3 download displays the progress bar. Subsequent call loads the artefact from cache. |
Contributor
Author
|
Manual Test: S3 Download with Native Executable Manual test is completed successfully. Restart the sandbox. Build the the native executable. Clear the items from cache. Use native executable to invoke the commands. Conduct load now accepts S3 URL as input. S3 download displays the progress bar. Subsequent call loads the artefact from cache. |
Contributor
Author
|
Manual test: malformed S3 URLs Manual test is completed successfully. The following error is displayed if file doesn't exist. The following error is displayed if S3 URL is incomplete, i.e. bucket name specified without key. The following error is displayed if attempting to access unauthorized S3 URL. |
Contributor
Author
|
Manual Test: offline mode S3 resolver does not support offline mode. |
Contributor
Author
|
Manual test: Credentials Manual test is completed successfully. Restart the sandbox. Delete items from cache. Use AWS credentials file containing on The Delete items from cache for the next test. Use invalid credentials. The following error message is displayed. |
Contributor
Author
|
Manual test: Testing other resolvers Other resolvers still works as expected. Restart the sandbox. Delete Items from cache. Bintray resolver works as expected. Bintray resolver is able to load from cache successfully. Docker resolver works as expected. Docker resolver loads from cache as expected as well. URI resolver works as expected - STDIN resolver works as expected. |
huntc
reviewed
Jun 16, 2017
|
|
||
| import os | ||
|
|
||
| # Temporary workaround until PyInstaller's boto hook imports the correct method |
huntc
suggested changes
Jun 16, 2017
| import urllib | ||
|
|
||
|
|
||
| UNSUPPORTED_SCHEMES = ['s3'] |
Contributor
There was a problem hiding this comment.
Let's remove this in favour of having a resolver be asked whether it supports a specific scheme or not. This way, if an explicit scheme is provided by the user then we can also skip resolvers that we know are not able to handle the scheme e.g. there is only one resolver that can handle s3:.
|
|
||
| import os | ||
|
|
||
| # Temporary workaround until PyInstaller's boto hook imports the correct method |
e5f3c23 to
7f72737
Compare
Contributor
Author
|
Manual test: S3 download Manual test is completed successfully. Restart the sandbox. Clear the cache. Bundle + configuration is loaded from S3. The Load from cache works as expected. |
Contributor
Author
|
Manual test: S3 download with native executable Manual test is completed successfully. Restart the sandbox. Rebuild the native executable. Clear the cache. Use the native executable, and load from S3. Loading from cache works as expected. |
Contributor
Author
|
Manual Test: S3 Credentials Manual test is completed successfully. Restart the sandbox. Clear the cache. S3 resolver works with S3 resolver will fail if invalid profile is specified. Put back the correct credentials after test. |
Contributor
Author
|
Manual Test: Malformed S3 URLs When S3 key points to a non-existent path, the following error is displayed: When S3 key can't be deduced from the S3 URL, the following error is displayed: When user has no permission, the following error is displayed: |
Contributor
Author
|
Manual Test: Offline S3 resolver S3 resolver doesn't support offline resolution. |
Contributor
Author
|
Manual Test: Bintray Resolver Manual test is completed successfully. Restart the sandbox. Simple bundle expression is resolved successfully. Loading from cache works as expected. Similarly offline mode. Loading bundle by latest tag works as expected. This also works when the full bundle and configuration version is specified. When repo is specified, resolution works as expected. When the full shorthand URN is specified, this is resolved from Bintray successfully. |
Contributor
Author
|
Manual test: Docker resolver Manual test is completed successfully. Restart the sandbox. Docker image can be resolved from simple image name. Also when namespace is specified. Also works when tag is specified. |
Contributor
Author
|
Manual Test: Docker resolver w/ 1.0 image The following error is displayed when trying to load |
Contributor
Author
|
Manual Test: Other resolvers Manual test is completed successfully. Restart the sandbox. Clear the cache. When file URI is specified, bundle is resolved and loaded successfully. STDIN resolver also works as expected. |
huntc
suggested changes
Jun 20, 2017
| BINTRAY_CONDUCTR_CORE_PACKAGE_NAME = 'ConductR-Universal' | ||
| BINTRAY_CONDUCTR_AGENT_PACKAGE_NAME = 'ConductR-Agent-Universal' | ||
|
|
||
| SCHEME_SUPPORTED_BY_OTHER_RESOLVERS = ['http', 'file', 'ftp', 's3'] |
Contributor
There was a problem hiding this comment.
Isn't this going to present a maintenance issue... isn't it better to concern oneself with just the URI schemes that the resolver supports?
| DOCKER_CREDENTIAL_FILE_PATH = '{}/.lightbend/docker.credentials'.format(os.path.expanduser('~')) | ||
| DOCKER_PROPERTIES_RE = re.compile('^(\S+)\s*=\s*([\S]+)$') | ||
|
|
||
| SCHEME_SUPPORTED_BY_OTHER_RESOLVERS = ['http', 'file', 'ftp', 's3'] |
Contributor
Author
|
@huntc - it's going to be tricky to do unfortunately because how loose the URL syntax is. So for example |
486b79f to
5012373
Compare
Provide additional resolver which will be able to resolve a given S3 URL: ``` s3://<bucket name>/<key to bundle or configuration zip> ``` The S3 resolver relies on AWS provided `boto3` library which has been added as dependency. **Boto3 PyInstaller Notes** Some workaround is in place to ensure `boto3` is included correctly into native executable. This is required due to PyInstaller's `boto3` hooks importing incorrect module name: pyinstaller/pyinstaller#2384 Fix is scheduled for PyInstaller 3.3, and the workaround should be able to be removed afterwards. Additionally, `boto3` in native executable requires having the module `configparser` to be declared as hidden import. **Modification to resolver mechanism** Resolver now declares `supported_schemes()` which returns a list of URI schemes supported by a particular resolver, i.e. `file`, `http`, or `urn:x-bundle`. Applicable URI schemes is now derived from the input URI. The applicable schemes derived from the input URI is compared against the URI schemes supported by each resolver, and the resolver chain is filtered based on the result of this comparison.
Contributor
Author
|
Implemented the change as per feedback:
|
Contributor
Author
|
Manual test is done successfully. I will post the recorded result shortly. |
Contributor
Author
|
Manual test: S3 download Manual test is completed successfully. Restart the sandbox. Clear the cache. Bundle + configuration is loaded from S3. The Load from cache works as expected. |
Contributor
Author
|
Manual test: S3 download with native executable Manual test is completed successfully. Restart the sandbox. Rebuild the native executable. Clear the cache. Use the native executable, and load from S3. Loading from cache works as expected. |
Contributor
Author
|
Manual Test: S3 Credentials Manual test is completed successfully. Restart the sandbox. Clear the cache. S3 resolver works with S3 resolver will fail if invalid profile is specified. Put back the correct credentials after test. |
Contributor
Author
|
Manual Test: Malformed S3 URLs When S3 key points to a non-existent path, the following error is displayed: When S3 key can't be deduced from the S3 URL, the following error is displayed: When user has no permission, the following error is displayed: |
Contributor
Author
|
Manual Test: Offline S3 resolver S3 resolver doesn't support offline resolution. |
Contributor
Author
|
Manual Test: Bintray Resolver Manual test is completed successfully. Restart the sandbox. Simple bundle expression is resolved successfully. Note the resolver chain includes Loading from cache works as expected. Similarly offline mode. Note Loading bundle by latest tag works as expected. This also works when the full bundle and configuration version is specified. When repo is specified, resolution works as expected. When the full shorthand URN is specified, this is resolved from Bintray successfully. Note that |
Contributor
Author
|
Manual test: Docker resolver Manual test is completed successfully. Restart the sandbox. Docker image can be resolved from simple image name. Also when namespace is specified. Also works when tag is specified. Also works when the shorthand specifying Docker registry URI is specified. |
Contributor
Author
|
Manual Test: Docker resolver w/ 1.0 image The following error is displayed when trying to load |
Contributor
Author
|
Manual Test: Other resolvers Manual test is completed successfully. Restart the sandbox. Clear the cache. When file URI is specified, bundle is resolved and loaded successfully. Note that only STDIN resolver also works as expected. Note that only |
huntc
approved these changes
Jun 21, 2017
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Provide additional resolver which will be able to resolve a given S3 URL:
The S3 resolver relies on AWS provided
boto3library which has been added as dependency.Boto3 PyInstaller Notes
Some workaround is in place to ensure
boto3is included correctly into native executable. This is required due to PyInstaller'sboto3hooks importing incorrect module name:pyinstaller/pyinstaller#2384
Fix is scheduled for PyInstaller 3.3, and the workaround should be able to be removed afterwards.
Additionally,
boto3in native executable requires having the moduleconfigparserto be declared as hidden import.Modification to resolver mechanism
Resolver now declares
supported_schemes()which returns a list of URI schemes supported by a particular resolver, i.e.file,http, orurn:x-bundle.Applicable URI schemes is now derived from the input URI. The applicable schemes derived from the input URI is compared against the URI schemes supported by each resolver, and the resolver chain is filtered based on the result of this comparison.
boto3can be packaged successfully using PyInstallers3://style URLs