Relevant updates to fully support Cassandra and DynamoDB#1305
Merged
Relevant updates to fully support Cassandra and DynamoDB#1305
Conversation
DynamoDB passing GeoWaveBasicSpatialVectorIT and GeoWaveBasicRasterIT
…support using async handler for batchWrite and PutItem. On flush, the pending async requests are waited on for (#1029)
* Adding support for asynchronous query in DynamoDBQuery * Make the Async Paginated Query cleaner with the following changes:- 1. Use a intrinsic sync block instead of specific re-entrant locks using try and finally. 2. Use linked list instead of array deque to allow inserting null locks. This prevents wait forevers when we have exception in async queries 3. Make sure we don't fire queries immediately on construction if there are already too many async queries piled up. 4. Reduce the max async query limit to 100. This can be made configurable through metadata in the future
* Initial checkin for casanamo mapreduce Specifically for DynamoDB, it has the ranges working for the split, need to next work on the dynamo db split creation in populateIntermediateSplits * Code review comments; mostly cleaning up comments and text changes
Note that something is still missing; it looks like HBase and DynamoDB have identical code but yet, HBase returns 5 entries whereas DynamoDB only returns one
Changes: Added support for region. Now a AWS region can be specified Made entrypoint optional. If both are specified- region will be considered. Added support for client configuration. Currently max number of http connections, protocol and cache response metadata is allowed. Further support can be added easily, by modifying the "getClientConfig()" function. Adding clear error messages when both or none of region/entrypoint are specified
* Adding support for asynchronous query in DynamoDBQuery * Make the Async Paginated Query cleaner with the following changes:- 1. Use a intrinsic sync block instead of specific re-entrant locks using try and finally. 2. Use linked list instead of array deque to allow inserting null locks. This prevents wait forevers when we have exception in async queries 3. Make sure we don't fire queries immediately on construction if there are already too many async queries piled up. 4. Reduce the max async query limit to 100. This can be made configurable through metadata in the future
… test cases to fail
Conflicts: extensions/datastores/dynamodb/src/main/java/mil/nga/giat/geowave/datastore/dynamodb/query/DynamoDBQuery.java
… noticed in other databases This caused the GeoWaveSpatialVectorIT test to fail
Ap casanamo adapterid 2
Signed-off-by: Johnathan Garrett <jd@prominentedge.com>
Signed-off-by: Johnathan Garrett <jd@prominentedge.com>
c42b426 to
4e21c5e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a long history of significant updates that were developed on a "casanamo" branch now ready to be merged into the master branch.