Skip to content

Relevant updates to fully support Cassandra and DynamoDB#1305

Merged
rfecher merged 32 commits intomasterfrom
casanamo-pr
Apr 2, 2018
Merged

Relevant updates to fully support Cassandra and DynamoDB#1305
rfecher merged 32 commits intomasterfrom
casanamo-pr

Conversation

@rfecher
Copy link
Copy Markdown
Contributor

@rfecher rfecher commented Mar 30, 2018

This is a long history of significant updates that were developed on a "casanamo" branch now ready to be merged into the master branch.

rfecher and others added 30 commits December 30, 2016 12:46
DynamoDB passing GeoWaveBasicSpatialVectorIT and GeoWaveBasicRasterIT
…support using async handler for batchWrite and PutItem. On flush, the pending async requests are waited on for (#1029)
* Adding support for asynchronous query in DynamoDBQuery

* Make the Async Paginated Query cleaner with the following changes:-

1. Use a intrinsic sync block instead of specific re-entrant locks using try and finally.
2. Use linked list instead of array deque to allow inserting null locks. This prevents wait forevers when we have exception in async queries
3. Make sure we don't fire queries immediately on construction if there are already too many async queries piled up.
4. Reduce the max async query limit to 100. This can be made configurable through metadata in the future
* Initial checkin for casanamo mapreduce
Specifically for DynamoDB, it has the ranges working for the split, need to next work on the dynamo db split creation in populateIntermediateSplits

* Code review comments; mostly cleaning up comments and text changes
Note that something is still missing; it looks like HBase and DynamoDB have identical code but yet, HBase returns 5 entries whereas DynamoDB only returns one
Changes:
Added support for region. Now a AWS region can be specified
Made entrypoint optional.
If both are specified- region will be considered.
Added support for client configuration. Currently max number of http connections, protocol and cache response metadata is allowed.
Further support can be added easily, by modifying the "getClientConfig()" function.
Adding clear error messages when both or none of region/entrypoint are specified
* Adding support for asynchronous query in DynamoDBQuery

* Make the Async Paginated Query cleaner with the following changes:-

1. Use a intrinsic sync block instead of specific re-entrant locks using try and finally.
2. Use linked list instead of array deque to allow inserting null locks. This prevents wait forevers when we have exception in async queries
3. Make sure we don't fire queries immediately on construction if there are already too many async queries piled up.
4. Reduce the max async query limit to 100. This can be made configurable through metadata in the future
Conflicts:
	extensions/datastores/dynamodb/src/main/java/mil/nga/giat/geowave/datastore/dynamodb/query/DynamoDBQuery.java
… noticed in other databases

This caused the GeoWaveSpatialVectorIT test to fail
Signed-off-by: Johnathan Garrett <jd@prominentedge.com>
Signed-off-by: Johnathan Garrett <jd@prominentedge.com>
@rfecher rfecher force-pushed the casanamo-pr branch 3 times, most recently from c42b426 to 4e21c5e Compare March 30, 2018 17:53
@rfecher rfecher merged commit 66e3cc2 into master Apr 2, 2018
@rfecher rfecher deleted the casanamo-pr branch April 2, 2018 02:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants