Optimizations to spatial join for other geometry types.#1343
Merged
rfecher merged 1 commit intolocationtech:masterfrom Jul 18, 2018
Merged
Optimizations to spatial join for other geometry types.#1343rfecher merged 1 commit intolocationtech:masterfrom
rfecher merged 1 commit intolocationtech:masterfrom
Conversation
rfecher
reviewed
Jun 7, 2018
Contributor
rfecher
left a comment
There was a problem hiding this comment.
a few minor changes requested
| } | ||
| if (rawGeometryRDD == null) { | ||
| JavaPairRDD<ByteArrayId, Tuple2<GeoWaveInputKey, Geometry>> indexedData = geowaveRDD | ||
| if (rawGeometryRDD == null || recalculate == true) { |
Contributor
There was a problem hiding this comment.
can we change this to simply recalculate without the == true
| "tornado_tracks"); | ||
| GeomWithinDistance distancePredicate = new GeomWithinDistance( | ||
| 0.01); | ||
| //GeomIntersects intersectsPredicate = new GeomIntersects(); |
| log4j.appender.stdout.filter.1.StringToMatch=Error closing output stream | ||
| log4j.appender.stdout.filter.1.StringToMatch=Error closing output stream. | ||
| log4j.appender.stdout.filter.1.AcceptOnMatch=false | ||
| log4j.appender.stdout.filter=org.apache.log4j.varia.DenyAllFilter |
| import org.slf4j.LoggerFactory; | ||
|
|
||
| import com.vividsolutions.jts.geom.Geometry; | ||
| import com.vividsolutions.jts.geom.prep.PreparedGeometry; |
Contributor
There was a problem hiding this comment.
added these prepared geometry imports but seemingly never use them (this is the case it seems in a bunch of the classes)
| Geometry geom1, | ||
| Geometry geom2 ) { | ||
| return geom1.distance(geom2) <= radius; | ||
| return geom1.distance( |
Contributor
There was a problem hiding this comment.
formatting change, can you run mvn on the commandline to make sure this is right? (do you run the formatter in eclipse IDE because that may be different?)
| Geometry geom1, | ||
| Geometry geom2 ) { | ||
| return geom1.equals(geom2); | ||
| return geom1.equals( |
Contributor
There was a problem hiding this comment.
format change, check if this is right
| Serializable | ||
| { | ||
| private JavaPairRDD<GeoWaveInputKey, SimpleFeature> rawRDD = null; | ||
| protected JavaPairRDD<GeoWaveInputKey, SimpleFeature> rawRDD = null; |
Contributor
There was a problem hiding this comment.
this is minor, but I can't say I see the reasoning for changing this to protected
674f201 to
70cef8e
Compare
rfecher
reviewed
Jun 14, 2018
| import mil.nga.giat.geowave.analytic.spark.RDDOptions; | ||
| import mil.nga.giat.geowave.core.store.query.QueryOptions; | ||
| import mil.nga.giat.geowave.analytic.spark.sparksql.SimpleFeatureDataFrame; | ||
| import mil.nga.giat.geowave.analytic.spark.sparksql.udf.GeomIntersects; |
rfecher
reviewed
Jun 14, 2018
| log4j.appender.stdout.layout=org.apache.log4j.PatternLayout | ||
| log4j.appender.stdout.layout.ConversionPattern=%d{dd MMM HH:mm:ss} %p [%c{2}] - %m%n | ||
| log4j.appender.stdout.filter.1=org.apache.log4j.varia.StringMatchFilter | ||
| log4j.appender.stdout.filter.1.StringToMatch=Error closing output stream. |
Contributor
There was a problem hiding this comment.
is this not message not already suppressed with log4j.category.org.apache.thrift=ERROR ?
rfecher
reviewed
Jun 14, 2018
Contributor
rfecher
left a comment
There was a problem hiding this comment.
I made a couple comments that are fairly minor. I can pull this in if you're comfortable with it, or make minor changes.
5d30395 to
36422b8
Compare
36422b8 to
ba499d3
Compare
Squashed commits: [ba499d3] Optimizations to spatial join for other geometry types. Changes include trimming insertion keys, and reducing overall algorithm stages.
ba499d3 to
49413c3
Compare
rfecher
reviewed
Jul 18, 2018
|
|
||
| Envelope internalEnvelope = geom.getEnvelopeInternal(); | ||
| if (internalEnvelope.isNull()) { | ||
| return Iterators.emptyIterator(); |
Contributor
There was a problem hiding this comment.
is that de.javakaffee library only because of this?
| // keys | ||
| // Flattened output array. | ||
| List<Tuple2<ByteArrayId, Tuple2<GeoWaveInputKey, Geometry>>> result = Lists | ||
| .newArrayListWithCapacity(insertIds.getSize()); |
| log4j.appender.stdout.filter.1=org.apache.log4j.varia.StringMatchFilter | ||
| log4j.appender.stdout.filter.1.StringToMatch=Error closing output stream | ||
| log4j.appender.stdout.filter.1.AcceptOnMatch=false | ||
| log4j.appender.stdout.filter=org.apache.log4j.varia.DenyAllFilter |
Contributor
There was a problem hiding this comment.
didn't we agree that this effectively suppressed warnings that we were trying to suppress?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes include trimming insertion keys, and reducing overall algorithm stages.