[SPARK-9480][SQL] add MapData and cleanup internal row stuff #7799

cloud-fan · 2015-07-30T16:22:45Z

This PR adds a MapData as internal representation of map type in Spark SQL, and provides a default implementation with just 2 ArrayData.

After that, we have specialized getters for all internal type, so I removed generic getter in ArrayData and added specialized toArray for it.
Also did some refactor and cleanup for InternalRow and its subclasses.

SparkQA · 2015-07-30T16:47:28Z

Test build #39068 has finished for PR 7799 at commit dc67221.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class InternalRow extends SpecializedGetters with Serializable
- trait ArrayBackendSpecializedGetters extends SpecializedGetters
- abstract class CodeGenMutableRow extends MutableRow
- final class SpecificRow extends $
- case class GetMapValue(child: Expression, key: Expression) extends BinaryExpression
- class GenericInternalRow(protected[sql] val array: Array[Any])
- class GenericInternalRowWithSchema(array: Array[Any], val schema: StructType)
- class GenericMutableRow(protected[sql] val array: Array[Any])
- abstract class ArrayData extends SpecializedGetters with Serializable
- class GenericArrayData(protected[sql] val array: Array[Any])
- abstract class MapData extends Serializable

cloud-fan · 2015-07-31T13:14:27Z

sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java

moved it to the private region.

SparkQA · 2015-07-31T22:40:21Z

Test build #1265 has finished for PR 7799 at commit 3e7c48a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class InternalRow extends GenericSpecializedGetters with Serializable
- trait GenericSpecializedGetters extends SpecializedGetters
- case class GetArrayItem(child: Expression, ordinal: Expression)
- case class GetMapValue(child: Expression, key: Expression)
- class ArrayBasedMapData(val keyArray: ArrayData, val valueArray: ArrayData) extends MapData
- class GenericArrayData(array: Array[Any]) extends ArrayData with GenericSpecializedGetters
- abstract class MapData extends Serializable

SparkQA · 2015-08-01T00:57:32Z

Test build #39298 has finished for PR 7799 at commit 08bcb40.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MultilayerPerceptronClassifier(override val uid: String)
- class KinesisUtils(object):
- class InitialPositionInStream(object):
- abstract class InternalRow extends GenericSpecializedGetters with Serializable
- trait GenericSpecializedGetters extends SpecializedGetters
- case class GetArrayItem(child: Expression, ordinal: Expression)
- case class GetMapValue(child: Expression, key: Expression)
- case class SoundEx(child: Expression) extends UnaryExpression with ExpectsInputTypes
- class ArrayBasedMapData(val keyArray: ArrayData, val valueArray: ArrayData) extends MapData
- class GenericArrayData(array: Array[Any]) extends ArrayData with GenericSpecializedGetters
- abstract class MapData extends Serializable

SparkQA · 2015-08-01T02:20:32Z

Test build #39306 has finished for PR 7799 at commit f482274.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class InternalRow extends GenericSpecializedGetters with Serializable
- trait GenericSpecializedGetters extends SpecializedGetters
- case class GetArrayItem(child: Expression, ordinal: Expression)
- case class GetMapValue(child: Expression, key: Expression)
- case class SoundEx(child: Expression) extends UnaryExpression with ExpectsInputTypes
- class ArrayBasedMapData(val keyArray: ArrayData, val valueArray: ArrayData) extends MapData
- class GenericArrayData(array: Array[Any]) extends ArrayData with GenericSpecializedGetters
- abstract class MapData extends Serializable

SparkQA · 2015-08-01T03:09:23Z

Test build #39311 has finished for PR 7799 at commit 5d13023.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class InternalRow extends GenericSpecializedGetters with Serializable
- trait GenericSpecializedGetters extends SpecializedGetters
- case class GetArrayItem(child: Expression, ordinal: Expression)
- case class GetMapValue(child: Expression, key: Expression)
- case class SoundEx(child: Expression) extends UnaryExpression with ExpectsInputTypes
- class ArrayBasedMapData(val keyArray: ArrayData, val valueArray: ArrayData) extends MapData
- class GenericArrayData(array: Array[Any]) extends ArrayData with GenericSpecializedGetters
- abstract class MapData extends Serializable

cloud-fan · 2015-08-01T03:51:34Z

retest this please.

SparkQA · 2015-08-01T05:18:37Z

Test build #184 has finished for PR 7799 at commit 5d13023.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-01T05:26:16Z

Test build #39322 has finished for PR 7799 at commit 5d13023.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class InternalRow extends GenericSpecializedGetters with Serializable
- trait GenericSpecializedGetters extends SpecializedGetters
- case class GetArrayItem(child: Expression, ordinal: Expression)
- case class GetMapValue(child: Expression, key: Expression)
- class ArrayBasedMapData(val keyArray: ArrayData, val valueArray: ArrayData) extends MapData
- class GenericArrayData(array: Array[Any]) extends ArrayData with GenericSpecializedGetters
- abstract class MapData extends Serializable

SparkQA · 2015-08-01T06:48:55Z

Test build #39328 has finished for PR 7799 at commit e8f6682.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class InternalRow extends GenericSpecializedGetters with Serializable
- trait GenericSpecializedGetters extends SpecializedGetters
- case class GetArrayItem(child: Expression, ordinal: Expression)
- case class GetMapValue(child: Expression, key: Expression)
- class ArrayBasedMapData(val keyArray: ArrayData, val valueArray: ArrayData) extends MapData
- class GenericArrayData(array: Array[Any]) extends ArrayData with GenericSpecializedGetters
- abstract class MapData extends Serializable

SparkQA · 2015-08-01T07:05:16Z

Test build #39330 has finished for PR 7799 at commit 77d482f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class InternalRow extends GenericSpecializedGetters with Serializable
- trait GenericSpecializedGetters extends SpecializedGetters
- case class GetArrayItem(child: Expression, ordinal: Expression)
- case class GetMapValue(child: Expression, key: Expression)
- class ArrayBasedMapData(val keyArray: ArrayData, val valueArray: ArrayData) extends MapData
- class GenericArrayData(array: Array[Any]) extends ArrayData with GenericSpecializedGetters
- abstract class MapData extends Serializable

rxin · 2015-08-01T07:10:05Z

Alright I'm going to merge this since this is a large change.

cc @davies can you take a quick look at this when you have a chance? Thanks.

cloud-fan force-pushed the map-data branch from 270e511 to dc67221 Compare July 30, 2015 16:27

cloud-fan force-pushed the map-data branch from dc67221 to 9875cbf Compare July 31, 2015 12:51

cloud-fan changed the title ~~[SPARK-9480][SQL][WIP] add MapData~~ [SPARK-9480][SQL] add MapData and cleanup internal row stuff Jul 31, 2015

cloud-fan reviewed Jul 31, 2015
View reviewed changes

sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java

Copy link

Contributor Author

cloud-fan Jul 31, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved it to the private region.

cloud-fan force-pushed the map-data branch from 3e7c48a to 08bcb40 Compare July 31, 2015 23:37

cloud-fan force-pushed the map-data branch from 5d13023 to e8f6682 Compare August 1, 2015 05:14

cloud-fan added 4 commits August 1, 2015 13:14

add MapData

a90aca1

some more cleanup

6e06ec9

add toString

40cc9db

skip MapData equality check in HiveInspectorSuite

e8f6682

fix python

77d482f

asfgit closed this in 1d59a41 Aug 1, 2015

[SPARK-9480][SQL] add MapData and cleanup internal row stuff #7799

[SPARK-9480][SQL] add MapData and cleanup internal row stuff #7799

Uh oh!

Conversation

cloud-fan commented Jul 30, 2015

Uh oh!

SparkQA commented Jul 30, 2015

Uh oh!

cloud-fan Jul 31, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 31, 2015

Uh oh!

SparkQA commented Aug 1, 2015

Uh oh!

SparkQA commented Aug 1, 2015

Uh oh!

SparkQA commented Aug 1, 2015

Uh oh!

cloud-fan commented Aug 1, 2015

Uh oh!

SparkQA commented Aug 1, 2015

Uh oh!

SparkQA commented Aug 1, 2015

Uh oh!

SparkQA commented Aug 1, 2015

Uh oh!

SparkQA commented Aug 1, 2015

Uh oh!

rxin commented Aug 1, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants