Core: add variant builder implementation #11857

aihuaxu · 2024-12-22T20:07:51Z

This PR adds the implementation to build a variant for primitive, object and array.

The usage:

Build from Json
VariantBuilder.parseJson(json_string);
build a primitive

VariantBuilder builder = new VariantBuilder();
VariantPrimitiveBuilder primBuilder = builder.createPrimitive();
Variant variant = primBuilder.writeNumeric(10).build();

build an object

 VariantBuilder builder = new VariantBuilder();
 VariantObjectBuilder objBuilder = builder.startObject();
 objBuilder.writeString("key1", "value1").writeNumeric("key2", 3).endObject();
 Variant variant = builder.build();

build an array

 VariantBuilder builder = new VariantBuilder();
 VariantArrayBuilder arrBuilder = builder.createArray();
 arrBuilder.writerNumeric(10).writerNumeric(20).endArray();
 Variant variant = builder.build();

Nested arrays and objects can be supported as well. Please see the test cases for the usage. Compared to build from JSON string, we also support some additional types such as timestamp, date.

Part of: #10392

aihuaxu · 2025-01-06T22:42:26Z

@rdblue, @RussellSpitzer Please help review. Thanks.

core/src/main/java/org/apache/iceberg/variants/VariantArray.java

core/src/main/java/org/apache/iceberg/variants/VariantSizeLimitException.java

core/src/main/java/org/apache/iceberg/variants/VariantImpl.java

core/src/main/java/org/apache/iceberg/variants/VariantConstants.java

rdblue · 2025-01-24T01:01:46Z

core/src/main/java/org/apache/iceberg/variants/VariantBuilder.java

+    Preconditions.checkArgument(
+        json != null && !json.isEmpty(), "Input JSON string cannot be null or empty.");
+
+    try (JsonParser parser = new JsonFactory().createParser(json)) {


There are existing utilities in JSONUtil for a lot of the boilerplate around parsing and generating JSON. Is there a reason not to use the same approach here?

We need to parse the JSON token by token so we can encode them in binary. Seems we only have the version to parse the JSON string into a full JSONNode and then create the object.

core/src/main/java/org/apache/iceberg/variants/VariantBuilder.java

core/src/main/java/org/apache/iceberg/variants/VariantBuilderBase.java

xxubai · 2025-02-14T09:25:30Z

core/src/main/java/org/apache/iceberg/variants/VariantBuilderBase.java

+  }
+
+  protected void writeNullInternal() {
+    valueBuffer.writePrimitive(Variants.PhysicalType.NULL, null);


Suggested change

valueBuffer.writePrimitive(Variants.PhysicalType.NULL, null);

valueBuffer.writePrimitive(PhysicalType.NULL, null);

github-actions · 2025-03-17T00:16:59Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

github-actions · 2025-03-25T00:16:42Z

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

github-actions bot added the core label Dec 22, 2024

aihuaxu marked this pull request as ready for review December 22, 2024 20:11

aihuaxu force-pushed the variant-add-writer-impl branch 6 times, most recently from f8f8e8f to d806c66 Compare December 24, 2024 03:54

aihuaxu force-pushed the variant-add-writer-impl branch from d806c66 to 16210a0 Compare January 6, 2025 22:26

aihuaxu force-pushed the variant-add-writer-impl branch from 16210a0 to d4f1d0e Compare January 6, 2025 22:57