Skip to content

Commit 30d37dd

Browse files
harshachintagauravpurohit06pjuhosgcf-owl-bot[bot]olavloite
authored
feat: add support for Proto Columns (#2779)
* feat: Support for Proto Messages & Enums (#2155) * feat: Importing Proto Changes Commit will be reverted, once PROTO changes are available publicly. * feat: Proto Message Implementation * feat: Adding support for enum * feat: Code refactoring Adding default implementation for newly added methods ByteArray compatability changes for Proto Messages * docs: Adding Java docs for all the newly added methods. * test: Sample Proto & Generated classes for unit test * feat: Adding bytes/proto & int64/enum compatability Adding Additional check for ChecksumResultSet * test: Adding unit tests * test: Adding unit tests for ValueBinder.java * feat: refactoring to add support for getValue & other minor changes * feat: Minor refactoring 1. Adding docs and formatting the code. 2. Adding additional methods for enum and message which accepts descriptors. * feat: Adding bytes/message & int64/enum compatability in Value * refactor: Minor refactoring * feat: Adding Proto Array Implementation * test: Implementing unit tests for array of protos and enums * refactor: adding clirr ignores * feat: Adding support for enum as Primary Key * feat: Code Review Changes, minor refactoring and adding docs * feat: Addressing review comments -Modified Docs/Comments -Minor Refactoring * refactor: Using Column instead of column to avoid test failures * feat: Minor refactoring -code review comments -adding function docs * samples: Adding samples for updating & querying Proto messages & enums (#2211) * samples: Adding samples for updating & querying Proto messages & enums * style: linting * style: linting * docs: Adding function and class doc * test: Proto Column Integration tests (#2212) * test: Adding Integration tests for Proto Messages & Enums * test: Adding additional test for Parameterized Queries, Primary Keys & Invalid Wire type errors. * style: Formatting * style: Formatting * test: Updating instance and db name * test: Adding inter compatability check while writing data * Configured jitpack.yml to use OpenJDK 11 (#2218) Co-authored-by: Pavol Juhos <pjuhos@google.com> * feat: add support for Proto Columns DDL (#2277) * feat: add code changes and tests for Proto columns DDL support * feat: add auto generated code * feat: code changes and tests for Proto columns DDL support * feat: add descriptors file * feat: code refactoring * feat: Integration tests and code refactoring * feat: code refactoring * feat: unit tests and clirr differences * feat: lint changes * feat: code refactor * feat: code refactoring * feat: code refactoring * feat: code refactoring * feat: add java docs to new methods * feat: lint formatting * feat: lint formatting changes * feat: lint formatting * feat: lint formatting * feat: test exception cases * feat: code refactoring * feat: add java docs and refactoring * feat: add java docs * feat: java docs refactor * feat: remove overload method setProtoDescriptors that accepts file path as input to avoid unexpected issues * feat: remove updateDdl method overload to update proto descriptor * teat: update pom file to run tests on cloud-devel region temporarily to validate main branch update * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix: revert host changes in pom.xml file * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat(spanner): revert autogenerated code * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat(spanner): remove samples * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat(spanner): remove clirr * feat(spanner): skip emulator test * feat(spanner): clirr * feat(spanner): fix javadoc * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat(spanner): fix javadoc * feat(spanner): fix javadoc * feat(spanner): fix javadoc * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat(spanner): fix emulator skip * build: ignore all changes in v1 package * feat(spanner): add optimizations to deserialize proto messages * feat(spanner): remove TODO * feat(spanner): remove TODO --------- Co-authored-by: Gaurav Purohit <gauravpurohit@google.com> Co-authored-by: Pavol Juhos <pjuhos@google.com> Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Knut Olav Løite <koloite@gmail.com>
1 parent 95f064f commit 30d37dd

38 files changed

+3945
-117
lines changed

‎google-cloud-spanner/clirr-ignored-differences.xml‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -192,6 +192,46 @@
192192
<className>com/google/cloud/spanner/StructReader</className>
193193
<method>java.util.List getPgJsonbList(java.lang.String)</method>
194194
</difference>
195+
<difference>
196+
<differenceType>7012</differenceType>
197+
<className>com/google/cloud/spanner/StructReader</className>
198+
<method>com.google.protobuf.ProtocolMessageEnum getProtoEnum(int, java.util.function.Function)</method>
199+
</difference>
200+
<difference>
201+
<differenceType>7012</differenceType>
202+
<className>com/google/cloud/spanner/StructReader</className>
203+
<method>com.google.protobuf.ProtocolMessageEnum getProtoEnum(java.lang.String, java.util.function.Function)</method>
204+
</difference>
205+
<difference>
206+
<differenceType>7012</differenceType>
207+
<className>com/google/cloud/spanner/StructReader</className>
208+
<method>com.google.protobuf.AbstractMessage getProtoMessage(int, com.google.protobuf.AbstractMessage)</method>
209+
</difference>
210+
<difference>
211+
<differenceType>7012</differenceType>
212+
<className>com/google/cloud/spanner/StructReader</className>
213+
<method>com.google.protobuf.AbstractMessage getProtoMessage(java.lang.String, com.google.protobuf.AbstractMessage)</method>
214+
</difference>
215+
<difference>
216+
<differenceType>7012</differenceType>
217+
<className>com/google/cloud/spanner/StructReader</className>
218+
<method>java.util.List getProtoEnumList(int, java.util.function.Function)</method>
219+
</difference>
220+
<difference>
221+
<differenceType>7012</differenceType>
222+
<className>com/google/cloud/spanner/StructReader</className>
223+
<method>java.util.List getProtoEnumList(java.lang.String, java.util.function.Function)</method>
224+
</difference>
225+
<difference>
226+
<differenceType>7012</differenceType>
227+
<className>com/google/cloud/spanner/StructReader</className>
228+
<method>java.util.List getProtoMessageList(int, com.google.protobuf.AbstractMessage)</method>
229+
</difference>
230+
<difference>
231+
<differenceType>7012</differenceType>
232+
<className>com/google/cloud/spanner/StructReader</className>
233+
<method>java.util.List getProtoMessageList(java.lang.String, com.google.protobuf.AbstractMessage)</method>
234+
</difference>
195235
<difference>
196236
<differenceType>7012</differenceType>
197237
<className>com/google/cloud/spanner/BatchClient</className>
@@ -222,6 +262,38 @@
222262
<className>com/google/cloud/spanner/connection/Connection</className>
223263
<method>com.google.cloud.spanner.ResultSet analyzeUpdateStatement(com.google.cloud.spanner.Statement, com.google.cloud.spanner.ReadContext$QueryAnalyzeMode, com.google.cloud.spanner.Options$UpdateOption[])</method>
224264
</difference>
265+
<difference>
266+
<differenceType>7012</differenceType>
267+
<className>com/google/cloud/spanner/DatabaseAdminClient</className>
268+
<method>com.google.spanner.admin.database.v1.GetDatabaseDdlResponse getDatabaseDdlResponse(java.lang.String, java.lang.String)</method>
269+
</difference>
270+
<difference>
271+
<differenceType>7012</differenceType>
272+
<className>com/google/cloud/spanner/DatabaseAdminClient</className>
273+
<method>com.google.api.gax.longrunning.OperationFuture updateDatabaseDdl(com.google.cloud.spanner.Database, java.lang.Iterable, java.lang.String)</method>
274+
</difference>
275+
<difference>
276+
<differenceType>7013</differenceType>
277+
<className>com/google/cloud/spanner/DatabaseInfo$Builder</className>
278+
<method>com.google.cloud.spanner.DatabaseInfo$Builder setProtoDescriptors(byte[])</method>
279+
</difference>
280+
<difference>
281+
<differenceType>7013</differenceType>
282+
<className>com/google/cloud/spanner/DatabaseInfo$Builder</className>
283+
<method>com.google.cloud.spanner.DatabaseInfo$Builder setProtoDescriptors(java.io.InputStream)</method>
284+
</difference>
285+
<difference>
286+
<differenceType>7013</differenceType>
287+
<className>com/google/cloud/spanner/DatabaseInfo$Builder</className>
288+
<method>com.google.cloud.spanner.DatabaseInfo$Builder setProtoDescriptors(java.lang.String)</method>
289+
</difference>
290+
<difference>
291+
<differenceType>7006</differenceType>
292+
<className>com/google/cloud/spanner/spi/v1/GapicSpannerRpc</className>
293+
<method>java.util.List getDatabaseDdl(java.lang.String)</method>
294+
<from>java.util.List</from>
295+
<to>com.google.spanner.admin.database.v1.GetDatabaseDdlResponse</to>
296+
</difference>
225297
<difference>
226298
<differenceType>7004</differenceType>
227299
<className>com/google/cloud/spanner/spi/v1/GapicSpannerRpc</className>
@@ -277,6 +349,13 @@
277349
<method>com.google.spanner.v1.ResultSet executeQuery(com.google.spanner.v1.ExecuteSqlRequest, java.util.Map)</method>
278350
<to>com.google.cloud.spanner.spi.v1.SpannerRpc$StreamingCall</to>
279351
</difference>
352+
<difference>
353+
<differenceType>7006</differenceType>
354+
<className>com/google/cloud/spanner/spi/v1/SpannerRpc</className>
355+
<method>java.util.List getDatabaseDdl(java.lang.String)</method>
356+
<from>java.util.List</from>
357+
<to>com.google.spanner.admin.database.v1.GetDatabaseDdlResponse</to>
358+
</difference>
280359
<difference>
281360
<differenceType>7004</differenceType>
282361
<className>com/google/cloud/spanner/spi/v1/SpannerRpc</className>

‎google-cloud-spanner/pom.xml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,15 @@
134134
<groupId>org.codehaus.mojo</groupId>
135135
<artifactId>flatten-maven-plugin</artifactId>
136136
</plugin>
137+
138+
<plugin>
139+
<groupId>org.codehaus.mojo</groupId>
140+
<artifactId>clirr-maven-plugin</artifactId>
141+
<configuration>
142+
<!-- Exclude (public) classes in this internal package -->
143+
<excludes>com/google/cloud/spanner/spi/v1/**</excludes>
144+
</configuration>
145+
</plugin>
137146
</plugins>
138147
<pluginManagement>
139148
<plugins>

‎google-cloud-spanner/src/main/java/com/google/cloud/spanner/AbstractResultSet.java‎

Lines changed: 137 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,10 +39,13 @@
3939
import com.google.common.collect.AbstractIterator;
4040
import com.google.common.collect.ImmutableMap;
4141
import com.google.common.collect.Lists;
42+
import com.google.common.io.CharSource;
4243
import com.google.common.util.concurrent.Uninterruptibles;
44+
import com.google.protobuf.AbstractMessage;
4345
import com.google.protobuf.ByteString;
4446
import com.google.protobuf.ListValue;
4547
import com.google.protobuf.NullValue;
48+
import com.google.protobuf.ProtocolMessageEnum;
4649
import com.google.protobuf.Value.KindCase;
4750
import com.google.spanner.v1.PartialResultSet;
4851
import com.google.spanner.v1.ResultSetMetadata;
@@ -58,6 +61,7 @@
5861
import java.io.IOException;
5962
import java.io.Serializable;
6063
import java.math.BigDecimal;
64+
import java.nio.charset.StandardCharsets;
6165
import java.util.AbstractList;
6266
import java.util.ArrayList;
6367
import java.util.Base64;
@@ -73,6 +77,7 @@
7377
import java.util.concurrent.Executor;
7478
import java.util.concurrent.LinkedBlockingQueue;
7579
import java.util.concurrent.TimeUnit;
80+
import java.util.function.Function;
7681
import java.util.logging.Level;
7782
import java.util.logging.Logger;
7883
import java.util.stream.Collectors;
@@ -477,6 +482,14 @@ private Object writeReplace() {
477482
case JSON:
478483
builder.set(fieldName).to(Value.json((String) value));
479484
break;
485+
case PROTO:
486+
builder
487+
.set(fieldName)
488+
.to(Value.protoMessage((ByteArray) value, fieldType.getProtoTypeFqn()));
489+
break;
490+
case ENUM:
491+
builder.set(fieldName).to(Value.protoEnum((Long) value, fieldType.getProtoTypeFqn()));
492+
break;
480493
case PG_JSONB:
481494
builder.set(fieldName).to(Value.pgJsonb((String) value));
482495
break;
@@ -500,6 +513,7 @@ private Object writeReplace() {
500513
builder.set(fieldName).toBoolArray((Iterable<Boolean>) value);
501514
break;
502515
case INT64:
516+
case ENUM:
503517
builder.set(fieldName).toInt64Array((Iterable<Long>) value);
504518
break;
505519
case FLOAT64:
@@ -521,6 +535,7 @@ private Object writeReplace() {
521535
builder.set(fieldName).toPgJsonbArray((Iterable<String>) value);
522536
break;
523537
case BYTES:
538+
case PROTO:
524539
builder
525540
.set(fieldName)
526541
.toBytesArrayFromBase64(
@@ -596,6 +611,7 @@ private static Object decodeValue(Type fieldType, com.google.protobuf.Value prot
596611
checkType(fieldType, proto, KindCase.BOOL_VALUE);
597612
return proto.getBoolValue();
598613
case INT64:
614+
case ENUM:
599615
checkType(fieldType, proto, KindCase.STRING_VALUE);
600616
return Long.parseLong(proto.getStringValue());
601617
case FLOAT64:
@@ -610,6 +626,7 @@ private static Object decodeValue(Type fieldType, com.google.protobuf.Value prot
610626
checkType(fieldType, proto, KindCase.STRING_VALUE);
611627
return proto.getStringValue();
612628
case BYTES:
629+
case PROTO:
613630
checkType(fieldType, proto, KindCase.STRING_VALUE);
614631
return new LazyByteArray(proto.getStringValue());
615632
case TIMESTAMP:
@@ -649,7 +666,8 @@ private static Struct decodeStructValue(Type structType, ListValue structValue)
649666
static Object decodeArrayValue(Type elementType, ListValue listValue) {
650667
switch (elementType.getCode()) {
651668
case INT64:
652-
// For int64/float64 types, use custom containers. These avoid wrapper object
669+
case ENUM:
670+
// For int64/float64/enum types, use custom containers. These avoid wrapper object
653671
// creation for non-null arrays.
654672
return new Int64Array(listValue);
655673
case FLOAT64:
@@ -664,6 +682,7 @@ static Object decodeArrayValue(Type elementType, ListValue listValue) {
664682
case TIMESTAMP:
665683
case DATE:
666684
case STRUCT:
685+
case PROTO:
667686
return Lists.transform(
668687
listValue.getValuesList(), input -> decodeValue(elementType, input));
669688
default:
@@ -699,6 +718,35 @@ public boolean isNull(int columnIndex) {
699718
return rowData.get(columnIndex) == null;
700719
}
701720

721+
@Override
722+
protected <T extends AbstractMessage> T getProtoMessageInternal(int columnIndex, T message) {
723+
Preconditions.checkNotNull(
724+
message,
725+
"Proto message may not be null. Use MyProtoClass.getDefaultInstance() as a parameter value.");
726+
try {
727+
return (T)
728+
message
729+
.toBuilder()
730+
.mergeFrom(
731+
Base64.getDecoder()
732+
.wrap(
733+
CharSource.wrap(((LazyByteArray) rowData.get(columnIndex)).base64String)
734+
.asByteSource(StandardCharsets.UTF_8)
735+
.openStream()))
736+
.build();
737+
} catch (IOException ioException) {
738+
throw SpannerExceptionFactory.asSpannerException(ioException);
739+
}
740+
}
741+
742+
@Override
743+
protected <T extends ProtocolMessageEnum> T getProtoEnumInternal(
744+
int columnIndex, Function<Integer, ProtocolMessageEnum> method) {
745+
Preconditions.checkNotNull(
746+
method, "Method may not be null. Use 'MyProtoEnum::forNumber' as a parameter value.");
747+
return (T) method.apply((int) getLongInternal(columnIndex));
748+
}
749+
702750
@Override
703751
protected boolean getBooleanInternal(int columnIndex) {
704752
return (Boolean) rowData.get(columnIndex);
@@ -768,6 +816,8 @@ protected Value getValueInternal(int columnIndex) {
768816
return Value.bool(isNull ? null : getBooleanInternal(columnIndex));
769817
case INT64:
770818
return Value.int64(isNull ? null : getLongInternal(columnIndex));
819+
case ENUM:
820+
return Value.protoEnum(getLongInternal(columnIndex), columnType.getProtoTypeFqn());
771821
case NUMERIC:
772822
return Value.numeric(isNull ? null : getBigDecimalInternal(columnIndex));
773823
case PG_NUMERIC:
@@ -782,6 +832,8 @@ protected Value getValueInternal(int columnIndex) {
782832
return Value.pgJsonb(isNull ? null : getPgJsonbInternal(columnIndex));
783833
case BYTES:
784834
return Value.internalBytes(isNull ? null : getLazyBytesInternal(columnIndex));
835+
case PROTO:
836+
return Value.protoMessage(getBytesInternal(columnIndex), columnType.getProtoTypeFqn());
785837
case TIMESTAMP:
786838
return Value.timestamp(isNull ? null : getTimestampInternal(columnIndex));
787839
case DATE:
@@ -812,6 +864,12 @@ protected Value getValueInternal(int columnIndex) {
812864
return Value.pgJsonbArray(isNull ? null : getPgJsonbListInternal(columnIndex));
813865
case BYTES:
814866
return Value.bytesArray(isNull ? null : getBytesListInternal(columnIndex));
867+
case PROTO:
868+
return Value.protoMessageArray(
869+
isNull ? null : getBytesListInternal(columnIndex), elementType.getProtoTypeFqn());
870+
case ENUM:
871+
return Value.protoEnumArray(
872+
isNull ? null : getLongListInternal(columnIndex), elementType.getProtoTypeFqn());
815873
case TIMESTAMP:
816874
return Value.timestampArray(isNull ? null : getTimestampListInternal(columnIndex));
817875
case DATE:
@@ -891,6 +949,61 @@ protected List<String> getJsonListInternal(int columnIndex) {
891949
return Collections.unmodifiableList((List<String>) rowData.get(columnIndex));
892950
}
893951

952+
@Override
953+
@SuppressWarnings("unchecked") // We know ARRAY<PROTO> produces a List<ByteArray>.
954+
protected <T extends AbstractMessage> List<T> getProtoMessageListInternal(
955+
int columnIndex, T message) {
956+
Preconditions.checkNotNull(
957+
message,
958+
"Proto message may not be null. Use MyProtoClass.getDefaultInstance() as a parameter value.");
959+
960+
List<LazyByteArray> bytesArray = (List<LazyByteArray>) rowData.get(columnIndex);
961+
962+
try {
963+
List<T> protoMessagesList = new ArrayList<>(bytesArray.size());
964+
for (LazyByteArray protoMessageBytes : bytesArray) {
965+
if (protoMessageBytes == null) {
966+
protoMessagesList.add(null);
967+
} else {
968+
protoMessagesList.add(
969+
(T)
970+
message
971+
.toBuilder()
972+
.mergeFrom(
973+
Base64.getDecoder()
974+
.wrap(
975+
CharSource.wrap(protoMessageBytes.base64String)
976+
.asByteSource(StandardCharsets.UTF_8)
977+
.openStream()))
978+
.build());
979+
}
980+
}
981+
return protoMessagesList;
982+
} catch (IOException ioException) {
983+
throw SpannerExceptionFactory.asSpannerException(ioException);
984+
}
985+
}
986+
987+
@Override
988+
@SuppressWarnings("unchecked") // We know ARRAY<ENUM> produces a List<Long>.
989+
protected <T extends ProtocolMessageEnum> List<T> getProtoEnumListInternal(
990+
int columnIndex, Function<Integer, ProtocolMessageEnum> method) {
991+
Preconditions.checkNotNull(
992+
method, "Method may not be null. Use 'MyProtoEnum::forNumber' as a parameter value.");
993+
994+
List<Long> enumIntArray = (List<Long>) rowData.get(columnIndex);
995+
List<T> protoEnumList = new ArrayList<>(enumIntArray.size());
996+
for (Long enumIntValue : enumIntArray) {
997+
if (enumIntValue == null) {
998+
protoEnumList.add(null);
999+
} else {
1000+
protoEnumList.add((T) method.apply(enumIntValue.intValue()));
1001+
}
1002+
}
1003+
1004+
return protoEnumList;
1005+
}
1006+
8941007
@Override
8951008
@SuppressWarnings("unchecked") // We know ARRAY<JSONB> produces a List<String>.
8961009
protected List<String> getPgJsonbListInternal(int columnIndex) {
@@ -1489,6 +1602,17 @@ protected String getStringInternal(int columnIndex) {
14891602
return currRow().getStringInternal(columnIndex);
14901603
}
14911604

1605+
@Override
1606+
protected <T extends AbstractMessage> T getProtoMessageInternal(int columnIndex, T message) {
1607+
return currRow().getProtoMessageInternal(columnIndex, message);
1608+
}
1609+
1610+
@Override
1611+
protected <T extends ProtocolMessageEnum> T getProtoEnumInternal(
1612+
int columnIndex, Function<Integer, ProtocolMessageEnum> method) {
1613+
return currRow().getProtoEnumInternal(columnIndex, method);
1614+
}
1615+
14921616
@Override
14931617
protected String getJsonInternal(int columnIndex) {
14941618
return currRow().getJsonInternal(columnIndex);
@@ -1574,6 +1698,18 @@ protected List<ByteArray> getBytesListInternal(int columnIndex) {
15741698
return currRow().getBytesListInternal(columnIndex);
15751699
}
15761700

1701+
@Override
1702+
protected <T extends AbstractMessage> List<T> getProtoMessageListInternal(
1703+
int columnIndex, T message) {
1704+
return currRow().getProtoMessageListInternal(columnIndex, message);
1705+
}
1706+
1707+
@Override
1708+
protected <T extends ProtocolMessageEnum> List<T> getProtoEnumListInternal(
1709+
int columnIndex, Function<Integer, ProtocolMessageEnum> method) {
1710+
return currRow().getProtoEnumListInternal(columnIndex, method);
1711+
}
1712+
15771713
@Override
15781714
protected List<Timestamp> getTimestampListInternal(int columnIndex) {
15791715
return currRow().getTimestampListInternal(columnIndex);

0 commit comments

Comments
 (0)