A test suite is available at https://github.com/BenoitHanotte/parquet-968 . It shows the different interpretation of the written schemas with Spark:

1. old behavior (default):

+-------------+----------------+--------+----------------+
|emptyRepeated|nonEmptyRepeated|emptyMap|     nonEmptyMap|
+-------------+----------------+--------+----------------+
|           []|          [1, 1]|      []|[[1, 1], [2, 2]]|
+-------------+----------------+--------+----------------+

2. new behavior (specs-compliant)

+-------------+----------------+--------+----------------+
|emptyRepeated|nonEmptyRepeated|emptyMap|     nonEmptyMap|
+-------------+----------------+--------+----------------+
|         null|          [1, 1]|    null|[1 -> 1, 2 -> 2]|
+-------------+----------------+--------+----------------+

Awesome!
Will have a look today

Looks good to me.

@BenoitHanotte, I am trying to write a local parquet file with your patch, but using a ProtoParquetWriter. I can't seem to be able to pass the writeSpecsCompliant flag to this class though.

ProtoParquetWriter<MessageOrBuilder> writer = 
   new ProtoParquetWriter<MessageOrBuilder>(file, cls, CompressionCodecName.GZIP, 256 * 1024 * 1024, 1 * 1024 * 1024);

Should we add some extra constructors there in the ProtoParquetWriter.java to be able to pass this flag? (https://github.com/BenoitHanotte/parquet-mr/blob/master/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoParquetWriter.java#L47)

I see we also have a unit test that uses this class: https://github.com/BenoitHanotte/parquet-mr/blob/master/parquet-protobuf/src/test/java/org/apache/parquet/proto/TestUtils.java#L177

@costimuraru Yes we may need to add the flag to the ProtoParquetWriter constructor to allow using it directly.

What do you think about doing it in a new PR? This PR has been thoroughly reviewed already, and I would like to avoid needing a new pass on it as it could take some time.
In the current state, the project is fully functional, and you can manually create your ParquetWriter by providing the ProtoWriteSupport as following:

Configuration conf = new Configuration();
ProtoWriteSupport.setWriteSpecsCompliant(true, conf); // set the flag in the configuration
new ParquetWriter(file, conf, new ProtoWriteSupport(protoClass));

(or any variation of this as ParquetWriter has multiple constructors that accept a configuration in which we can set the flag)

Also, it seems that the only non-deprecated ParquetWriter constructor is one that accepts a Configuration as a parameter, thus I propose we do the following:

1. Add a new constructor to ProtoParquetWrtier that accepts the flag for pure convenience. The flag is then set in a configuration object (as in the example above) and we call a ParquetWriter constructor that takes the configuration as parameter.
2. In a second step, it would be a good idea to refactor the ProtoParquetWriter class as it currently calls deprecated constructors from ParquetWriter (that would need to be in another PR)

What do you think?

Sounds good @BenoitHanotte !
I was able to create the parquet file by creating a ParquetWriter (instead of a ProtoParquetWriter):

Configuration conf = new Configuration();
ProtoWriteSupport.setWriteSpecsCompliant(conf, true);

ParquetWriter<MyMessage> writer 
   = new ParquetWriter<>(file, new ProtoWriteSupport<MyMessage>(cls), CompressionCodecName.GZIP, 256 * 1024 * 1024, 1 * 1024 * 1024,  1 * 1024 * 1024, true, false, ParquetWriter.DEFAULT_WRITER_VERSION, conf);

I was then able to take the parquet file, upload it to AWS S3, and query the file using AWS Athena (which uses Presto). Queries worked for our complex schema (with 90+ columns, lists, maps and inner fields). All seems to be good!

Perfect, thanks :)