https://issues.apache.org/jira/browse/PARQUET-129
@rdblue

This is a suggestion only at the moment and not complete. Need help to complete the feature - see below. Should not be much work for a more experienced person, I hope.
I have added a Patient.asvc schema into the resource folder as a complex example.

Note:

• The code has one TODO.
• And it needs the matching changes to actually write the records into the correct fields.
• The reader is not impacted!! Just the writing!
• No backward compatibility issue as currently such circular schema are not possible and for non-circular schema nothing does change
• No issue with Spark etc as the reader would return regular fields still, and nothing Spark cannot handle.

I tried to be as conservative as possible to not break something, however had to change GroupType and MessageType classes to allow creating them first and adding fields second. That makes sense anyhow, not always you have the complete list of fields at the point in time this object is instantiated.

Just to define the term "circular references" correctly, we have the following scenarios:

1. SchemaN is defined in one element and reused in a parallel field, e.g. there are two fields, buildingaddress and postaladdress. The first defines the recordtype ADDRESS, the second is using it. --> no problem.
2. SchemaN has a field that is of type SchemaN --> a direct circular reference causing a stack overflow
3. SchemaN has a field that has a field that.... has a field of type SchemaN --> an indirect circular reference causing a stack overflow

The way to resolve that is to check before creating a new GroupType for SchemaN, if such a Schema had been created already in one of the parents. It should trigger only for case 2&3, not case 1 above.
If such a thing happens, then instead of creating the full GroupType for this field, it is rather an INT64 pointer. And the SchemaN definiton should get an additional INT64 field holding the reference pointer.

So you will end up with rows coming from the first occurrence of that schema, all having that INT64 empty, plus rows from the child columns of type SchemaN with the INT64 being set.

Example for Case 2:
Person: {firstname STRING, lastname String, [children: Person]}
would be turned into
Person: {firstname STRING, lastname String, [children: INT64], __ID}

My example Patient.asvc AvroSchema is converted into a nice Parquet schema, that is working.
What is missing:

1. SchemaN might not be an array in the initial definition but just a single nested structure. Then we cannot add more records to it, hence it has to be turned into a REPEATABLE schema. I am not sure if I understand all about the Parquet model, hence would like you to do that.
2. The entire data writing has to be implemented.

The reason I have not implemented the writer is for the following questions:

• That __ID column, can we reuse an internal pointer for that? Something that will make joins faster by jumping to the record directly instead of scanning the column? Not important for Spark but might be interesting later.
• When creating the Parquet schema, can we add properties to tell the writer where to add the record? Currently that can be derived from the AvroSchema but requires the tree-building in the writer a second time. Would like to avoid that.