Note: This top comment is heavily edited. It was kept up with the current state until this issue closed.

Motivation

Flatbuffers should allow users to choose to use optional types. There has been some interest in distinguishing between default values (which are not stored in the binary) and some notion of None which the user controls.

Here are some links to previous interest in this idea:

• Return null for missing optional field? #333
• check which fields are filled out and which aren't in C++/Java #3777
• https://groups.google.com/forum/#!topic/flatbuffers/1hrNtBI0BQI
• What would a "FlatBuffers2" binary format look like? #5875 (comment)

Currently, a user can control a field's presence in the binary by specifying "force_defaults" and checking "IsFieldPresent" which is a bit of a hack. This proposal should define proper Flatbuffers optional types, which should be a better way of doing this. Use of this feature is only advisable for new fields, since changing default values is in general backwards-incompatible.

How do we represent this in the schema file?

We will specify it like so

table Monster { mana: int = null; }

This visually implies that optional types are at odds with default values and is "consistent" since the value to the right of the equals sign is what we interpret non-presence to mean.

Change to Schema Grammar:

field_decl = ident : type [ = (scalar | null) ] metadata ;

We can add a field tag, e.g. "optional" or "no_default", that triggers this behavior. Hopefully no one is using those tags. Maybe we can make it specifiable to flatc, an "--optional-field-keyword-tag" flag, just in case people are using it and can't stop.

How do we represent this in the binary?

We are going with option (A).

(A) Non-Presence means None

Instead of omitting zero-like values, the generated code must store them. Non-presence for optional fields no longer means "whatever the default is," now it means None. You can interpret it as "the default value is None". This also means we cannot specify both specify a non-null default and mark the field as optional.

Pros:

• This seems more intuitive.
• It aligns with the "force_defaults" + "IsFieldPresent" hacky manual approximation of this feature.
• If Nones are more common than zero-likes then this will have smaller binaries.

Cons:

• @aardappel thinks this is harder to implement

  "making presence an indicator would require we pass this special field status down to the field construction code to override the current val == default check, which means slowdown, codegen and runtime changes in all languages.. whereas my "least likely to be used default" trick requires no changes"

(B) Some Sentinel value means None

In this scenario, zero-like values are still not stored. Instead we choose some "sentinel" value which we interpret to be None (e.g. int can use int_min and float can use some kind of Nan).

Pros:

• @aardappel thinks this is easier to implement

  "it requires the schema parser to set default values for you, and no changes anywhere else"

• If zero-likes are more common than None then this will have smaller binaries

Cons:

• Someone might want to use the sentinel value (Hyrum's law).
  • This can be mitigated by publishing the sentinels and letting users decide whether they need the sentinels.
• This probably won't work for fields representing raw bits.

How do we represent this in every language API?

We'll need to change the type signature of all generated code (building/reading/mutating/object/etc) around the optional type to signal its optional-ness. I think we should use the language's local standard for optional types. Suggestions:

• Python: Optional[T].
• Rust: Option<T>.
• C++17 has std::optional<T> but its not obvious what to use for earlier versions. T* would work.
• Java: Optional shows up in Java 1.8 and triggers autoboxing, so idk :/

The exact generated-API for a language should be discussed in the PR implementing this feature in that language.

Out of scope

(I'll add links if you make issues for these feature requests)

• Syntactic types
• Default values for strings and tables

TODO

[edits]

• added todo list
• added points from the discussion to each section.
• added out of scope section
• Decision: go with (A) and use = null syntax in schema file (cross out alternatives)
• Updated TODO list, finished parser, Rust in progress
• Change to schema grammar, link to swift PR, Note at top
• Added more languages to the TODO
• Lobster support 🦞
• Kotlin and C support
• Java, C#, TS/JS support and docs, issue closed, no longer editing.