From a side conversation with @htuch I think that with a combination of:

1. https://github.com/abseil/abseil-cpp/blob/master/absl/types/variant.h
2. https://github.com/abseil/abseil-cpp/blob/master/absl/container/inlined_vector.h

We can almost identically replicate the current behavior with little/no performance delta, and in the process remove all of the manual memory management. I don't think this would be very difficult.