-
Notifications
You must be signed in to change notification settings - Fork 5.3k
Description
It appears envoy hashes protobuf config structures to determine whether a config has changed. This is a problem since configs are placed into typed_configs as Any, and protobuf specifically does not guarantee deterministic serialization. The main non-determinism comes from map ordering.
If the control plane repeatedly serializes and pushes same configs without change, then envoy might incorrectly assume a change, and drain/reload listeners causing a significant CPU spike and drained connections.
For golang, the solution is to force the control plane to serialize maps in a stable manner. C++ has a global SetDefaultSerializationDeterministic. I assume the same applies to python and java, but that needs to be checked with protobuf libraries for those languages.