UpdateOpts struct for the listeners contains:

SniContainerRefs []string `json:"sni_container_refs,omitempty"`

Which means that SniContainerRefs cannot be set to an empty list.

I suggest to add the:

SniContainerRefs *[]string `json:"sni_container_refs,omitempty"`

UPD: the same has to be done for the DefaultTlsContainerRef and AllowedCIDRs.

UPD2: If the protocol is TERMINATED_HTTPS, then DefaultTlsContainerRef is required and cannot be unset.