CacheFilter doesn't yet notice 'vary' response headers. At a minimum, it should refuse to cache such responses. For full functionality, it should enforce CacheConfig.allowed_vary_headers during insertion, and copy listed headers to LookupRequest::vary_headers_ during lookup.
Doing at least the minimum is a blocker for deployment.