admin: limit number of bytes an admin point can emit #31082
Description
Title: admin endpoints may yield more bytes than browsers can hold in memory, and it should be possible (and maybe default behavior) to limit that.
Description:
See related issue #16139 which is for /stats/prometheus in particular. But the same issue is present for other admin endpoints.
There are several approaches to this, which may need to be combined in some cases.
- don't buffer up data in the server but flush it out to the admin http stream periodically. this is done for /stats but not for other endpoints that need it. This addresses server memory for the most part. It is also not done yet for /stats/prometheus though. Prom stats perf improvements #24998 attempts to fix; it proved harder than expected.
- use high/low watermark hooks to pause generating and streaming more content until the client has read it. I implemented this but failed to write a test proving it worked: admin: apply flow control via high/low watermarks #31087
- use a paginated API to only provide explicitly asked for data. I implemented this in stats but did not fully develop it. (WiP admin: Add paging UI to admin stats page. #19413)
- use a hierarchical API to segment the data being asked for. I implemented this for stats also, including a UI with an outline-type view like a file-explorer, and ajax requests to fill in scopes as you expand them. This worked really nicely but I never got the PR ready for review. (RFC admin: hierarchical stats viewer #18670)
- Simply limit the default number of bytes any admin endpoint can emit, and provide a query-param to override the default. The risk is that if the output format is JSON then truncating it would break it. However, https://github.com/envoyproxy/envoy/blob/main/source/common/json/json_streamer.h could easily be enhanced to give it a maximum byte count, after which it will auto-close all hierarchy and suppress further streaming. That API is used only for /stats?format=json (it's significantly faster than populating a protobuf and serializing) but it could be used for other endpoints to safely limit output.