The current approach for displaying and streaming logs to the client is grossly inefficient. This is especially important to me, personally, since some days I see 500MB of TX data from my home server. Way too high for just streaming logs.
Unnecessary logs sent on initial load
When the client requests the status page the server gathers limit number of logs, for each bot/subreddit that is accessible to the user, and returns all of them in the initial page load. This increases page load time and is a waste of sent data since the user is only ever viewing one set of logs at load.
Solution(?): Do not send any logs other than bot/subreddit being viewed on initial load. Instead load backlog of logs on first visit to bot/subreddit tab.
Simplify stream on-load logic
Currently using a complex middle man of client websocket session to recognize/authenticate what can be streamed, then having the CM client stream from CM server and relay this back to browser. Additionally, this streams all accessible logs even if they are never viewed -- lots of wasted data.
Instead, should investigate using already-built proxy endpoint on client to request streamed logs directly from server. The proxy already passes an api key with user info so authentication, which is already done by server, should be enough. The downside would be a slight delay before log streaming begins when a new tab is switched to but, IMO, this is an acceptable drawback to drastically reducing streaming complexity and reducing bandwidth.
Superfluous data in payload
Currently sending the entire log object, the "transformed" log string message, and (for web) the HTML formatted string in the payload. The transformed and html messages can and should be derived as needed by including transform functions in the web client javascript, rather than sending all data.
Data is uncompressed
Sending the data from server to client, and client to browser as normal, stringified json. Bandwidth could be reduced by 50-70% by using zlib or zstd compression at one or both stages of transmission.
Nice to have: This could be configurable between each stage so that mobile or LAN users could turn off compression for better performance. (Does this make a big difference though??)
The current approach for displaying and streaming logs to the client is grossly inefficient. This is especially important to me, personally, since some days I see 500MB of TX data from my home server. Way too high for just streaming logs.
Unnecessary logs sent on initial load
When the client requests the
statuspage the server gatherslimitnumber of logs, for each bot/subreddit that is accessible to the user, and returns all of them in the initial page load. This increases page load time and is a waste of sent data since the user is only ever viewing one set of logs at load.Solution(?): Do not send any logs other than bot/subreddit being viewed on initial load. Instead load backlog of logs on first visit to bot/subreddit tab.
Simplify stream on-load logic
Currently using a complex middle man of client websocket session to recognize/authenticate what can be streamed, then having the CM client stream from CM server and relay this back to browser. Additionally, this streams all accessible logs even if they are never viewed -- lots of wasted data.
Instead, should investigate using already-built proxy endpoint on client to request streamed logs directly from server. The proxy already passes an api key with user info so authentication, which is already done by server, should be enough. The downside would be a slight delay before log streaming begins when a new tab is switched to but, IMO, this is an acceptable drawback to drastically reducing streaming complexity and reducing bandwidth.
Superfluous data in payload
Currently sending the entire log object, the "transformed" log string message, and (for web) the HTML formatted string in the payload. The transformed and html messages can and should be derived as needed by including transform functions in the web client javascript, rather than sending all data.
Data is uncompressed
Sending the data from server to client, and client to browser as normal, stringified json. Bandwidth could be reduced by 50-70% by using zlib or zstd compression at one or both stages of transmission.
Nice to have: This could be configurable between each stage so that mobile or LAN users could turn off compression for better performance. (Does this make a big difference though??)