Skip to content

Conversation

@mhils
Copy link
Member

@mhils mhils commented Mar 28, 2025

Description

This PR introduces a new contentview API that makes it possible for users to edit prettified views. This is particular useful for binary formats such as protobuf, which are otherwise hard to tinker with.

Feedback from everyone is welcome! For practical purposes it's useful to only look at the changes in CHANGELOG.md, docs/ and examples/.

Major changes

  • Contentviews can now be interactive and re-encode prettified data.
    For example, the new Protobuf view pretty-prints to YAML, which the user
    can edit and then re-serialize into binary representation.
  • Replace the existing gRPC and Protobuf contentviews with an interactive contentview that
    supports both existing proto definitions and completely unknown protos.
  • The MsgPack contentview is now interactive, too.
  • The contentview API has been drastically simplified.
    Contentviews now return a plain str with the prettified data.
    Syntax highlighting is now signaled off-band (and based on [tree-sitter]).
  • Docs: Add new documentation page and API reference for contentviews.
  • Contentviews can now be written in Rust for better performance and access to
    the crates ecosystem.

Regressions:

  • mitmweb does not the new syntax highlighting yet. The plan is to move to a new editor component altogether.

Known gaps:

  • Support for editing contentviews in mitmweb is still missing.
  • The new Protobuf contentview does not support hot-reloading for proto files.

Related Issues

@mhils mhils mentioned this pull request Mar 28, 2025
15 tasks
@mhils mhils force-pushed the contentviews-api branch 2 times, most recently from 9f6a4d1 to 1b82068 Compare March 29, 2025 01:02
Copy link
Member

@errorxyz errorxyz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great stuff 🚀

@mhils mhils force-pushed the contentviews-api branch from 698ca1b to 9357af7 Compare March 30, 2025 14:19
Copy link
Member

@emanuele-em emanuele-em left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯

@mhils mhils force-pushed the contentviews-api branch 3 times, most recently from 8dc7c71 to 0be29d3 Compare April 17, 2025 09:37
@mhils mhils force-pushed the contentviews-api branch from 0be29d3 to 0369a07 Compare April 17, 2025 09:41
@mhils mhils marked this pull request as ready for review April 17, 2025 10:17
@mhils mhils force-pushed the contentviews-api branch from 5941f11 to 5295e92 Compare April 17, 2025 12:00
@mhils mhils requested review from Prinzhorn and emanuele-em April 17, 2025 12:03
@mhils
Copy link
Member Author

mhils commented Apr 17, 2025

This should be good to go! 🚀

I understand that it's incredibly hard to review, but I'd really appreciate a second look over docs/src/content/addons/contentviews.md and examples/. 😃

@bburky
Copy link
Contributor

bburky commented Apr 17, 2025

For my SAML use case, I wanted my prettify to do things like autoformatting XML text with indentation, which makes it significantly more readable.

However, for editing, I actually wanted to operate on the maybe-deflated and base64-decoded data, because my prettify would break XML signatures.

  1. Decode
  2. Prettify for UI visualization only
  3. edit (on the output of Decode) and re-encode when complete

Is this setup possible with this new feature?

@mhils
Copy link
Member Author

mhils commented Apr 17, 2025

Yes. Here's an example for gRPC/Protobuf: https://asciinema.org/a/MivOWmlDxV3juRNvjhxFU7FH8

With the new API, you can implement:

  1. prettify(data: bytes, metadata: Metadata) -> str to implement autoformatting/indentation, and
  2. reencode(prettified: str, metadata: Metadata) -> bytes to implement reencoding + signature calculation.

Oh, and you specify syntax_highlighting = "xml" to get syntax highlighting for your use case. :)

@bburky
Copy link
Contributor

bburky commented Apr 17, 2025

Not just syntax highlighting. Adding whitespace and things, which is actually destructive with XML signatures. Any other similar destructive feature is actually nice to do in prettify for better visualization. But, I want an edit feature to run on un-prettifyed text, because prettify may have destroyed data. But I still do want to have decoding (deflate and base64) performed even for edit.

Something like:

  • `decode(data: bytes, metadata: Metadata) -> decode binary format for editing only
  • prettify(data: bytes, metadata: Metadata) -> str to implement autoformatting/indentation
  • reencode(decoded: bytes, metadata: Metadata) -> bytes to implement reencoding. Note: operates on edited decoded bytes, not prettified str

What you have is great for formats that roundtrip through prettify without losing data. Maybe it isn't meant to map exactly to my use case. I guess I could still use this new API by having two contentviews: one for editing that does non-destructive things, and a second that's better for visualization and does prettify aggressively.

@mhils
Copy link
Member Author

mhils commented Apr 17, 2025

But, I want an edit feature to run on un-prettifyed text, because prettify may have destroyed data.

I think this sounds like a very specific use case that may not warrant complicating the API. As you mention, a simple workaround would be to have two separate contentviews here. But more fundamentally, the reencoder could also undo the whitespace changes. That probably not super easy to implement... but feels very much possible. 😃 Let's keep this complexity in the individual contentview, and not in the API that's shared by all contentviews.

For context, while MsgPack reencoding is laughably simple (msgpack.rs#L35-L46), Protobuf is a convoluted dance. I think it's expected that some contentviews are a bit more complex than others.

Copy link
Member

@lups2000 lups2000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 Really nice!

@mhils mhils merged commit d0a35dc into mitmproxy:main Apr 21, 2025
26 of 27 checks passed
@sujaldev sujaldev mentioned this pull request Oct 15, 2025
1 task
maelvls added a commit to maelvls/kubectl-incluster that referenced this pull request Dec 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

6 participants