https://blog.wakayos.com/2023-03-19T00:00:00-04:00Developer, Devops Guy in a Govtech World2023-03-19T00:00:00-04:002023-03-19T00:00:00-04:00Matthew Martintag:blog.wakayos.com,2023-03-19:/python-markpickle-2023.html

<p>I wrote a library that serializes and deserializes markdown to and from python, but more in the style of pickle</p>

<p>and less in the style of a document object model.</p> <h1>Hello markpickle</h1> <p>I got the name from ChatGPT.</p> <h2>Origin Story</h2> <p>I wanted an API to support as many mime types as possible. Some mime types are for the people, such as PDF, Excel/CSV. Some time types are for the machines, like json, bson.</p> <p>I thought, what about HTML and text? HTML has a problem with XSS risks. Text has a problem of not having much of a standard story for "styled" plaintext. Except for Markdown. Can I use markdown as a mime type for what starts out as a JSON object?</p> <h2>Challenges</h2> <p>The app is two functions, loads and dumps and they take or return an <code>Any</code> type and take or return a string. The <code>Any</code> is a problem, because mypy doesn't know how to deal with arbitrarily complex types. It seems to get confused with <code>Union</code>s of <code>Union</code>s of <code>Union</code>s. It gets confused with generics, e.g. lists that can work with any contents.</p> <h2>Impedance Mismatches</h2> <p>Markdown has some analogies:</p> <ul> <li>header groupings + paragraphs make dicts, or even nested dicts</li> <li>Lists or lists of lists</li> <li>tables are dicts, but you can only put scalars in the values. This makes python objects of a certain level of nesting unrepresentable.</li> <li>strings and non-string types can be represented, but you can only infer a number should be integer. This blocks round tripping.</li> </ul> <p>Some things are impossible</p> <ul> <li>empty dictionaries, empty lists exist in python but have no representation in markdown. An empty list is just zero-length whitespace where the list is not. This blocks round tripping.</li> </ul> <p>Python has no use for a lot of markdown formatting, such as bold, or code block or quote. Those things would just be strings and they'd continue to hold markdown in the string.</p>