From this I derive that we can inject malicious/faulty behavior, but this behavior should not in principle halt or produce irrecoverable errors in the chain?

I've thought about this for a few minutes and I'm not sure how to add it to the document.

If you knew that running a particular command did something strange on a 3-node testnet, and you were curious what effect it had on a 100-node testnet: I think it would be okay to run that against the main testnet, even with an uncertain risk that it would halt the testnet.

But if you were certain that you could halt the main testnet with a single command, then I would recommend that you would not run it -- unless perhaps you were going to confirm that the next software update was capable of recovering from said halt.

This is a kind of fuzzy area that I think is fine to omit from the document for now.

Is the notion that the testnet will run continuously, with each deploy simply re-installing the tendermint code? Or would the goal be to create a fresh Tendermint instance - new DBs, new configs, etc - on every deploy?

If we keep the testnet running continuously, that will somewhat force us to ensure that intermediate versions of unreleased software are compatible with each other. I think that that burden may be a bit higher than we'd like to bear but I'm open to being convinced otherwise.

I clarified in f6d33c4 that the testnet is intended to be run continuously, but engineers should be very empowered to destroy and recreate the testnet if they think handling an unreleased breaking change is not worth the time.

IMO, if it's a couple-line commit to make an internally breaking change forwards-compatible, and it's five minutes to write that code, I would hope the engineer would take that approach. If it would take at least an hour to write code to handle an unreleased breaking change, I would have no issue with the decision to wipe out the old testnet and recreate it.

KV store seems a-ok to me!

I like how small it is to start. Get our feet wet while we build the muscle.

I would contend that using kube, in general, should be a non-goal of any such project. Our users almost exclusively do not use it from what I've observed. They run the binary on VMs and we should attempt to mimic what they do as much as possible.

I've heard that the Cosmos ecosystem is a decent mix where the "smaller" validators tend to run directly on VMs, and the "larger" ones tend to run in containerized environments. But I don't have any hard data to back that up.

If that is accurate, I think there is some value in the long term in running Tendermint nodes in containers.

In either case, I am comfortable beginning without any plans for containerized nodes.

Mind defining a few 'observability' goals in this section as well? Basic prom / node-exporter is likely sufficient?

In f6d33c4 I clarified that MVP has no observability goals other than alerting on a chain halt or unexpected process exits, but I did add some detail to the medium-term goals around observability. Prometheus and node-exporter should get us pretty far, but I think it is important that the application does publish some meaningful metrics too.

I don't expect Tendermint to survive FS corruption so I'm not sure this should exist as a medium-term goal.

I like both of these a lot, but they are basically their own projects. If we're going to do this, I think we should spend a bit of time on these dotting our i's and crossing our t's. A MVP of these dashboards will be probably a bit sloppy and ramping up to something releasable will take some polishing. My concern, ultimately, is around how we position any such stack/dashboard. If we communicate that this is the Tendermint stack, we may come to own its use in the wild in a way we don't intend. I think we'd need to make sure to position any such resources as 'things that we use but are very much on the user to tinker with'.

Like this idea a lot.

I think the prometheus node exporter can just get us this? Maybe there's more we need tho? If not, we can hit this goal pretty quickly.

@joeabbey was running an experiment using a fork of some chain, where he was using the Go runtime to collect a profile for some particular ABCI step on every block. I could see providing that as some kind of opt-in configuration, or maybe it would be a configured sample rate, in the eventual custom application for the testnet.

Whether that is something we actually need, should become more clear as we approach the long term.

I'm not sure where, but I think we should endeavor to not do any experimentation on these networks. As soon as we introduce config-skew into the networks, we will reduce confidence in the results of running the testnets considerably.

"Wait, was that the node that you changed the max connections on? Maybe it's nothing" etc.

f6d33c4 clarifies that it is okay to interact with the blockchain, but configuration should not be modified.