I don't think these changes are correct...

@erikzhang can you be more explicit what part you think is incorrect about it?

1. This PR makes is so that all functions in all scenarios allow persisting to the real backend (as suggested in DataCache functions have inconsistent behaviour #1186).
2. The keyCount values are different because TryGet() no longer produces NONE state trackables and thus in some scenarios we have an additional trackable returned from GetChangeSet()
3. Because of point 2 the implementations of GetOrAdd() and GetAndChange() can be simplified by reusing existing code.

Let's say we have a contract doing the following steps:

TryGet(key1)
TryGet(key2)
......
TryGet(key100)
GetAndChange(somekey)

It reads 100 entries, and update 1 entry. If we allow both TryGet() and GetAndChange() to persist data, we need to write to disk 101 times on commit. So we need TryGet() to keep entries being TrackState.None so that we write to disk only once on commit.

I agree it will attempt to modify 101 entries (assuming LevelDB doesn't do anything smart if it sees no changes). However, we need to keep in mind that everything is put into a WriteBatch and disk I/O happens only once when calling Commit() on the snapshot, not 101 times.

My fear is that there is a security issue lurking around the corner because some people believe TryGet() is always read only. Example (#855 (comment)):

Because anything that you do with TryGet never will be committed, with GetAndChange, your changes will be applied

and this is not true as shown in this table (case 1 + 6)

and the code example here: #855 (comment)

What about if we use a different dictionaries for caching TryGet (readonly) and the others?

Option B, if it's already in the cache, return a copy when use TryGet

Option B, a copy would streamline the behaviour in all situations.

Alternatively, an optional read_only parameter could be added. Then make TryGet results persistable by default, which is also how C# dictionaries work (for TryGetValue), and which also allows to re-use and simplify the code following this PR.