Still, don't understand why TXs are bigger in old version cause I did not change random.sh params!

I've tried running the build.sh script both locally on my Mac and inside VirtualBox (Linux)

Well technically I think you did, at https://github.com/tendermint/tendermint/pull/672/files/e874c021dc21d536302e23f3201aa71bf1584fff#diff-d7715a8a80c6b17d47e8a27ed32d059bL72

And also it makes sense that you have an 8X reduction in size

package main

import (
	"log"

	"github.com/tendermint/go-wire"
	"github.com/tendermint/tendermint/consensus"
)

var data = map[string]string{
	"old": `{"Height":1,"Round":0,"Part":{"index":0,"bytes":"0101010F74656E6465726D696E745F74657374010114B96164A30A118001620000000001141F6753D22BACA2180B1EADD722434EB28444D91D0114354594CBFC1A7BCA1AD0050ED6AA010023EADA3900010162011A3631363236333634333133303344363436333632363133313330011A3631363236333634333133313344363436333632363133313331011A3631363236333634333133323344363436333632363133313332011A3631363236333634333133333344363436333632363133313333011A3631363236333634333133343344363436333632363133313334011A3631363236333634333133353344363436333632363133313335011A3631363236333634333133363344363436333632363133313336011A3631363236333634333133373344363436333632363133313337011A3631363236333634333133383344363436333632363133313338011A3631363236333634333133393344363436333632363133313339011A3631363236333634333233303344363436333632363133323330011A3631363236333634333233313344363436333632363133323331011A3631363236333634333233323344363436333632363133323332011A3631363236333634333233333344363436333632363133323333011A3631363236333634333233343344363436333632363133323334011A36313632363336","proof":{"aunts":["49F4B71E3D7C457415069E2EA916DB12F67AA8D0","D35A72BEDAAAAC17045D7BFAAFA94C2EC0B0A4C2","705BC647374F3495EE73C3F44C21E9BDB4731738"]}}}`,
	"new": `{"Height":1,"Round":0,"Part":{"index":0,"bytes":"0101010F74656E6465726D696E745F74657374010114E67C5AEDA6288000000000000001147297262C6CD96190E46846C9A0DE1227E76077CF00010001000000","proof":{"aunts":["C81B94933420221A7AC004A90242D8B1D3E5070D"]}}}`,
}

func main() {
	for name, datum := range data {
		var err error
		part := wire.ReadJSON(&consensus.BlockPartMessage{}, []byte(datum), &err).(*consensus.BlockPartMessage)
		if err != nil {
			log.Printf("%s: decoding: %v", name, err)
			continue
		}
		log.Printf("%s: len(part.Bytes): %d\n", name, len(part.Part.Bytes))
	}
}

giving

 go run main.go 
2017/09/25 17:53:41 old: len(part.Bytes): 512
2017/09/25 17:53:41 new: len(part.Bytes): 64

This setting only changes part_size for small_block2 case. I was talking more about all cases, not only this one.

Minor nit: if this function is in a hot path, we can declare as a global var crc32c = crc32.MakeTable(crc32.Castagnoli)

nice idea

I see similar code duplicated from consensus/replay.go, perhaps we can extract it to a common helper

var crc32CastagnoliTable = crc32.MakeTable(crc32.Castagnoli)
func crc32Castagnoli(msg interface{}) (msgBytes []byte, crc uint32) {
   msgBytes = wire.JSONBytes(msg)
   crc = crc32.Checksum(msgBytes, crc32CastagnoliTable)
   return msgBytes, crc
}

for 2 lines. don't think it's worth doing

I think we probably shouldn't use JSON here. It's kind of convenient for glancing at the wal, but for the sake of speed we should use a binary format. We can have a tool to map the binary lines to json for human readability.

Also, in that setting, I think we want the message size immediately after the checksum, ie. as Kyle put it "One simple approach to writing recoverable WALs is to write a fixed-size checksum, fixed-size length header, then variable-length record."

By here you mean the whole WAL? or just this intermediate encoding?

yeh both. certainly dont really see any utility to the outer encoding being json if the inner encoding is binary, so might as well do it all binary. Do you have a different opinion?

No, it is just that we will lose readability. I guess if we will provide a wal2json-viewer script, which outputs WAL as json, it should be fine.

doh, all tmlibs/autofile/group code is about strings (separated with "\n") including searching for "#ENDHEIGHT". I am rewriting it right now

"You can open a Group to keep restrictions on an AutoFile, like
the maximum size of each chunk, and/or the total amount of bytes
stored in the group."

interesting. so now it is possible to have one part of an msg in file X and the rest of it in file Y.

That doesn't sound good. Are you saying AutoFile won't work with binary format?

No, I will work, but autofile/group won't. There is no Write([]byte) method to start with. And search functionality will be broken once we start using binary format.

eh ?

-e ?

it does not work on MacOS without -e

block_part_size is now in the genesis file, not the config, so we have to edit that!

lets try not to reuse import names

i think I figured it out - you need to install gnu-sed. eg. brew install gnu-sed --with-default-names ... that should make your sed point to usr/local/bin/sed which is actually gnu-sed.

We should document this better, but eg. https://www.topbug.net/blog/2013/04/14/install-and-use-gnu-command-line-tools-in-mac-os-x/

Cool! Thanks

Actually, the script does not contain sed anymore, so we can remove this line safely. But good to know anyway!

512

what happens when this sequence shows up in a block ?

If we're using our old group#Search method or gr.ReadLine, we would definitely get bad results. Imagine for some reason we failed to write a separator "\n" or it's lost due to FS errors, then if we use gr.ReadLine we'll get 2 messages, but only parse one.

{"time":"2017-10-24T08:13:04.564Z","msg":{"height":6,"round":0,"step":"RoundStepNewHeight"}}{"time":"2017-10-24T08:13:04.564Z","msg":{"height":7,"round":0,"step":"RoundStepNewHeight"}}\n

new WAL#SearchForHeight method sequentially reads the WAL file and panics if no separator is found.

what happens when this sequence shows up in a block ?

answering your original question, nothing bad will happen because as I explained above we're no longer splitting the log (using strings.Split or bytes.Split; bytes.Split is only used in tests), but rather read it byte by byte (4bytes CRC -> 4 bytes length -> (length) data -> separator -> 4bytes CRC -> 4 bytes length ...)

Yep thanks

does this mean we're hoping to do all this without a separator ?

see #672 (comment)

if err == nil, do we always have n == length ?

And how is this reported err different from EOF here?

I had an impression that Read always returns err if n < length, but that seems not to be the case When Read encounters an error or end-of-file condition after successfully reading n > 0 bytes, it returns the number of bytes read. It may return the (non-nil) error from the same call or return the error (and n == 0) from a subsequent call.. We probably need to write more tests and revisit implementation of groupReader#Read.

tendermint/tmlibs#69

< >

I think it should be above if !found.

It seems in all cases where !found, we also have gr == nil. Otherwise, we panic when we try to close gr.Close().