Comments on custody spec / beaconchain shard management, after working with the experimental code and discussing with Vitalik:

• As mentioned earlier, the custody game is single-round (= immediate) now. Since we have an upper-bound of ~512 kB shard blocks. (Or 1MB or whatever the number is going to be)
  • This means completely removing the challenge/record/response/elapse stuff.
  • Merging the challenge and response to directly process custody data verification
  • New incentives necessary, since there is no more "challenger" and the slashing work is easily stolen by any proposer:
    • any validator can sign custody-slashing work to claim it is valid.
    • invalid claims get slashed
    • valid claims slash the custody bit malefactor, and reward the attestation committee
    • the committee is at least incentivised to produce signed custody slashings.
    • other actors can volunteer to sign if they care about the deflation and network health.
    • the proposer gets a small inclusion reward
  • Improvised to get custody bits for the new data. Because of the immediate processing, the old intermediate custody chunks don't help. No merkle branches either. Just compute as if it was one big custody chunk.
    • Haven't done much with the previous custody spec, so I probably made a mistake somewhere. I really just want to get this unstuck and moving, quick. We needed it yesterday to have implementers try the phase 1 spec. Review/improvements welcome, but if the scope is bigger we maybe should move it to a future PR.
    • I urge everyone to stop using the previous "challenge chunk" for subdivided custody data. Legendre bits only are constructed from a constant 48 bytes of data, which I call an "atom", since we don't subdivide it (classically...). The "chunk" is only confusing with the new chunking of shard blocks, which is different. See below.
  • This introduces a new CustodySlashing: provide the data to reconstruct the custody bit, along with the data to match it to an attestation, check the attestation, and identify the "malefactor": literally a validator who made a wrong custody bit.
    • The proof can be compressed more later; attestation -> transition -> shard data has extra unnecessary data along the way that we don't really need in the slashing.
• The work of V has a new interesting split of shard block data in the proposal:
  • Shards are tracked by shard-transitions, which in turn can contain multiple shard blocks to catch up missed slots.
  • Each of these shard blocks contains data split in chunks:
    • We do not want an mostly empty block (e.g. 23%) to cost as much space as a full one (100%). Instead, it is split in chunks to limit it to increments of 1 / MAX_SHARD_BLOCK_CHUNKS. Current increments are 0/25/50/75/100 %. These chunks can be concatenated and padded with zeroes to do the regular custody bit computation with.
      • Thought about ByteList here instead, but the goal is that all data can be concatenated for a single data availability root. And lists don't align well to subtrees to support that. "So the point is that you don't want the shard data to be a list because you want to be able to concat it all into a shared data root"
      • The CustodySlashing does not split data as intelligently yet. We can change the custody bit computation to deal with empty chunks. E.g. void atom custody bit == 0
      • I dislike it, as the padding is not natural. List-mixins of ByteList are the blocker to change things here.
    • Making shard blocks smaller and allowing a proposer to produce max. e.g. 4 consecutively in the same slot of them introduces more headers/network overhead, but making the shard block a fixed-size logistical unit sounds good. We have ShardTransition already to deal with multiple of them, and it would flatten this lists of lists thing (transition -> blocks -> chunks into transition -> blocks).
• Because of the multiple blocks per transition, and multiple chunks per block, the custody-bit for an attestation is complicated: each block has its own custody bitfield. Chunks of a block combine into a single custody bit still.

Also, shard_block_lengths is not really enforced other than the right amount of chunks. A length-mixin as part of the shard data root could fix that.

In the method get_shard_committee(beacon_state: BeaconState, epoch: Epoch, shard: Shard) the parameter shard is not utilized within the method

@dangerousfood Thanks. It was pointed out in the earlier PR already though. I think it's related to the shard-rotation TODO that's left. Will look into it 👍

clsing in favor of #1504