Skip to content

store/store.BlockStore.SaveBlock is not atomic #5888

@dshulyak

Description

@dshulyak

Tendermint version (use tendermint version or git rev-parse --verify HEAD if installed from source):

Tip of the master branch https://github.com/tendermint/tendermint/tree/1d16e39c0ea72a1e337662c35e4d50641a3df5fa

In this method:

tendermint/store/store.go

Lines 419 to 472 in 1d16e39

func (bs *BlockStore) SaveBlock(block *types.Block, blockParts *types.PartSet, seenCommit *types.Commit) {
if block == nil {
panic("BlockStore can only save a non-nil block")
}
height := block.Height
hash := block.Hash()
if g, w := height, bs.Height()+1; bs.Base() > 0 && g != w {
panic(fmt.Sprintf("BlockStore can only save contiguous blocks. Wanted %v, got %v", w, g))
}
if !blockParts.IsComplete() {
panic("BlockStore can only save complete block part sets")
}
// Save block parts. This must be done before the block meta, since callers
// typically load the block meta first as an indication that the block exists
// and then go on to load block parts - we must make sure the block is
// complete as soon as the block meta is written.
for i := 0; i < int(blockParts.Total()); i++ {
part := blockParts.GetPart(i)
bs.saveBlockPart(height, i, part)
}
// Save block meta
blockMeta := types.NewBlockMeta(block, blockParts)
pbm := blockMeta.ToProto()
if pbm == nil {
panic("nil blockmeta")
}
metaBytes := mustEncode(pbm)
if err := bs.db.Set(blockMetaKey(height), metaBytes); err != nil {
panic(err)
}
if err := bs.db.Set(blockHashKey(hash), []byte(fmt.Sprintf("%d", height))); err != nil {
panic(err)
}
// Save block commit (duplicate and separate from the Block)
pbc := block.LastCommit.ToProto()
blockCommitBytes := mustEncode(pbc)
if err := bs.db.Set(blockCommitKey(height-1), blockCommitBytes); err != nil {
panic(err)
}
// Save seen commit (seen +2/3 precommits for block)
// NOTE: we can delete this at a later height
pbsc := seenCommit.ToProto()
seenCommitBytes := mustEncode(pbsc)
if err := bs.db.SetSync(seenCommitKey(height), seenCommitBytes); err != nil {
panic(err)
}
}

Parts of the block, block metadata, commited hash and precommited hash are written to the database, and the last write executed with Sync=true. It seems to me that the intent is to fully flush all these items on to the disk, but Sync=true doesn't guarantee that previous writes will be flushed.

This is definitely not the case with goleveldb, because it rotates journals that are used for write, not sure about other used data stores.

As a result here:

tendermint/consensus/state.go

Lines 1528 to 1533 in 1d16e39

if cs.blockStore.Height() < block.Height {
// NOTE: the seenCommit is local justification to commit this block,
// but may differ from the LastCommit included in the next block
precommits := cs.Votes.Precommits(cs.CommitRound)
seenCommit := precommits.MakeCommit()
cs.blockStore.SaveBlock(block, blockParts, seenCommit)

Height might be updated, because metadata is flushed but actual block parts will be missing.

It does seem that this is an easy fix with batch writer, but if it is already handled in some other way feel free to close.

Metadata

Metadata

Assignees

No one assigned

    Labels

    T:jankType Jank! Non-urgent but still high-impact fixes.good first issueContributions Welcome!!

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions