Add rocksdb by Stumble · Pull Request #12 · tendermint/tm-db

Stumble · 2019-07-30T00:28:04Z

Using my fork of gorocksdb from https://github.com/tecbot/gorocksdb for two reasons: 1. The upstream repo encourages vendoring for API stability. 2. The default LDFLAGS setting in upstream repo enforces linking unnecessary compression libraries(https://github.com/tecbot/gorocksdb/blob/162552197222a834c6857385ae890e4decdd5049/dynflag.go#L5), so that if the library is not installed, link will fail. IMHO, tendermint should fork this wrapper and use it here.

To run tests:
First, install rocksdb,

homedir="$PWD"
sudo apt-get update
sudo apt-get install -y build-essential gcc g++ make libgflags-dev libsnappy-dev
wget -O rocks.tar.gz https://github.com/facebook/rocksdb/archive/v6.1.2.tar.gz && \
  tar -zxvf rocks.tar.gz && \
  cd rocksdb-6.1.2/ && \
  make static_lib -j4 && \
  sudo cp librocksdb.a /usr/local/lib/ && \
  cd include/ && \
  sudo rm -rf /usr/local/include/rocksdb && \
  sudo cp -r rocksdb /usr/local/include/ && \
  sudo ldconfig
cd "$homedir"
rm rocks.tar.gz
rm -rf rocksdb-6.1.2

Run tests:

GO_LDFLAGS="-lsnappy" go test -v -tags "rocksdb"

melekes

👍 looking good. left a few comments. would be great to have more tests!

db/db.go

db/c_level_db_test.go

melekes · 2019-07-31T08:24:50Z

db/rocks_db.go

+	ro := gorocksdb.NewDefaultReadOptions()
+	wo := gorocksdb.NewDefaultWriteOptions()
+	woSync := gorocksdb.NewDefaultWriteOptions()
+	woSync.SetSync(true)


what this line does?

Do you mean these variables? read/write options?

I mean woSync.SetSync(true)

woSync.SetSync(true) prepares a sync write option. https://godoc.org/github.com/tecbot/gorocksdb#WriteOptions.SetSync

melekes · 2019-07-31T08:26:47Z

db/rocks_db.go

+	key = nonNilBytes(key)
+	res, err := db.db.Get(db.ro, key)
+	if err != nil {
+		res.Free()


why not put this above with defer? that way we'll avoid duplication s.Free() in moveSliceToBytes and here.

turns out res.Free() when err != nil is not necessary, removed. I would prefer to have a function that moves c bytes to go slice.

melekes · 2019-08-02T07:47:16Z

Using my fork of gorocksdb from https://github.com/tecbot/gorocksdb for two reasons:

Have you tried to contribute back to original lib?

Stumble · 2019-08-05T21:47:13Z

Using my fork of gorocksdb from https://github.com/tecbot/gorocksdb for two reasons:

Have you tried to contribute back to original lib?

Nope, as vendoring was suggested for gorocksdb anyway. I just sent an issue in the upstream, let's see how it goes.

Stumble · 2019-08-09T17:42:18Z

@melekes , the upstream has not respond to the issue yet. tecbot/gorocksdb#167

Stumble · 2019-08-10T21:54:56Z

Rocksdb's stats does not work with https://github.com/cosmos/cosmos-sdk/blob/e4c8bd72b72b18035ede103cb4a47da5825d330e/server/export.go#L95.

db.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

tac0turtle · 2019-10-14T14:58:35Z

@Stumble it seems gorocksdb has picked up steam could you rebase your fork until your PR get merged?

Stumble · 2019-10-17T18:38:01Z

@marbar3778 The PR is merged, I suggest we switch to the original branch.

Closes #3831 The support for rocksdb was added a while back in tendermint/tm-db#12. This commit merely updates the documentation.

* Create codeql-analysis.yml * Update codeql-analysis.yml Co-authored-by: Marko <marbar3778@yahoo.com>

melekes reviewed Jul 31, 2019

View reviewed changes

Stumble added 3 commits July 31, 2019 12:12

add rocksdb

d8b2a04

move codes

059f930

dir update and misc fixes

f50c457

Stumble force-pushed the yumin/add-rocksdb branch from 9b8d7d8 to f50c457 Compare July 31, 2019 19:23

fix tests

5414ac1

melekes approved these changes Jul 31, 2019

View reviewed changes

melekes reviewed Sep 18, 2019

View reviewed changes

db.go Outdated Show resolved Hide resolved

Update db.go

9ec1059

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

melekes approved these changes Sep 18, 2019

View reviewed changes

melekes merged commit 37a6176 into tendermint:master Sep 18, 2019

melekes mentioned this pull request Sep 18, 2019

Switch to original techbot rocksdb wrapper #18

Closed

melekes added a commit to tendermint/tendermint that referenced this pull request Dec 11, 2019

config: add rocksdb as a db backend option

51a9332

Closes #3831 The support for rocksdb was added a while back in tendermint/tm-db#12. This commit merely updates the documentation.

melekes mentioned this pull request Dec 11, 2019

config: add rocksdb as a db backend option tendermint/tendermint#4239

Merged

5 tasks

melekes added a commit to tendermint/tendermint that referenced this pull request Dec 11, 2019

config: add rocksdb as a db backend option (#4239)

15e80d2

Closes #3831 The support for rocksdb was added a while back in tendermint/tm-db#12. This commit merely updates the documentation.

faddat pushed a commit to notional-labs/tm-db that referenced this pull request May 8, 2022

chore: revise goleveldb iterator (tendermint#12)

3299517

tac0turtle added a commit that referenced this pull request Aug 25, 2022

Create codeql-analysis.yml (#12)

1c9ceed

* Create codeql-analysis.yml * Update codeql-analysis.yml Co-authored-by: Marko <marbar3778@yahoo.com>

Conversation

Stumble commented Jul 30, 2019

Uh oh!

melekes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

melekes Jul 31, 2019

Choose a reason for hiding this comment

Uh oh!

Stumble Jul 31, 2019

Choose a reason for hiding this comment

Uh oh!

melekes Aug 2, 2019

Choose a reason for hiding this comment

Uh oh!

Stumble Aug 5, 2019

Choose a reason for hiding this comment

Uh oh!

melekes Jul 31, 2019

Choose a reason for hiding this comment

Uh oh!

Stumble Jul 31, 2019

Choose a reason for hiding this comment

Uh oh!

melekes commented Aug 2, 2019

Uh oh!

Stumble commented Aug 5, 2019

Uh oh!

Stumble commented Aug 9, 2019

Uh oh!

Stumble commented Aug 10, 2019

Uh oh!

Uh oh!

tac0turtle commented Oct 14, 2019

Uh oh!

Stumble commented Oct 17, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants