Skip to content

Pebble#283

Closed
faddat wants to merge 84 commits intotendermint:masterfrom
notional-labs:pebble-again
Closed

Pebble#283
faddat wants to merge 84 commits intotendermint:masterfrom
notional-labs:pebble-again

Conversation

@faddat
Copy link
Contributor

@faddat faddat commented Jul 28, 2022

Pebble is a rocks-like kv store written in pure go.

  • no cgo
  • faster than rocks especially when used with cw
  • faster than goleveldb
  • more meaningfully configurable than goleveldb
  • complies with all tm-db tests, and tested with additional scrutiny by Notional

pebble is the best possible candidate for "the universal cosmos kv store".

catShaark and others added 30 commits April 13, 2022 18:37
@catShaark
add pebble
16f0d61
@orthory @faddat
fix: remove mutex in prefixdb
5c9bfe8
@faddat
fmt
a0ff309
@faddat
fmt
636377b
@faddat
Delete .gitpod.yml
0155808
@faddat
remove mutex entirely instead of commenting out
973a74a
@faddat
fmt
34ca0e7
@faddat
remove .gitpod.yml
d81b67b
@faddat
update changelog
4c9b066
@faddat
Merge branch 'master' into remove-mutex
0566540
@faddat
Merge https://github.com/tendermint/tm-db
51bdeca
@faddat
prefixdb test
f531a3f
@vuong177
all test passed in prefixdb_test.go
cc11706
@catShaark
Merge branch 'master' into remove-mutex
417a583
@vuong177
simple taskKey in prefixdb_test.go
3214560
@faddat @creachadair
Update prefixdb.go
fd0bb3d 
Co-authored-by: M. J. Fromberger <michael.j.fromberger@gmail.com>
@faddat
Merge branch 'master' into remove-mutex
31c47b6
@faddat
fmt db test so pr passes linter
835d8fd
@vuong177
refactor func Run
cb000ee
@vuong177
check nil err before create PrefixDB in TestWithGolevelDB
c6d827f
@catShaark
add other db tests
b78aed6
@faddat @creachadair
Update prefixdb_test.go
e2c3c4e 
Co-authored-by: M. J. Fromberger <michael.j.fromberger@gmail.com>
@faddat
Merge branch 'master' into remove-mutex
d56e038
@catShaark
update prefixdb_test.go
84a67af
@vuong177
minor defer
72c47a4
@faddat
Merge branch 'master' into remove-mutex
0931c90
@faddat
fixup with build tags
5c81bf8
@faddat
Create .gitpod.yml
11fa71a
@faddat
fix linting
d57bb1f
@faddat
itr.isInvalid would be unconditionally set anyhow, which is why the i…
8e4e401 
…tr.isInvalid were considered to be ineffective assignments by the linter.
@mergify mergify bot mentioned this pull request Jul 28, 2022
Closed
@codecov
Copy link

codecov bot commented Jul 28, 2022 •
edited
Loading

Codecov Report

Merging #283 (4508c24) into master (d1b9b74) will decrease coverage by 1.79%.
The diff coverage is 77.08%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #283      +/-   ##
==========================================
- Coverage   68.54%   66.75%   -1.80%     
==========================================
  Files          27       26       -1     
  Lines        2130     1940     -190     
==========================================
- Hits         1460     1295     -165     
+ Misses        595      575      -20     
+ Partials       75       70       -5
Impacted Files Coverage Δ
cleveldb.go 70.99% <ø> (ø)
cleveldb_batch.go 82.35% <ø> (ø)
db.go 35.00% <ø> (ø)
prefixdb.go 63.55% <ø> (-6.79%) ⬇️
prefixdb_iterator.go 81.01% <ø> (-0.24%) ⬇️
rocksdb.go 72.26% <ø> (ø)
rocksdb_batch.go 84.31% <ø> (ø)
pebble_iterator.go 72.60% <72.60%> (ø)
pebble.go 77.12% <77.12%> (ø)
pebble_batch.go 78.43% <78.43%> (ø)
... and 4 more
Impacted Files Coverage Δ
cleveldb.go 70.99% <ø> (ø)
cleveldb_batch.go 82.35% <ø> (ø)
db.go 35.00% <ø> (ø)
prefixdb.go 63.55% <ø> (-6.79%) ⬇️
prefixdb_iterator.go 81.01% <ø> (-0.24%) ⬇️
rocksdb.go 72.26% <ø> (ø)
rocksdb_batch.go 84.31% <ø> (ø)
pebble_iterator.go 72.60% <72.60%> (ø)
pebble.go 77.12% <77.12%> (ø)
pebble_batch.go 78.43% <78.43%> (ø)
... and 4 more

@mhughdo mhughdo mentioned this pull request Jul 28, 2022
Closed
@faddat faddat closed this Jul 29, 2022
@faddat
Copy link
Contributor Author

faddat commented Jul 29, 2022

wrong branch, sorry if anyone used this one.

@faddat faddat mentioned this pull request Jan 15, 2024
Merged
@faddat faddat mentioned this pull request Jan 29, 2024
Merged
4 tasks
github-merge-queue bot pushed a commit to cometbft/cometbft that referenced this pull request Jan 29, 2024
@faddat @melekes
feat: add pebbledb (https://github.com/cockroachdb/pebble) (#2132)
82fa3c0 
This PR integrates pebbledb, a key-value store by Cockroachdb based on
Cockroach Labs findings about cgo:

* https://www.cockroachlabs.com/blog/the-cost-and-complexity-of-cgo/


Unlike any other kv store that cometbft currently supports:

* pebble is actively maintained unlike:
  * goleveldb
  * boltdb
  * badgerdb 
* pebble does not require the use of CGO like:
  * cleveldb
  * rocksdb

In benchmarks:

* pebble performs better than goleveldb, cleveldb and rocksdb
* pebble performs consistently

At Notional Ventures, Pte, we've used and submitted pebble for two years
for:

* archive nodes that would be crushed if they used goleveldb
* high performance RPC infrastructure

Here are the hitstorical pull requests to merge pebble:

* tendermint/tm-db#230
* tendermint/tm-db#231
* tendermint/tm-db#281
* tendermint/tm-db#282
* tendermint/tm-db#283
* tendermint/tm-db#284
* tendermint/tm-db#304
* tendermint/tm-db#321

Pebble snapshots for cosmos blockchains are available here:

* https://snapshot.notional.ventures/

---

#### PR checklist

- [ ] Tests written/updated
- [x] Changelog entry added in `.changelog` (we use
[unclog](https://github.com/informalsystems/unclog) to manage our
changelog)
- [x] Updated relevant documentation (`docs/` or `spec/`) and code
comments
- [x] Title follows the [Conventional
Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec

---------

Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
mergify bot pushed a commit to cometbft/cometbft that referenced this pull request Jan 29, 2024
@faddat @melekes @mergify
feat: add pebbledb (https://github.com/cockroachdb/pebble) (#2132)
2650214 
This PR integrates pebbledb, a key-value store by Cockroachdb based on
Cockroach Labs findings about cgo:

* https://www.cockroachlabs.com/blog/the-cost-and-complexity-of-cgo/

Unlike any other kv store that cometbft currently supports:

* pebble is actively maintained unlike:
  * goleveldb
  * boltdb
  * badgerdb
* pebble does not require the use of CGO like:
  * cleveldb
  * rocksdb

In benchmarks:

* pebble performs better than goleveldb, cleveldb and rocksdb
* pebble performs consistently

At Notional Ventures, Pte, we've used and submitted pebble for two years
for:

* archive nodes that would be crushed if they used goleveldb
* high performance RPC infrastructure

Here are the hitstorical pull requests to merge pebble:

* tendermint/tm-db#230
* tendermint/tm-db#231
* tendermint/tm-db#281
* tendermint/tm-db#282
* tendermint/tm-db#283
* tendermint/tm-db#284
* tendermint/tm-db#304
* tendermint/tm-db#321

Pebble snapshots for cosmos blockchains are available here:

* https://snapshot.notional.ventures/

---

#### PR checklist

- [ ] Tests written/updated
- [x] Changelog entry added in `.changelog` (we use
[unclog](https://github.com/informalsystems/unclog) to manage our
changelog)
- [x] Updated relevant documentation (`docs/` or `spec/`) and code
comments
- [x] Title follows the [Conventional
Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec

---------

Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
(cherry picked from commit 82fa3c0)
melekes added a commit to cometbft/cometbft that referenced this pull request Jan 29, 2024
@faddat @melekes
feat: add pebbledb (https://github.com/cockroachdb/pebble) (#2132)
01e8d77 
This PR integrates pebbledb, a key-value store by Cockroachdb based on
Cockroach Labs findings about cgo:

* https://www.cockroachlabs.com/blog/the-cost-and-complexity-of-cgo/


Unlike any other kv store that cometbft currently supports:

* pebble is actively maintained unlike:
  * goleveldb
  * boltdb
  * badgerdb 
* pebble does not require the use of CGO like:
  * cleveldb
  * rocksdb

In benchmarks:

* pebble performs better than goleveldb, cleveldb and rocksdb
* pebble performs consistently

At Notional Ventures, Pte, we've used and submitted pebble for two years
for:

* archive nodes that would be crushed if they used goleveldb
* high performance RPC infrastructure

Here are the hitstorical pull requests to merge pebble:

* tendermint/tm-db#230
* tendermint/tm-db#231
* tendermint/tm-db#281
* tendermint/tm-db#282
* tendermint/tm-db#283
* tendermint/tm-db#284
* tendermint/tm-db#304
* tendermint/tm-db#321

Pebble snapshots for cosmos blockchains are available here:

* https://snapshot.notional.ventures/

---

#### PR checklist

- [ ] Tests written/updated
- [x] Changelog entry added in `.changelog` (we use
[unclog](https://github.com/informalsystems/unclog) to manage our
changelog)
- [x] Updated relevant documentation (`docs/` or `spec/`) and code
comments
- [x] Title follows the [Conventional
Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec

---------

Co-authored-by: Anton Kaliaev <anton.kalyaev@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alexanderbez alexanderbez Awaiting requested review from alexanderbez

@cmwaters cmwaters Awaiting requested review from cmwaters

@creachadair creachadair Awaiting requested review from creachadair

@tac0turtle tac0turtle Awaiting requested review from tac0turtle

@tychoish tychoish Awaiting requested review from tychoish

@williambanfield williambanfield Awaiting requested review from williambanfield

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@faddat @catShaark @orthory @vuong177