Skip to content
This repository was archived by the owner on Aug 2, 2021. It is now read-only.

smart syncronisation #23

Merged
zelig merged 78 commits intobzzfrom
bzzsyncer
Nov 26, 2015
Merged

smart syncronisation #23
zelig merged 78 commits intobzzfrom
bzzsyncer

Conversation

@zelig
Copy link
Copy Markdown
Member

@zelig zelig commented Nov 7, 2015

This is a major overhaul of all storage/syncronisation logic.

  • Syncer implements priority queues for outgoing store request handling
  • All traffic can now go through batch confirmations (unsyncedKeys (batch) -> deliveryRequest (batch) -> delivery) roundtrips
  • dbStore sync iterator to help sync address and accessCount bands of the chunkdb
  • request db to persist backlog of outgoing deliveries (resilient to disconnects)
  • syncronisation state with the remote peers is persisted as part of KADDB
  • configuration of confirmation roundtrips and priority levels avaiable for 5 store request types: delivery, propagation, deletion, history, backlog

Code changes:

  • chunker: add serialisation to Key
  • config: add syncParams to config params
  • dbStore: add public accessor to accessCnt dataIdx oops
  • dbStore: dbSyncIterator slightly modified, dbSyncParams abstracted out
  • netStore: fix incorrect address when handling self lookup from remote peeer
  • netStore: rename propagateResponse -> deliver and store-> propagate, remove old deliver
  • netStore: now uses the syncer for store requests
  • protocol: add syncRequestMsg, deliveryRequestMsg and unsyncedKeysMsg
  • protocol/kademlia: add meta data storage support to kaddb, persists and initialises advertised sync state
  • kademlia: major refactor
  • protocol: remove request db and storage loop handling (now syncer)
  • syncer/syndb: new files implementing the smart syncing and batch requests

zelig and others added 30 commits October 26, 2015 22:24
* xeth, rpc: implement eth_getNatSpec for tx confirmations
* rename silly docserver -> httpclient
* eth/backend: httpclient now accessible via eth.Ethereum init-d via config.DocRoot
* cmd: introduce separate CLI flag for DocRoot (defaults to homedir)
* common/path: delete unused assetpath func, separate HomeDir func
XEth.gpo was being initialized as needed. WithState copies the XEth
struct including the gpo field. If gpo was nil at the time of the copy
and Call or Transact were invoked on it, an additional GPO listenLoop
would be spawned.

Move the lazy initialization to GasPriceOracle instead so the same GPO
instance is shared among all created XEths.

Fixes #1317
Might help with #1930
eth, xeth: fix GasPriceOracle goroutine leak
makefile: fix evm ld flags, pass them to xgo too
cmd/utils, crypto: add --lightkdf flag for lighter KDF
 check argument type before parsing params
 recover from panic in ipc channel
eth: don't block sync goroutines that short circuit
Argument parsing can lead to panic in rpc channel
There are a bunch of changes required to make this work:

- in miner: allow unregistering agents, fix RemoteAgent.Stop
- in eth/filters: make FilterSystem.Stop not crash
- in rpc/comms: move listen loop to platform-independent code

Fixes #1930. I ran the shell loop there for a few minutes and didn't see
any changes in the memory profile.
Changed field `input` to new `inputs`. Addad Hash and Address as input
types.

Added bytes[N] and N validation
Godeps: upgrade github.com/huin/goupnp
obscuren and others added 5 commits November 5, 2015 20:27
eth: fix error casting regression during database open
event/filter: fix data race in the test
rpc/api: fix #1972 api regression (nil eth panic) in attach
* chunker: add serialisation to Key
* config: add syncParams to config params
* dbStore: add public accessor to accessCnt
* dbStore: dbSyncIterator slightly modified, dbSyncParams abstracted out
* netStore: fix incorrect address when handling self lookup from remote peeer
* netStore: rename propagateResponse -> deliver and store-> propagate, remove old deliver
* netStore: now uses the syncer for store requests
* protocol: add  deliveryRequestMsgd (0x05) and unsyncedKeysMsg (0x06), add sync state to statusMsgData and handling
* protocol/kademlia: add meta data storage support to kaddb, persists and initialises advertised sync state
* protocol: remove request db and storage loop handling (now syncer)
* syncer/syndb: new files implementing the smart syncing and batch requests
zelig and others added 19 commits November 7, 2015 22:37
* invalid hash gives different error
* check for empty counter range in dbsync iterator, error response
* remove caps from bzz hanshake staatusmsgdata
* remove metadata (unused)
* all bzzprotocol writes return error and drop peer on error
* 1 minute chequebook confirmation interval
* register httpclient bzz scheme within swarm.go
* fix sync() func
* improve logs, counts etc

Merge remote-tracking branch 'upstream/develop' into bzzsyncer

Conflicts:
	cmd/utils/flags.go
	rpc/api/admin.go
	rpc/api/utils.go
* hive: peer wrapper to implement kademlia Node interface moved from protocol
* hive: adapt to new Kademlia API
* config: include kademlia purge-related params
* protocol: syncRequest new message added need to send after handshake, sets up syncer
* syncer: fix sync() tasks and sync state transitions
* chunker: fix Hex on Key
* deliver func now takes a channel argument to wait on in order to quit when (most commonly) routine is blocked on delivery
* addKey, addDelivery, doDelivery all quit if extra channel closed
* check cache contention now moved inside select case and switches to db already at len == cap - 1
* launch the db iterator after len(cache) items from cache are delivered
* do extra round of cleanup from db while blocking read from cache so that items read after iterator quit and before signal sent do not get stuck in the db
* Close the db when syncdb stops
* save is simplified, counter incremented AFTER put (now starts with 0)
* iterator start key is now precalculated as start field
* rename: cache -> buffer , caching -> usedb (word cache is misleading)
* db put and delete now done inn batches applied only before each new iteration cycle
* add test for correct order of db retrieval and deletion on delivery
Resolve bzz:// through http protocol handler
zelig added a commit that referenced this pull request Nov 26, 2015
@zelig zelig merged commit a4c24bc into bzz Nov 26, 2015
@zelig zelig deleted the bzzsyncer branch November 29, 2016 09:27
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants