localstore: fix index leak when NN chunks enter localstore#1940
Conversation
def1d03 to
ee99c08
Compare
|
The code cannot be compiled: |
a chunk needs be et after being Put into the database
ee99c08 to
e10b1e4
Compare
nolash
left a comment
There was a problem hiding this comment.
I see @zelig reviewed it while I had breakfast halfway. Anyway, curious about the negative bin id
Also curious about the description;
as a result, anonymous content syncing on light nodes could be severely damaged
what is anonmous content syncing on light nodes?!
under certain connectivity condition
I would like descriptions to be more precise, helps us puny humans that don't have context to understand better and do better review work.
|
@nolash the issue is as follows: in the new stream this is not the case. upstream peers establish streams according to their own kademlia and regardless of the downstream peer's connectivity. while this is more resilient implementation (and less attack prone), it did not in its naive implementation cater for the fact that the downstream peer might be the closest node to certain chunks. the connectivity pattern that was manifesting was that the downstream peer appeared outside of saturation depth of other nodes, thus syncing of certain chunks was not bound to happen (since the downstream peer was the closest one and the upstream peers all had not been subscribed to that node's higher bins since it was outside of their depth). this therefore manifested in a localstore leak since some chunks were never |
|
@janos I think I understand more or less. Thanks. |
|
Still not sure about the "what is anonmous content syncing on light nodes" though :) |
This PR fixes an index leak that was manifesting when chunks that would enter the node's local storage would, under certain connectivity conditions be not added to the GC index.
This is due to the fact that the uploader node would be the closest node to the chunk, thus again, under certain connectivity condition this would result in the chunk never syncing, and the
retrievalDataIndexeventually growing out of bound, since no matter how many chunks in lower bins would be garbage collected, the higher bins would always make the store size eventually overflow.This fix introduces a pluggable function to localstore that is called when a chunk enters the local storage.
During implementation several weaknesses were unraveled:
fixes #1920