rabbit_khepri: Optimize cleanup in topic binding projection#15214
rabbit_khepri: Optimize cleanup in topic binding projection#15214
Conversation
|
This not ready for review or testing at all, my understanding of the problem is incorrect. |
fefa84a to
66a5ba5
Compare
cb5836c to
ca182d1
Compare
ca182d1 to
7f0976a
Compare
|
I thnik we need to bump Horus or remove this dep_horus: As it is, Horus 0.3.1 is used on this branch, causing a known/fixed issue. |
|
Good catch, I forgot about this old temporary change. |
[Why] Before this patch, the projection function that managed the topic bindings graph only kept track of "topic word -> topic word" = "this node ID". Reading and updating this graph was efficient. However, deleting a node in this graph was expensive because it used `ets:match/3` to determine if an edge was pointing to nothing and could be reclaimed. `ets:match()` does a full scan of the table. On my laptop, this scan took 20 ms with 100k topic bindings, thus >30 minutes to delete all of them. [How] The new projection function tracks the number of children a target node has as well. This way, it knows that if this counter reaches 0, the edge can be reclaimed. The same test with 100k topic bindings takes 3.5 seconds to delete them.
7f0976a to
612d93d
Compare
This was fixed in #15230. |
mkuratczyk
left a comment
There was a problem hiding this comment.
As discussed, I can still make this behave poorly as we need one more change (fix for exchange serials) but this part is good and the serials fix is coming soon. thanks!
|
The mentionned fix to exchange serials is now submitted serapately in #15236. |
rabbit_khepri: Optimize cleanup in topic binding projection (backport #15214)
Why
Before this patch, the projection function that managed the topic bindings graph only kept track of "topic word -> topic word" = "this node ID". Reading and updating this graph was efficient.
However, deleting a node in this graph was expensive because it used
ets:match/3to determine if an edge was pointing to nothing and could be reclaimed.ets:match()does a full scan of the table. On my laptop, this scan took 20 ms with 100k topic bindings, thus >30 minutes to delete all of them.How
The new projection function tracks the number of children a target node has as well. This way, it knows that if this counter reaches 0, the edge can be reclaimed.
The same test with 100k topic bindings takes 3.5 seconds to delete them.