[Colossus] Accept pending data objects enhancements#4971
Conversation
…ept pending data objects txs
mnaamani
left a comment
There was a problem hiding this comment.
Left a few suggestions.
I think we need to start thinking about add a better way to manage the state of the node between runs. Depending on presence of files in directories is brittle and many things can go wrong as the file system is easy to corrupt my moving files around accidentally.
| this.push(dataObject.id, storageBucket.id, dataObject.storageBag.id) | ||
| } else { | ||
| logger.warn( | ||
| `Data object ${dataObject.id} in pending directory is not assigned to any of the upload buckets: ${this.uploadBuckets}.` |
There was a problem hiding this comment.
We would probably want to remove these objects at some point, probably in the "pruning" worker once we merge.
There was a problem hiding this comment.
Yeah, wasn't sure where that should be done, here OR in the pruning service, that's why I left it as it is.
This scenario would mostly happen if the data object is moved (i.e. object's bag is moved). And, I think removing objects due to this action is best suited for the Pruning service (which extensively ensures that moved data objects at least exist in N different buckets before deleting it). WDYT?
Maybe I should create an issue for that?
There was a problem hiding this comment.
Yes lets leave it to the pruning service, and create an issue for it.
Just fyi the version of colossus with the pruning service is merged into colossus-beta branch.
mnaamani
left a comment
There was a problem hiding this comment.
Minor fix in runWithInterval()
| this.push(dataObject.id, storageBucket.id, dataObject.storageBag.id) | ||
| } else { | ||
| logger.warn( | ||
| `Data object ${dataObject.id} in pending directory is not assigned to any of the upload buckets: ${this.uploadBuckets}.` |
There was a problem hiding this comment.
Yes lets leave it to the pruning service, and create an issue for it.
Just fyi the version of colossus with the pruning service is merged into colossus-beta branch.
fixes #4968
This PR:
AcceptPendingObjectsServicethat is responsible for periodically sending batchaccept_pending_data_objectsfor all the pending data objectsPOST /filesendpoint now no longer calls theaccept_pending_data_objectsextrinsic for individual uploads, instead, it registers all the pending objects withAcceptPendingObjectsServicependingoraccepted)