CellIndicesNotUnique error during reconstruction #7991
Description
Saw these logs on a devnet 3 node
Sep 03 11:51:25.493 DEBUG Component added to data availability checker component: "data_columns", status: "block 1 data_columns 99/128", block_root: 0x2b40faf5fdf22ea758649b7a9f3759d7db5166d3982e348e12aeb156d0c0289c
Sep 03 11:51:25.495 DEBUG Successfully verified gossip data column sidecar slot: 303707, block_root: 0x2b40faf5fdf22ea758649b7a9f3759d7db5166d3982e348e12aeb156d0c0289c, index: 127, index: 127, block_root: 0x2b40faf5fdf22ea758649b7a9f3759d7db5166d3982e348e12aeb156d0c0289c, slot: Slot(303707)
Sep 03 11:51:25.495 DEBUG Component added to data availability checker component: "data_columns", status: "block 1 data_columns 100/128", block_root: 0x2b40faf5fdf22ea758649b7a9f3759d7db5166d3982e348e12aeb156d0c0289c
Sep 03 11:51:25.502 DEBUG Successfully verified gossip data column sidecar slot: 303707, block_root: 0x2b40faf5fdf22ea758649b7a9f3759d7db5166d3982e348e12aeb156d0c0289c, index: 6, index: 6, block_root: 0x2b40faf5fdf22ea758649b7a9f3759d7db5166d3982e348e12aeb156d0c0289c, slot: Slot(303707)
Sep 03 11:51:25.502 DEBUG Component added to data availability checker component: "data_columns", status: "block 1 data_columns 101/128", block_root: 0x2b40faf5fdf22ea758649b7a9f3759d7db5166d3982e348e12aeb156d0c0289c
Sep 03 11:51:25.587 DEBUG Self rate limiting due to the number of concurrent requests peer_id: 16Uiu2HAmRvyB9mAsSKucs92vJ61KWN18KPnz2LAmEHRRAZYFcXkH, protocol: data_column_sidecars_by_range
Sep 03 11:51:25.653 ERROR Error reconstructing data columns block_root: 0x2b40faf5fdf22ea758649b7a9f3759d7db5166d3982e348e12aeb156d0c0289c, error: PeerDASKZG(Recovery(CellIndicesNotUnique))
Sep 03 11:51:25.653 ERROR Error during data column reconstruction block_root: 0x2b40faf5fdf22ea758649b7a9f3759d7db5166d3982e348e12aeb156d0c0289c, error: AvailabilityCheck(ReconstructColumnsError(PeerDASKZG(Recovery(CellIndicesNotUnique))))
This recent change in the kzg library crate-crypto/rust-eth-kzg#421 seems to have triggered it.
Not sure if its an issue with our reconstruction or some issue with the da checker storing the same columns repeatedly.
Needs investigation.