hashing and chunking example tools by assafvayner · Pull Request #496 · huggingface/xet-core

assafvayner · 2025-09-17T00:01:39Z

Adds some basic examples tools (compiled with cargo build --examples on data crate) to compute hashes and chunk boundaries.

…-examples

data/examples/chunk/main.rs

data/examples/hash/main.rs

data/examples/xorb-check/main.rs

seanses

LGTM with some minor suggestions.

hoytak · 2025-09-23T18:40:35Z

data/examples/chunk/main.rs

+
+    // set up writer to output chunks information
+    let mut output: Box<dyn Write> = if let Some(save) = args.output {
+        Box::new(BufWriter::new(File::create(save).unwrap()))


Nit -- might be good to use expect instead of unwrap

This is a "dumb" example tool, I consider unwraps here fine

assafvayner added 6 commits September 16, 2025 15:10

example chunker

2fc00f6

hash and chunk tools

722dced

xorb check tool

17ad411

fmt

f8af45e

Merge remote-tracking branch 'origin' into assaf/hashing-and-chunking…

5294949

…-examples

rename type

d4c3138

assafvayner marked this pull request as ready for review September 22, 2025 23:32

assafvayner requested review from hoytak, rajatarya and seanses September 22, 2025 23:32

assafvayner added 4 commits September 22, 2025 16:41

comments for CLI params

a669aea

fmt

bd5b285

Update Cargo.toml

4cbe3f0

fix and update locks

4cd9918