Bit worried about this on Windows: i suspect concurrent async writes (multiple ipfs add) may cause file handle issues, things like golang/go#34681 – flatfs uses https://github.com/alexbrainman/goissue34681 for a reason, we likely need similar wrapper on windows here (old code used tempFileOnce from util_windows.go).

@gammazero too late on my end to write code, but we probably need to create createFile equivalent that does something similar (pseudocode):

util_unix.go:

  func createFile(name string) (*os.File, error) {
  	return os.Create(name)
  }

util_windows.go:

  func createFile(name string) (*os.File, error) {
  	return goissue34681.OpenFile(name, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0666)
  }

Then finally here:

  file, err := createFile(tempFile)

Windows concurrency is janky, unsure if we need a retry wrapper, maybe can go without it, but in case, it could be something like:

util_windows.go:

  func createFile(name string) (*os.File, error) {
  	var file *os.File
  	var err error
  	for i := 0; i < RetryAttempts; i++ {
  		file, err = createFileOnce(name)
  		if err == nil || !isTooManyFDError(err) {
  			break
  		}
  		time.Sleep(time.Duration(i+1) * RetryDelay)
  	}
  	return file, err
  }

Done, without retry wrapper.

Hm.. the Query method documentation says:

Keys Put multiple times appear only once (last write wins)

BUT the duplicate key handling in here seem to silently discard updates?

This means:

batch.Put(ctx, key, []byte("first"))
batch.Put(ctx, key, []byte("second"))
batch.Commit(ctx)  // writes "first", not "second"

@gammazero late friday so maybe I misunderstood intent here, but would it be more intuitive for the batch to implement "last write wins" semantics instead of "first write wins"?

My main worry is that the current behavior could cause subtle bugs for callers expecting standard transaction semantics where later operations override earlier ones.

Thinking was that duplicate keys should have the same data (assuming key is CID), so instead of blocking/overwriting when key already exists, discard the write instead. If "last write wins" is needed within a batch, because a write with same key could have different data than previous write, then we need change to change behavior to block/overwrite implementation or not do asynchronous writes.

Should the batch behavior be configurable? Last-write-wins or discard-duplicate-write

Thanks, I stepped back and did some research on this.

TL;DR: Seems that I was wrong, this is not a genera-purpose datastore: the current behavior (first-successful-writer-wins, discard duplicates) is correct for flatfs's intended use case (limited to CID→block), but we should document it more explicitly (for humans and LLMs).

Why this is fine for flatfs

1. flatfs is explicitly a special-purpose datastore for CID:block pairs only - Kubo's docs already state: "flatfs must only be used as a block store (mounted at /blocks) as it only partially implements the datastore interface"

2. CIDs guarantee value determinism - A CID is a cryptographic hash of the content. If Put(cid, A) and Put(cid, B) are called with the same CID, then A == B by definition (or one is corrupted). First-writer-wins and last-writer-wins produce identical results.

3. Non-batch Put already has this behavior - The existing doWriteOp documents: "we assume that the first succeeding operation on that key was the last one to happen after all successful others"

4. go-datastore Batch interface doesn't guarantee ordering - The interface docs say batches "do NOT have transactional semantics"

Next steps

I've pushed 410cfd2 with documentation updates that make the "first-successful-writer-wins" semantics explicit in :

• README.md restrictions section
• Batch type documentation
• Put method documentation

It also points users needing last-writer-wins for non-deterministic values should use leveldb, or pebble instead.

If we are ok with "first-successful-writer-wins" for flatfs, and it being limited to cid→block use, I think this should be enough. Thoughts @gammazero?

I agree with "first-successful-writer-wins" for this limited purpose datastore, and given rule 2 above, it is the most correct behavior.