I'm not understanding this operation. It seems like there are actually two states? In one case, we did the chunking and figured out the object name but we just haven't written it back yet. In teh second case, we did that already, and overwrite it again, so now it has new/different content. In that case, we don't actually want to write it back to the same object, do we? (Or is the content-based naming not addressed yet? Sorry, my memory is fuzzy on what has been done so far and what hasn't!)

There is one or two state depending on the object size. For example,

1. figure out dirty chunks's size (because we don't know actual dirty chunks,
   this will be used to check the flush completion)
2. send write messages
3. receive acks
   3.1. check completion (whether all dirty chunk we sent are completed or not)
   3.2. do remaining jobs

Regarding remaining jobs,
I thought we needed a way to avoid resource starvation. For example,
manifest_flush() needs read(storage) ->write(network) operations. but, if the object size is huge, flush() repeats such operations without yielding resources.
So, I added the following logic. If the object size (many dirty chunks) is larger than the threshold value (for example, 4MB), just stop sending data and wait for the previous flush request to complete.
This flush() will continue when the previous requests (for flush()) are complete.

The flush model sounds right.

I'm thinking of this scenario:

1. write object
2. turn it into many chunks
3. chunks are flushed to lower tier
4. part of object is overwritten in base, and some chunks are marked dirty

At this point, the implemenation of flush above is assuming that the chunks are mutable, so just marking them dirty so they can be updated later is sufficient. This is fine for normal tiering, but won't work for dedup (where the chunk is immutable, and the manifest chunk isn't dirty, it's more like 'dangling'). Maybe that's fine for now, but what is the plan here? Will these chunks have a state bit like "immutable" or "cas" so that the base tier knows how to deal with them? Because I think the "mark chunks dirty when we overwrite parts of the object" code will need to change. This mid-point is a but odd because I don't think you would actually chunk an object into mutable chunks... chunking is only really useful for the dedup case.

I think the chunk (both dedup and normal tier) can be "mutable" because we can consider following scenarios.

• Chunk states for normal tiering (no problem):
  Missing -> (overwritten) -> dirty -> (flushed) ->clean or missing -> (overwritten) -> dirty
  -> (flushed) -> clean or missing...

• Chunk states for dedup tiering (no dangling):
  Missing -> (overwritten) -> dirty -> (flushed) -> clean or missing -> (overwritten) -> dirty
  -> (if the chunk was flushed, we can see old chunk,target_oid (== old cas object id).
  So, just send decrease reference message to old cas object and then, flush()) ->
  (if flush is finished, chunk.target_oid can be updated by new value) -> clean or missing ...

We only need to check whether the chunk was flushed and send decrease message before the chunks is flushed to lower tier. If the cas object's reference is 0, this will be deleted.

Did I misunderstand your question or are there any uncertainties in what I said?

That makes sense!