Index stale operations to Lucene to have complete history#29679
Index stale operations to Lucene to have complete history#29679dnhatn merged 7 commits intoelastic:ccrfrom
Conversation
Today, when processing out of order operations, we only add it into translog but skip adding into Lucene. Translog, therefore, has a complete history of sequence numbers while Lucene does not. Since we would like to have a complete history in Lucene, this change makes sure that stale operations will be added to Lucene as soft-deleted documents if required.
|
Pinging @elastic/es-distributed |
| * @param in the input directory reader | ||
| * @return the wrapped reader including soft-deleted documents. | ||
| */ | ||
| public static DirectoryReader includeSoftDeletes(DirectoryReader in) throws IOException { |
There was a problem hiding this comment.
I think we should just call wrapAllDocsLive(DirectoryReader in) it's really unrelated to soft deletes
| /** | ||
| * Returns an internal posting list of the given uid | ||
| */ | ||
| PostingsEnum getPostingsOrNull(BytesRef id) throws IOException { |
| if (postingsEnum == null) { | ||
| continue; | ||
| } | ||
| final NumericDocValues seqNoDV = leaf.reader().getNumericDocValues(SeqNoFieldMapper.NAME); |
There was a problem hiding this comment.
I think we can assert that both seqNoDV and primaryTermDV are non null. They are required at least for this code to work.
| LUCENE_DOC_NOT_FOUND | ||
| } | ||
|
|
||
| private OpVsLuceneDocStatus compareToLuceneHistory(final Operation op, final Searcher searcher) throws IOException { |
There was a problem hiding this comment.
can we have a javadoc for this method?
| /** the op is older or the same as the one that last modified the doc found in lucene*/ | ||
| OP_STALE_OR_EQUAL, | ||
| /** the op is stale but its history is existed in Lucene */ | ||
| OP_STALE_HISTORY_EXISTED, |
There was a problem hiding this comment.
maybe OP_STALE_HISTORY_EXISTS ?
| if (op.primaryTerm() > existingTerm) { | ||
| status = OpVsLuceneDocStatus.OP_NEWER; | ||
| } else { | ||
| status = OpVsLuceneDocStatus.OP_STALE_OR_EQUAL; |
There was a problem hiding this comment.
can you please add comments here in what cases we can get into these situations?
|
I talked to @dnhatn and I'm a bit uncomfortable with the PR changes to how we resolve stale operations in the engine. It seems that the main goal here was to try to avoid indexing duplicates of the same stale operation into Lucene. I'm not sure we need to do this. This is very rare. The alternative would be to accept duplicates under the condition that duplicate seq# also means the same doc version. A few things that are worth noting on top of it:
If we accept the above, we can keep the current (simpler) way we resolve stale operations and follow the following:
|
|
|
||
| /** | ||
| * Wraps a directory reader to include all live docs. | ||
| * The wrapped reader can be used to query documents which are soft-deleted. |
There was a problem hiding this comment.
just say to query all documents
|
@elasticmachine test this please. |
|
run sample packaging tests |
|
@elasticmachine run sample packaging tests |
Today, when processing out of order operations, we only add it into translog but skip adding into Lucene. Translog, therefore, has a complete history of sequence numbers while Lucene does not. Since we would like to have a complete history in Lucene, this change makes sure that stale operations will be added to Lucene as soft-deleted documents if required. Relates #29530
Since elastic#29679 we started adding stale operations to Lucene to have a complete history in Lucene. As the stale docs are rare, we accepted to have duplicate copies of them to keep an engine simple. However, we now need to make sure that we have a single copy per stale operation in Lucene because the Lucene rollback requires a single document for each sequence number.
Today, when processing out of order operations, we only add it into
translog but skip adding into Lucene. Translog, therefore, has a
complete history of sequence numbers while Lucene does not.
Since we would like to have a complete history in Lucene, this change
makes sure that stale operations will be added to Lucene as soft-deleted
documents if required.