Description

TLDR: Relax index upgrade policy across major versions to only be as strict as necessary.

Here is an attempted summary of a recent discussion about this.

Currently, Lucene's policy requires a full reindex when upgrading across more than one major version, which can create significant friction for users with large indexes. We propose relaxing this policy to allow upgrades across multiple major versions when it is safe to do so. The goal is to provide flexibility for users without compromising data integrity or flexibility.

Proposed Changes:

• Modify Upgrade Policy: Allow upgrades across multiple major versions, replacing the existing restriction with a configurable MIN_SUPPORTED_MAJOR version in Version.java.

• Controlled Version Bumping: Bump MIN_SUPPORTED_MAJOR only when necessary, due to index format changes that prevent safe upgrades (e.g., changes to norms encoding).

• Improved Documentation: Clearly document which versions can be safely upgraded to the current version without reindexing.

• Retain Reindexing When Necessary: Ensure that reindexing is still required when necessary to maintain correctness or prevent the propagation of corruption.

Benefits:

• Reduces friction and operational overhead for users with large indexes.

• Facilitates more frequent major releases by reducing mandatory reindexing.

• Maintains safety and integrity by reindexing only when required.

Implementation Plan:

• Modify Version.java to use a configurable MIN_SUPPORTED_MAJOR.

• Update the index upgrade logic to check against MIN_SUPPORTED_MAJOR rather than just the previous major version.

• Enhance documentation to provide clear guidelines on safe upgrade paths and scenarios requiring reindexing.

Request for Feedback: We welcome feedback from the community on this proposal, especially regarding its potential impact, implementation details, and any concerns about safety and backward compatibility.

• Note: Upgrading from Lucene 20 to Lucene 23 would require first going from 20 to 21, from 21 to 22, and then 22 to 23. Unless of course a change occurred in one of those versions that would prevent you from doing so, in which case a reindex would be required.