Upgrade to lucene-9.0.0-snapshot-32a0a16aff0 by romseygeek · Pull Request #73324 · elastic/elasticsearch

romseygeek · 2021-05-24T12:49:51Z

This commit upgrades elasticsearch 8.0 to use a snapshot of
Lucene 9.0.0.

elasticmachine · 2021-05-24T12:49:54Z

Pinging @elastic/es-analytics-geo (Team:Analytics)

elasticmachine · 2021-05-24T12:49:54Z

Pinging @elastic/es-search (Team:Search)

elasticmachine · 2021-05-24T12:49:54Z

Pinging @elastic/es-distributed (Team:Distributed)

romseygeek · 2021-05-24T12:55:10Z

There are a few nocommits, but surprisingly few. Everything compiles.

Still to do:

I have silently remapped the SIMPLEFS store type to just use NIOFS. I'm not sure if this is correct and we will just deprecate the option, or if it should throw an error.
The Version 7.14 constant has the wrong lucene version on it, because we haven't released 8.9 yet
There are a bunch of classes around segment memory accounting that I have left in but are now essentially redundant, eg RamAccountingRefreshListener and SegmentStats. Segment itself also now has unused memoryInBytes and ramTree fields, which I am guessing should be removed but will need some careful BWC handling.
CuckooFilter uses PackedInts.getReader() and save(), both of which are now gone. Again this was being used as part of the Streamable impl and so will need BWC to be addressed when this is reimplemented.

dnhatn · 2021-05-25T17:58:29Z

I have silently remapped the SIMPLEFS store type to just use NIOFS. I'm not sure if this is correct and we will just deprecate the option, or if it should throw an error.

I think we should deprecate the simplefs store option in 7.x and remove it in 8.x.

iverase · 2021-06-01T12:32:01Z

Hey @romseygeek,

I have been looking into the CuckooFilter issue. I think we should have our or version of PackedInts.Mutable, but before anything we need to improve our bwc testing for that part of the code. It seems it is never exercise so we can easily introduce a bug: #73585

dnhatn · 2021-07-16T14:23:37Z

@romseygeek We need to build a new Lucene 9.0 snapshot as the current snapshot doesn't have the version 8.9.0, which is being used in 7.14 and 7.15.

…e-9-upgrade

iverase

LGTM. I have only reviewed the bits affected for the change on endianness of Lucene directory API. I think we should revisit the strategy on KeyStoreWrapper and MetadataStateFormat but it can be done as a follow up.

jtibshirani

I looked through with a focus on the new vector format. I left minor comments and also noticed we need to update LuceneFilesExtensions to contain the vector index file .vex.

It'd also be good to extend the disk usage analyzer to cover vectors, but this seems best as a follow-up.

jtibshirani · 2021-09-20T16:51:16Z

server/src/main/java/org/elasticsearch/index/mapper/DocumentLeafReader.java


+    @Override
+    public VectorValues getVectorValues(String field) throws IOException {
+        return null;


Seems best to throw UnsupportedOperationException here.

jtibshirani · 2021-09-20T17:07:26Z

server/src/main/java/org/elasticsearch/index/engine/RecoverySourcePruneMergePolicy.java


        }

+        private static class FilterStoredFieldVisitor extends StoredFieldVisitor {


Do we need this new private class when we already have one in org.elasticsearch.search.internal?

dnhatn

I've reviewed all files and I have a question, but changes look great.

dnhatn · 2021-09-20T20:37:10Z

server/src/main/resources/org/elasticsearch/bootstrap/security.policy

  // needed for loading plugins which may expect the context class loader to be set
  permission java.lang.RuntimePermission "setContextClassLoader";
+  // needed for SPI class loading
+  permission java.lang.RuntimePermission "accessDeclaredMembers";


Can we reduce the scope of this permission?

Not easily, because it's needed by the PluginsService (which used to farm this requirement out to lucene but can't any longer because lucene has removed the class). @rjernst and I have talked very briefly about it, and I see that you've added it to the follow-up list on #74057

romseygeek · 2021-09-21T08:06:51Z

I looked through with a focus on the new vector format. I left minor comments and also noticed we need to update LuceneFilesExtensions to contain the vector index file .vex.

It'd also be good to extend the disk usage analyzer to cover vectors, but this seems best as a follow-up.

I think both of these come under the disk usage and field usage actions in #74057

Upgrade to lucene-9.0.0-snapshot-a12260eb950

5d57ce4

romseygeek added >enhancement :Analytics/Aggregations Aggregations :Search/Search Search-related issues that do not fall into other categories :Distributed/Engine Anything around managing Lucene and the Translog in an open shard. v8.0.0 labels May 24, 2021

romseygeek self-assigned this May 24, 2021

elasticmachine added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:Search Meta label for search team Team:Distributed Meta label for distributed team. labels May 24, 2021

romseygeek added 2 commits June 10, 2021 13:56

Merge remote-tracking branch 'origin/master' into lucene-9-upgrade

2b3cc5b

upgrade to new snapshot; precommit happy

a83cff2

romseygeek mentioned this pull request Jun 14, 2021

Upgrade to lucene 9 #74057

Closed

16 tasks

joshdevins mentioned this pull request Jul 9, 2021

Investigate various implementations of ann search for vector fields #42326

Closed

jpountz and others added 5 commits July 13, 2021 16:28

Merge branch 'master' into lucene-9-upgrade

3af9f65

More compile fixes.

eef74fc

Undo unnecessary change.

3e8ab5c

Merge remote-tracking branch 'elastic/master' into lucene-9-upgrade

449050b

fix compile

44c6f49

romseygeek added 4 commits July 30, 2021 11:08

Merge remote-tracking branch 'origin/master' into lucene-9-upgrade

6e7b433

compilation + precommit

3e8da46

small updates

2dc6a3d

Merge remote-tracking branch 'romseygeek/lucene-9-upgrade' into lucen…

e13aeb2

…e-9-upgrade

iverase approved these changes Sep 20, 2021

View reviewed changes

romseygeek and others added 5 commits September 20, 2021 15:06

No need for endianness shenanigans in TransportSearchHelper

46fc8c1

Merge branch 'master' into lucene-9-upgrade

f79933e

Remove NORELEASE

cd50069

Do not remove max_merge_at_once_explicit

a4e6b32

use existing filtered store fields visitor

e53d3be

jtibshirani approved these changes Sep 20, 2021

View reviewed changes

dnhatn added 8 commits September 20, 2021 14:16

minimize changes

7790565

Remove ram tree

1cbf2cd

delay expose analyzers

251cb8b

stylecheck

e5609d0

remove japanese_completion

4fcb718

sort optimization

9d7e27f

precommit

b3b57de

fix unknown analyzer

16b26da

dnhatn approved these changes Sep 21, 2021

View reviewed changes

Merge remote-tracking branch 'origin/master' into lucene-9-upgrade

fcbf2b4

deef

b9d9c26

romseygeek changed the title ~~Upgrade to lucene-9.0.0-snapshot-a12260eb950~~ Upgrade to lucene-9.0.0-snapshot-32a0a16aff0 Sep 21, 2021

romseygeek merged commit 524d1ea into elastic:master Sep 21, 2021

romseygeek deleted the lucene-9-upgrade branch September 21, 2021 09:48

iverase removed the :Analytics/Aggregations Aggregations label Sep 27, 2021

elasticmachine removed the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Sep 27, 2021

This was referenced Sep 27, 2021

Don't use endianness reverse util for writing KeyStore file #78304

Merged

Don't use endianness reverse util for writing metadata state file #78309

Merged

wwang500 mentioned this pull request Oct 6, 2021

Hit java.lang.IndexOutOfBoundsException in es log after recent Lucene9 snapshot upgrade #78785

Closed

jakelandis added v8.0.0-beta1 and removed v8.0.0 labels Oct 27, 2021


		}

		private static class FilterStoredFieldVisitor extends StoredFieldVisitor {

Conversation

romseygeek commented May 24, 2021

Uh oh!

elasticmachine commented May 24, 2021

Uh oh!

elasticmachine commented May 24, 2021

Uh oh!

elasticmachine commented May 24, 2021

Uh oh!

romseygeek commented May 24, 2021

Uh oh!

dnhatn commented May 25, 2021

Uh oh!

iverase commented Jun 1, 2021

Uh oh!

dnhatn commented Jul 16, 2021

Uh oh!

iverase left a comment

Choose a reason for hiding this comment

Uh oh!

jtibshirani left a comment

Choose a reason for hiding this comment

Uh oh!

jtibshirani Sep 20, 2021

Choose a reason for hiding this comment

Uh oh!

jtibshirani Sep 20, 2021

Choose a reason for hiding this comment

Uh oh!

dnhatn left a comment

Choose a reason for hiding this comment

Uh oh!

dnhatn Sep 20, 2021

Choose a reason for hiding this comment

Uh oh!

romseygeek Sep 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

romseygeek commented Sep 21, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

romseygeek Sep 21, 2021 •

edited

Loading