Community
Participate
Working Groups
CQ 2729 is opened to update Lucene to the latest available in Orbit. We currently have 1.9.1 in the SDK and I am wondering if the UA team has bandwith to update to this version. Not sure if this has been planned but it's generally a good idea to update to the latest library each release.
https://dev.eclipse.org/ipzilla/show_bug.cgi?id=2729
Looks like Lucene 2.4 is coming out soon: http://www.nabble.com/draft-2.4-announcement-to19729760.html
This seems like a good idea and I don't expect that it would have any negative impact on the help system, however the UA team does not have any spare cycles right now. Are there any other consumers of Lucene other than UA?
I don't think so. However, I'll be willing to help with the move.
Chris G., can you file a CQ and I'll do the rest?
Chris, can you file a CQ for Lucene 2.4.0? It was just released: http://lucene.apache.org/java/2_4_0/changes/Changes.html
ping :)
Upgrading to version 2.4 sounds like a good idea, first I want to check to make sure that there is no downside to doing this, such as requiring a higher JRE version. I also want to make sure that the help system continues to work with the new version.
Ok, but without a CQ we are still dead in the water. There are also performance improvements in 2.4 that may even improve crawling the index.
If the CQ is filed, we can do some of this work in parallel.
Lucene 2.4 is still J2SE-1.4 so we're good there.
If this transition was relatively easy and had no downside I would be in favor of making the switch. I decided to download a copy of Lucene 2.4 and see if I could just switch the jar files and use in the Eclipse SDK. Here are the sizes of the jar files not including source Lucene 1.9.1 434K Lucene 2.4 800K Lucene Analyzers 1.9.1 69K Lucene Analyzers 2.4 141K We have one performance test which adds documents to the search index, org.eclipse.ua.tests.help.performance.BuildHtmlSearchIndex. There was no noticable difference in performance. One thing I noticed was that there were a number of deprecations, see attachment. These deprecations will require some recoding to eliminate them. At this point I have to ask the question - what is the main motivation for uopgrading to 2.4? Given that we are stretched thin on development resources and there is probably some recoding required to eliminate deprecations I want to make sure that there are clear benefits before I commit any time to this.
Created attachment 115465 [details] List of deprecation warnings when using Lucene 2.4
Copying McQ
From the above, it appears to be the case that the new Lucene is significantly larger, and does not perform better for our use cases. That certainly doesn't argue strongly in favor of upgrading. The remaining issues are: 1) If we didn't upgrade, what would the impact be on our consumers? I.e. are there others who rely on the Lucene that we provide, who would like to be able to move to the new one? 2) Is Lucene development proceeding in a way that would likely cause us to need to move up to a new version in the future? (In which case, working through the deprecations now might be worth the effort.)
(In reply to comment #15) > 1) If we didn't upgrade, what would the impact be on our consumers? I.e. are > there others who rely on the Lucene that we provide, who would like to be able > to move to the new one? Yes, there are some projects in Technology and RT that want to use a newer Lucene: https://dev.eclipse.org/ipzilla/show_bug.cgi?id=2725 https://dev.eclipse.org/ipzilla/show_bug.cgi?id=2556 > 2) Is Lucene development proceeding in a way that would likely cause us to need > to move up to a new version in the future? (In which case, working through the > deprecations now might be worth the effort.) It seems Lucene development is pretty active. The library they just recently release is 2.4.0 and the one we are on 1.9.1 was released 2006-03-02. There were five releases between what we use currently and what just came out.
I asked for (further) PMC feedback on eclipse-pmc. Did we resolve whether or not this impacts the execution environment (i.e. are there new JRE dependencies, etc.)?
(In reply to comment #12) > We have one performance test which adds documents to the search index, > org.eclipse.ua.tests.help.performance.BuildHtmlSearchIndex. There was no > noticable difference in performance. It would be a good idea to scan release notes since 1.9.1 release and maybe update code to pickup performance improvements implemented in recent Lucene releases. There is also an interesting feature that allows to compress certain index fields, which I find extremely useful when indexing large documents. Also, for what it worth, technology.m2e project will also need to use recent Lucene core, but we haven't submitted CQs yet.
(In reply to comment #17) > I asked for (further) PMC feedback on eclipse-pmc. Thanks McQ. > Did we resolve whether or not this impacts the execution environment (i.e. are > there new JRE dependencies, etc.)? There are no impacts. Lucene 2.4 is still J2SE-1.4 so we're good there.
If the PMC approves, can we get this on the plan :)?
I still haven't seen any evidence that there is any advantage of moving up. It is twice the size, and doesn't seem faster for our use cases. Other projects wanting 2.4 isn't a good enough reason by itself IMO - since this isn't a singleton bundle our choice of version doesn't impact anyone. We're not Orbit...
(In reply to comment #21) > I still haven't seen any evidence that there is any advantage of moving up... One thing to note is that not all Platform plugins explicitly specify Lucene version (e.g. org.eclipse.ui.cheatsheets) and it is unclear if that would cause issues if there is more then one instance of Lucene is available at runtime, especially because there is several non-backward compatible changes in the binary index format since version 1.9.1. Also, I am not buying an argument in regards "twice as size". Maybe for the raw Platform that would be the case, but in many cases users use more then the raw Platform, so they would end up with old + new Lucene version and in result even more then "twice as size".
(In reply to comment #22) > Also, I am not buying an argument in regards "twice as size". Maybe for the raw > Platform that would be the case, but in many cases users use more then the raw > Platform, so they would end up with old + new Lucene version and in result even > more then "twice as size". That's one of my major concerns also. Once the other projects in the Eclipse ecosystem use Lucene and start complaining that the version in the Platform is old and not meeting their needs (as evident in some CQs). We'll end up with duplicate versions for no reason. People generally look at the Platform for leading things. Also, it doesn't make us look awesome for having a library that is 2+ years old that has rapidly evolved since than.
Regarding performance there is only one performance test in the UA test suite relating to Lucene and that involves adding entries to the index. I don't have any data regarding the performance of the search itself, which would be another useful datapoint. I agree that in general it is better to move up to newer versions of third party libraries but I also think it is important for the UA team to work on higher priority bugs first and then see if there is time available to make this change.
If upgrading doesn't fix any problem or add any new features, why would we do it given that it causes work? Are we holding other components back by not upgrading?
Lucene 2.4 does fix a lot of problems and adds many new features. However, I think this depends pretty much on the perspective if those are direct benefits for Eclipse or not. I also wouldn't rely on one test case to judge whether an upgrade is justified. The Lucene committers and community should have far more tests and comments to form a better opinion. Even if there are no benefits for the platform one has to acknowledge that Lucene is also exposed and used outside the platform's regular use cases. I'm thinking of the help system which can run stand alone and is powering some large scale documentation sites. Those will definitely appreciate the benefits of an upgrade in a different way than the platform does. FWIW, a lot enhancements are actually scalability and stability improvements such as no index corruption on OS/machine crash, reduced lock contention on index access and optimization for read-only access for higher concurrency. Those are benefits which a platform user using embedded help doesn't benefit as much as an admin maintaining a help system.
FYI, Lucene 2.4 is now in Orbit if you're interested in using it (there would still be a need for a CQ to be filed as a request for reuse). I'll see what I can do about a patch for the deprecated calls. Than we can make a decision on including it in the SDK.
FWIW, I did a query for "search" in Bugzilla in Eclipse - Platform - User Assistance - and Status anything but Resolved/Verified/Closed, to see if there seemed to be any open items that Lucene 2.4 might help with. The only ones that seem remotely related to Lucene level (for example, parser related) seem to be: https://bugs.eclipse.org/bugs/show_bug.cgi?id=53143 https://bugs.eclipse.org/bugs/show_bug.cgi?id=175116 https://bugs.eclipse.org/bugs/show_bug.cgi?id=219928 https://bugs.eclipse.org/bugs/show_bug.cgi?id=120913 https://bugs.eclipse.org/bugs/show_bug.cgi?id=107683 https://bugs.eclipse.org/bugs/show_bug.cgi?id=80792 175116 is "Move lucene demo code to Orbit". Is that related to the Orbit item in comment #27?
Created attachment 117561 [details] org.eclipse.help.base.patch First stab at a patch to remove some of the deprecated warnings. I also add Thai as one of the lucene analyzers since it seems to be new. If anyone else is more familiar with Lucene and wants to take a stab at things, please do. Also Chris G., where are the tests for Help?
Created attachment 117562 [details] org.eclipse.help.base.patch Let's try that patch again
Chris A, we discussed this at one of the Eclipse architecture meetings which you missed. The current thinking is that we have too many demands on our limited resources to want to commit to what appears to be a low priority change. Maybe you can reraise the topic at a future call.
Well that's not fair if I wasn't on the call :) In the end, I don't see how this is a problem given that: 1) there's a patch that removes about half the deprecated warnings and things still work 2) I don't want to see two copies of Lucene in Galileo because the Platform didn't want to upgrade It would be nice to be able to run the tests, but I can't seem to find the tests plug-in. Any help on that front?
there is probably a must or should do in the Galileo release plan that covers this.
The Galileo must do's only cover multiple copies of the same version of a bundle: "the final Galileo release will not have duplicate third-party libraries (note that this only applies to identical versions of the libraries; thus if project A requires foo.jar 1.6 and project B uses foo.jar 1.7, that's ok)."
fair enough. My bias is still to move to the latest lucene baring any significant perfromance or other costs.
I'm planning on upgrading to a newer Lucene for 3.6M4. What is the version I should be targeting, is it still Lucene 2.4? Question for Chris A - do I need to file a CQ?
Yes, you need a CQ for the Eclipse project's consumption of the Orbit bundle. Our CQ would just piggyback on the Orbit CQ and is typically approved immediately.
(In reply to comment #36) > I'm planning on upgrading to a newer Lucene for 3.6M4. What is the version I > should be targeting, is it still Lucene 2.4? 2.9 is the latest as of 2009-09-25.
2.4 is the latest version approved, I filed a CQ for 2.9 http://dev.eclipse.org/ipzilla/show_bug.cgi?id=3583 2.9 looks nice: http://lucene.apache.org/java/docs/#25+September+2009+-+Lucene+Java+2.9.0+available It has new analyzers that may be useful too (PersianAnalyzer, ArabicAnalyzer, SmartChineseAnalyzer)
BTW, when updating to a new version you may want to take into consideration that new Lucene versions can have different binary index format which can't be read by older versions, e.g. you can't share the same workspace between Eclipse versions that uses old and new Lucene. So, if this format incompatibility isn't handled transparently, it need to be fixed.
(In reply to comment #40) > BTW, when updating to a new version you may want to take into consideration > that new Lucene versions can have different binary index format which can't be > read by older versions, e.g. you can't share the same workspace between Eclipse > versions that uses old and new Lucene. So, if this format incompatibility isn't > handled transparently, it need to be fixed. You can't share workspaces between Eclipse versions in general. Various kinds of changes happen in the metadata across releases that might not be readable by older versions. This might happen to work in some cases but it's not something we attempt to support.
Created attachment 153546 [details] Updated patch This updates Chris A's previous patch to be current with the latest code in HEAD.
When I use the API tooling, I see the errors below when using Luceen 2.4. This is because org.apache.lucene and org.apache.lucene.analysis are exported by org.eclipse.help.base. I'm not sure how to get around this problem because the reexport has turned Lucene into part of the API for org.eclipse.help.base. The major version should be incremented in version 3.4.100.qualifier, since API breakage occurred since version 3.4.0.v200906111540 MANIFEST.MF /org.eclipse.help.base/META-INF line 5 Version Numbering Problem The re-exported type org.apache.lucene.analysis.de.WordlistLoader has been removed from org.eclipse.help.base_3.4.0 MANIFEST.MF /org.eclipse.help.base/META-INF line 1 Compatibility Problem The re-exported type org.apache.lucene.analysis.standard.CharStream has been removed from org.eclipse.help.base_3.4.0 MANIFEST.MF /org.eclipse.help.base/META-INF line 1 Compatibility Problem The re-exported type org.apache.lucene.analysis.standard.FastCharStream has been removed from org.eclipse.help.base_3.4.0 MANIFEST.MF /org.eclipse.help.base/META-INF line 1 Compatibility Problem The re-exported type org.apache.lucene.analysis.standard.ParseException has been removed from org.eclipse.help.base_3.4.0 MANIFEST.MF /org.eclipse.help.base/META-INF line 1 Compatibility Problem The re-exported type org.apache.lucene.analysis.standard.StandardTokenizerConstants has been removed from org.eclipse.help.base_3.4.0 MANIFEST.MF /org.eclipse.help.base/META-INF line 1 Compatibility Problem The re-exported type org.apache.lucene.analysis.standard.StandardTokenizerTokenManager has been removed from org.eclipse.help.base_3.4.0 MANIFEST.MF /org.eclipse.help.base/META-INF line 1 Compatibility Problem The re-exported type org.apache.lucene.analysis.standard.Token has been removed from org.eclipse.help.base_3.4.0 MANIFEST.MF /org.eclipse.help.base/META-INF line 1 Compatibility Problem The re-exported type org.apache.lucene.analysis.standard.TokenMgrError has been removed from org.eclipse.help.base_3.4.0 MANIFEST.MF /org.eclipse.help.base/META-INF line 1 Compatibility Problem The re-exported type org.apache.lucene.index.SegmentTermPositionVector in org.eclipse.help.base_3.4.0 is no longer API MANIFEST.MF /org.eclipse.help.base/META-INF line 1 Compatibility Problem The re-exported type org.apache.lucene.search.DateFilter has been removed from org.eclipse.help.base_3.4.0 MANIFEST.MF /org.eclipse.help.base/META-INF line 1 Compatibility Problem The re-exported type org.apache.lucene.search.DisjunctionSumScorer in org.eclipse.help.base_3.4.0 is no longer API MANIFEST.MF /org.eclipse.help.base/META-INF line 1 Compatibility Problem The re-exported type org.apache.lucene.search.PhrasePrefixQuery has been removed from org.eclipse.help.base_3.4.0 MANIFEST.MF /org.eclipse.help.base/META-INF line 1 Compatibility Problem The re-exported type org.apache.lucene.store.InputStream has been removed from org.eclipse.help.base_3.4.0 MANIFEST.MF /org.eclipse.help.base/META-INF line 1 Compatibility Problem The re-exported type org.apache.lucene.store.OutputStream has been removed from org.eclipse.help.base_3.4.0 MANIFEST.MF /org.eclipse.help.base/META-INF line 1 Compatibility Problem
The current state of this bug is as follows: There is a CQ (3583) open to get IP approval for Lucene 2.9, this is currently help up waiting for a resolution on SorterTemplate.jave, if we removed this we may be able to resolve the CQ. Meanwhile Lucene 2.4 is already in Orbit and has been through IP review. I'm not sure if 2.9 has any must have features as compared to 2.4, it claims to be faster. Note that the Lucene version jumped from 2.4 to 2.9, so there is no version 2.5, 2.6, 2.7 or 2.8. There is also a Lucene 3.0 available but I would not recommend that we consider switching to that any time soon since it will have a lot of API changes, including removing elements which were deprecated in Lucene 2.x, some of which we have not yet coded around. Also Lucene 3.0 requires Java 5 or higher. Lucene 2.4 is a relatively easy port for the help system and I have verified that it does build the search index faster, the performance test BuildHTMLSearchIndex is 30% faster. The main problem is that the reexport of org.apache.lucene by org.eclipse.help causes API compatibility errors due to changes in the classes exported by the lucene bundles. The API class /org.eclipse.help.base/src/org/eclipse/help/search/LuceneSearchParticipant.java uses org.apache.lucene.document.Document org.apache.lucene.document.Field in public methods, which may explain why the reexport was initially added. One reason to switch to 2.4 is that other projects within Helios are already using it or want to use it. I think at this stage we should drop the idea of getting Lucene 2.9 into Helios since it is going to require time that we do not have to work this new version through the IP process. I also think that we should move ahead with switching to Lucene 2.4. John can you comment on how to deal with the API breakage due to the handful of classes that are no longer reexported. The API tool is telling me to upgrade the major version of org.eclipse.help.base, that seems like it would cause more problems than it would solve.
(In reply to comment #44) > John can you comment on how to deal with the API breakage due to > the handful of classes that are no longer reexported. The API tool is telling > me to upgrade the major version of org.eclipse.help.base, that seems like it > would cause more problems than it would solve. This is due to our old misguided practice of always reexporting bundles whose classes we expose in our API. This was never a good idea, and for third party libraries it's just disastrous because it means we are making promises about code we don't own or control. I think we should do the following when we move to Lucene 2.x: - Remove the re-export clause for our required lucene bundles - Remove the package export for org.apache.lucene.demo.html (I don't even see where this package is declared) - Notify we are making this change on eclipse-dev and cross-project-issues-dev (both the platform move to Lucene 2.x and the removal of re-exports) - Document the change in our platform migration guide. It would say something like "Bundles that reference Lucene API should add an explicit dependency on lucene rather than relying on the implicit dependency via org.eclipse.help.base. - Increment the minor version segment for org.eclipse.help.base. I could buy the argument that we should increase the major version, but like Chris I think this introduces more problems than it solves. The help API itself is not changing in an incompatible way, and only a small number of clients of help who use lucene without depending on it directly are affected. This is not a binary-compatible change, but I think we are early enough in the Helios cycle that we can make this change. Downstream clients who use lucene but don't have a dependency just need to add their own dependency to adapt to this change.
Is this planned to be announced soon? I believe Lucene 2.9 should be in Orbit now.
DJ is working on getting the bits into Orbit, see Bug 297617. Meanwhile I'm studying the API changes to see whether upgrading to Lucene 2.x will be likely to break any clients of the help system, some of the API classes in the help system have methods with parameters of type org.apache.lucene.document.Document. Referencing a Lucene class in this way could cause a reexport of binary incompatibility, but I need to study this some more. I would like to get the new Lucene being used by the help system early January 2010 if possible.
(In reply to comment #46) > Is this planned to be announced soon? > > I believe Lucene 2.9 should be in Orbit now. zx, are you doing this? (see https://dev.eclipse.org/ipzilla/show_bug.cgi?id=3635#c4)
I've just been looking into the impact on clients of switching to Lucene 2.9 and because org.eclipse.help.base exposes Lucene classes in the API this is not a benign change, see Bug 298510. I'm leaning towards making the change for Eclipse 4.0 and not for 3.6 but I would be interested in getting more opinions on that from John A, McQ or others.
Removing the target milestone because upgrading to Lucene 2.x is not binary compatible and any client of the extension point org.eclipse.help.base.luceneSearchParticipants would be affected by this. I have taken the first step to extricating ourselves from this swamp which is to create a new API that does not expose Lucene classes and deprecate org.eclipse.help.base.luceneSearchParticipants and the classes it uses. See Bug 298510 for details. This still leaves us in the situation where for Eclipse 3.x we have to wait for users to have time to respond to the deprecation before we introduce a change which is binary incompatible for users of those deprecated classes. The other related problem is that org.eclipse.help.base reexports org.apache.lucene and org.apache.lucene.analysis. I'm still not sure whether it is appropriate to introduce the new Lucene in 4.0 but its clear that the binary incompatibility prevents this change from being made in Eclipse 3.6.
The latest 2.x Lucene is 2.9 and I am planning to move to 2.9 in Eclipse 3.7.
Created attachment 182335 [details] Patch for API filters This patch suppresses API errors which originate from the reexported API classes in Lucene when it is upgraded to Eclipse 2.9. See the previous comments for a discussion on why we need to do this if we don't want to be forever tied to Lucene 1.x. I have committed this patch to HEAD
Lucene 2.9.1 is now in build N20100928-2000. Closing as Fixed
