Do not compute name hashes for private members.#2325
Conversation
Add a pending test that shows a bug in calculation of name hashes. By design, only non-private members should contribute to name hashes. The test shows that name hashes are calcuclated for strictly private members too.
Mention that private members are being extracted and included in the api structures but ignored in many other parts of incremental compiler. I've made a mistake of assuming that private members are ignored at api extraction time. This manifested itself as bug sbt#2324.
There was a problem hiding this comment.
What's p.qualifier for private[this]?
There was a problem hiding this comment.
Ah, I see, it's not an IdQualifier
|
Can one of the admins verify this patch? |
|
Looks like spurious failure? |
|
Indeed. Restarted the 2 jobs |
|
This causes a weird while compiling the following Java file: |
The crash manifested as a `NotSerializableException`, because we don't visit the private subtrees of the API graph, which would previously force all Lazy stubs as a side-effect. There was a backstop in `AbstractLazy` (`writeReplace`), but somehow that didn't work. ``` java.io.NotSerializableException: sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl [...] at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329) at xsbt.api.SourceFormat$.writes(SourceFormat.scala:24) at xsbt.api.SourceFormat$.writes(SourceFormat.scala:15) at sbt.inc.TextAnalysisFormat21661anonfun.apply(TextAnalysisFormat.scala:329) at sbt.inc.TextAnalysisFormat21661anonfun.apply(TextAnalysisFormat.scala:329) at sbt.inc.TextAnalysisFormat21661anonfun at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.javr$.aggregate(TextAnalysisFormat.scala:18) at sbt.inc.TextAnalysisFormat$.objToString(TextAnalysisFormat.scala:329) at sbt.inc.TextAnalysisFormat21661anonfun.apply(TextAnalysisFormat.scala:213) at sbt.inc.TextAnalysisFormat21661anonfun.apply(TextAnalysisFormat.scala:213) at sbt.inc.TextAnalysisFormat21661anonfun21661writeMap.apply(TextAnalysisFormat.scala:381) at sbt.inc.TextAnalysisFormat21661anonfun21661writeMap.apply(TextAnalysisFormat.scala:377) at scala.collection.mutable.ResizableArray.foreach(Resizable at sbt.inc.TextAnalysisFormalection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at sbt.inc.TextAnalysisFormat$.sbt21661writeMap(TextAnalysisFormat.scala:377) at sbt.inc.TextAnalysisFormat$.write(TextAnalysisFormat.scala:216) at sbt.inc.TextAnalysisFormat21661anonfun.apply(TextAnalysisFormat.scala:64) at sbt.inc.TextAnalysisFormat21661anonfun.apply(TextAnalysisFormat.scala:64) at sbt.inc.TextAnalysisFormat21661anonfun.apply(TextAnalysisFormat.scala:64) at sbt.inc.FormatTimer$.aggregate(TextAnalysisFo at sbt.inc.TextAnalysisFormat21661anonfun.scala:25) at sbt.inc.TextAnalysisFormat$.write(TextAnalysisFormat.scala:64) at sbt.inc.FileBasedStore21661anon21661anonfun.apply(FileBasedStore.scala:12) at sbt.inc.FileBasedStore21661anon21661anonfun.apply(FileBasedStore.scala:12) at sbt.Using.apply(Using.scala:24) at sbt.inc.FileBasedStore21661anon.set(FileBasedStore.scala:12) at sbt.inc.AnalysisStore21661anon.set(AnalysisStore.scala:16) at sbt.inc.AnalysisStore21661anon.set(AnalysisStore.scala:27) at sbt.Defaults21661anonfun.apply(Defaults.scala:799) at sbt.Defaults21661anonfun at sbt.inc.TextAts.scala:793) at scala.Function121661anonfun.apply(Function1.scala:47) at sbt.21661anonfun21661u2219.apply(TypeFunctions.scala:40) at sbt.std.Transform21661anon.work(System.scala:63) at sbt.Execute21661anonfun21661anonfun.apply(Execute.scala:226) at sbt.Execute21661anonfun21661anonfun.apply(Execute.scala:226) at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17) at sbt.Execute.work(Execute.scala:235) at sbt.Execute21661anonfun.apply(Exe at sbt.Using.apply(Using.scala:24) mit.apply(Execute.scala:226) at sbt.ConcurrentRestrictions21661anon21661anonfun.apply(ConcurrentRestrictions.scala:159) at sbt.CompletionService21661anon.call(CompletionService.scala:28) at java.util.concurrent.FutureTask.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors.call(Executors.java:439) at java.util.concurrent.FutureTask.innerRun(FutureTask.java:303 at sbt.std.Transform21661anon.work(System(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:695) [error] (compile:compile) java.io.NotSerializableException: sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl [error] Total time: 2 s, completed Dec 21, 2015 12:05:19 PM ```
Fix for regression triggered by sbt#2325 Somehow the old way of implementing `writeReplace` didn't work. The fix is captured, indirectly, by the scripted test `source-dependencies/java-analysis-serialization-error`.
The crash manifested as a `NotSerializableException`, because we don't visit the private subtrees of the API graph, which would previously force all Lazy stubs as a side-effect. There was a backstop in `AbstractLazy` (`writeReplace`), but somehow that didn't work. ``` java.io.NotSerializableException: sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl [...] at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329) at xsbt.api.SourceFormat$.writes(SourceFormat.scala:24) at xsbt.api.SourceFormat$.writes(SourceFormat.scala:15) at sbt.inc.TextAnalysisFormat21661anonfun.apply(TextAnalysisFormat.scala:329) at sbt.inc.TextAnalysisFormat21661anonfun.apply(TextAnalysisFormat.scala:329) at sbt.inc.TextAnalysisFormat21661anonfun at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.javr$.aggregate(TextAnalysisFormat.scala:18) at sbt.inc.TextAnalysisFormat$.objToString(TextAnalysisFormat.scala:329) at sbt.inc.TextAnalysisFormat21661anonfun.apply(TextAnalysisFormat.scala:213) at sbt.inc.TextAnalysisFormat21661anonfun.apply(TextAnalysisFormat.scala:213) at sbt.inc.TextAnalysisFormat21661anonfun21661writeMap.apply(TextAnalysisFormat.scala:381) at sbt.inc.TextAnalysisFormat21661anonfun21661writeMap.apply(TextAnalysisFormat.scala:377) at scala.collection.mutable.ResizableArray.foreach(Resizable at sbt.inc.TextAnalysisFormalection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at sbt.inc.TextAnalysisFormat$.sbt21661writeMap(TextAnalysisFormat.scala:377) at sbt.inc.TextAnalysisFormat$.write(TextAnalysisFormat.scala:216) at sbt.inc.TextAnalysisFormat21661anonfun.apply(TextAnalysisFormat.scala:64) at sbt.inc.TextAnalysisFormat21661anonfun.apply(TextAnalysisFormat.scala:64) at sbt.inc.TextAnalysisFormat21661anonfun.apply(TextAnalysisFormat.scala:64) at sbt.inc.FormatTimer$.aggregate(TextAnalysisFo at sbt.inc.TextAnalysisFormat21661anonfun.scala:25) at sbt.inc.TextAnalysisFormat$.write(TextAnalysisFormat.scala:64) at sbt.inc.FileBasedStore21661anon21661anonfun.apply(FileBasedStore.scala:12) at sbt.inc.FileBasedStore21661anon21661anonfun.apply(FileBasedStore.scala:12) at sbt.Using.apply(Using.scala:24) at sbt.inc.FileBasedStore21661anon.set(FileBasedStore.scala:12) at sbt.inc.AnalysisStore21661anon.set(AnalysisStore.scala:16) at sbt.inc.AnalysisStore21661anon.set(AnalysisStore.scala:27) at sbt.Defaults21661anonfun.apply(Defaults.scala:799) at sbt.Defaults21661anonfun at sbt.inc.TextAts.scala:793) at scala.Function121661anonfun.apply(Function1.scala:47) at sbt.21661anonfun21661u2219.apply(TypeFunctions.scala:40) at sbt.std.Transform21661anon.work(System.scala:63) at sbt.Execute21661anonfun21661anonfun.apply(Execute.scala:226) at sbt.Execute21661anonfun21661anonfun.apply(Execute.scala:226) at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17) at sbt.Execute.work(Execute.scala:235) at sbt.Execute21661anonfun.apply(Exe at sbt.Using.apply(Using.scala:24) mit.apply(Execute.scala:226) at sbt.ConcurrentRestrictions21661anon21661anonfun.apply(ConcurrentRestrictions.scala:159) at sbt.CompletionService21661anon.call(CompletionService.scala:28) at java.util.concurrent.FutureTask.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors.call(Executors.java:439) at java.util.concurrent.FutureTask.innerRun(FutureTask.java:303 at sbt.std.Transform21661anon.work(System(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:695) [error] (compile:compile) java.io.NotSerializableException: sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl [error] Total time: 2 s, completed Dec 21, 2015 12:05:19 PM ```
Fix for regression triggered by sbt#2325 Somehow the old way of implementing `writeReplace` didn't work. The fix is captured, indirectly, by the scripted test `source-dependencies/java-analysis-serialization-error`.
|
@gkossakowski, I submitted a PR to your PR with a test & fix for a regression uncovered by your fix. |
Fix for regression triggered by sbt#2325 From https://docs.oracle.com/javase/7/docs/api/java/io/Serializable.html `writeReplace` must be protected for its override to be inherited. > `Serializable` classes that need to designate an alternative object to be used > when writing an object to the stream should implement this special method with > the exact signature: > `ANY-ACCESS-MODIFIER Object writeReplace() throws ObjectStreamException;` > > This `writeReplace` method is invoked by serialization if the method exists and > it would be accessible from a method defined within the class of the object > being serialized. Thus, the method can have `private`, `protected` and > `package-private` access. > > **Subclass access to this method follows java accessibility rules.** (Thanks to retronym for digging up the docs.) The fix is captured, indirectly, by the scripted test `source-dependencies/java-analysis-serialization-error`.
Fix for regression triggered by sbt#2325 -- apparently, the API visitor would force all the lazy stubs so the `AbstractLazy` `writeReplace` was not exercised. With the private subtrees being ignored, the visitor didn't force the lazy stub, and the `writeProtected` method was not inherited in `SafeLazy.Impl`. From https://docs.oracle.com/javase/7/docs/api/java/io/Serializable.html `writeReplace` must be protected for its override to be inherited. > `Serializable` classes that need to designate an alternative object to be used > when writing an object to the stream should implement this special method with > the exact signature: > `ANY-ACCESS-MODIFIER Object writeReplace() throws ObjectStreamException;` > > This `writeReplace` method is invoked by serialization if the method exists and > it would be accessible from a method defined within the class of the object > being serialized. Thus, the method can have `private`, `protected` and > `package-private` access. > > **Subclass access to this method follows java accessibility rules.** (Thanks to retronym for digging up the docs.) The fix is captured, indirectly, by the scripted test `source-dependencies/java-analysis-serialization-error`.
The crash manifested as a `NotSerializableException`, because we don't visit the private subtrees of the API graph, which would previously force all Lazy stubs as a side-effect. There was a backstop in `AbstractLazy` (`writeReplace`), but somehow that didn't work. ``` java.io.NotSerializableException: sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl [...] at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329) at xsbt.api.SourceFormat$.writes(SourceFormat.scala:24) at xsbt.api.SourceFormat$.writes(SourceFormat.scala:15) at sbt.inc.TextAnalysisFormat21661anonfun.apply(TextAnalysisFormat.scala:329) at sbt.inc.TextAnalysisFormat21661anonfun.apply(TextAnalysisFormat.scala:329) at sbt.inc.TextAnalysisFormat21661anonfun at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.javr$.aggregate(TextAnalysisFormat.scala:18) at sbt.inc.TextAnalysisFormat$.objToString(TextAnalysisFormat.scala:329) at sbt.inc.TextAnalysisFormat21661anonfun.apply(TextAnalysisFormat.scala:213) at sbt.inc.TextAnalysisFormat21661anonfun.apply(TextAnalysisFormat.scala:213) at sbt.inc.TextAnalysisFormat21661anonfun21661writeMap.apply(TextAnalysisFormat.scala:381) at sbt.inc.TextAnalysisFormat21661anonfun21661writeMap.apply(TextAnalysisFormat.scala:377) at scala.collection.mutable.ResizableArray.foreach(Resizable at sbt.inc.TextAnalysisFormalection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at sbt.inc.TextAnalysisFormat$.sbt21661writeMap(TextAnalysisFormat.scala:377) at sbt.inc.TextAnalysisFormat$.write(TextAnalysisFormat.scala:216) at sbt.inc.TextAnalysisFormat21661anonfun.apply(TextAnalysisFormat.scala:64) at sbt.inc.TextAnalysisFormat21661anonfun.apply(TextAnalysisFormat.scala:64) at sbt.inc.TextAnalysisFormat21661anonfun.apply(TextAnalysisFormat.scala:64) at sbt.inc.FormatTimer$.aggregate(TextAnalysisFo at sbt.inc.TextAnalysisFormat21661anonfun.scala:25) at sbt.inc.TextAnalysisFormat$.write(TextAnalysisFormat.scala:64) at sbt.inc.FileBasedStore21661anon21661anonfun.apply(FileBasedStore.scala:12) at sbt.inc.FileBasedStore21661anon21661anonfun.apply(FileBasedStore.scala:12) at sbt.Using.apply(Using.scala:24) at sbt.inc.FileBasedStore21661anon.set(FileBasedStore.scala:12) at sbt.inc.AnalysisStore21661anon.set(AnalysisStore.scala:16) at sbt.inc.AnalysisStore21661anon.set(AnalysisStore.scala:27) at sbt.Defaults21661anonfun.apply(Defaults.scala:799) at sbt.Defaults21661anonfun at sbt.inc.TextAts.scala:793) at scala.Function121661anonfun.apply(Function1.scala:47) at sbt.21661anonfun21661u2219.apply(TypeFunctions.scala:40) at sbt.std.Transform21661anon.work(System.scala:63) at sbt.Execute21661anonfun21661anonfun.apply(Execute.scala:226) at sbt.Execute21661anonfun21661anonfun.apply(Execute.scala:226) at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17) at sbt.Execute.work(Execute.scala:235) at sbt.Execute21661anonfun.apply(Exe at sbt.Using.apply(Using.scala:24) mit.apply(Execute.scala:226) at sbt.ConcurrentRestrictions21661anon21661anonfun.apply(ConcurrentRestrictions.scala:159) at sbt.CompletionService21661anon.call(CompletionService.scala:28) at java.util.concurrent.FutureTask.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors.call(Executors.java:439) at java.util.concurrent.FutureTask.innerRun(FutureTask.java:303 at sbt.std.Transform21661anon.work(System(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:695) [error] (compile:compile) java.io.NotSerializableException: sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl [error] Total time: 2 s, completed Dec 21, 2015 12:05:19 PM ```
Fix for regression triggered by sbt#2325 -- apparently, the API visitor would force all the lazy stubs so the `AbstractLazy` `writeReplace` was not exercised. With the private subtrees being ignored, the visitor didn't force the lazy stub, and the `writeProtected` method was not inherited in `SafeLazy.Impl`. From https://docs.oracle.com/javase/7/docs/api/java/io/Serializable.html `writeReplace` must be protected for its override to be inherited. > `Serializable` classes that need to designate an alternative object to be used > when writing an object to the stream should implement this special method with > the exact signature: > `ANY-ACCESS-MODIFIER Object writeReplace() throws ObjectStreamException;` > > This `writeReplace` method is invoked by serialization if the method exists and > it would be accessible from a method defined within the class of the object > being serialized. Thus, the method can have `private`, `protected` and > `package-private` access. > > **Subclass access to this method follows java accessibility rules.** (Thanks to retronym for digging up the docs.) The fix is captured, indirectly, by the scripted test `source-dependencies/java-analysis-serialization-error`.
…-members Test & fix for regression in Lazy serialization uncovered by sbt#2325
|
@eed3si9n, your worry about preserving treatment of private members in traits is warranted. I'll settle it with adding a functional (scripted) test for it. |
|
I realized there's a scripted test for private members in traits and it's been marked as passing by #2160. We're good here. I think we're good to go. |
Add a test that shows the scenario described in sbt#2324 is handled correctly now.
…sakowski/sbt into nameHashing/private-members
|
cadacy failure seems to be false positive. LGTM |
Do not compute name hashes for private members.
The NameHashing classes assumed that extracted API data structure have private members filtered out already. That assumption was wrong and resulted in bug #2324.
We fix the bug by simply reusing the same logic as used by SameAPI class.
Fixes #2324.