External Library Models Integration#922
Conversation
…o library_model_stub_abhijitk
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #922 +/- ##
============================================
- Coverage 87.15% 86.08% -1.07%
- Complexity 2011 2018 +7
============================================
Files 78 79 +1
Lines 6522 6612 +90
Branches 1265 1280 +15
============================================
+ Hits 5684 5692 +8
- Misses 422 510 +88
+ Partials 416 410 -6 ☔ View full report in Codecov by Sentry. |
msridhar
left a comment
There was a problem hiding this comment.
A few quick comments to start
lib-model/lib-model-consume/src/main/java/com/uber/nullaway/libmodel/LibModelInfoExtractor.java
Outdated
Show resolved
Hide resolved
lib-model/lib-model-consume/src/main/java/com/uber/nullaway/libmodel/LibModelInfoExtractor.java
Outdated
Show resolved
Hide resolved
...odel/lib-model-consume/src/main/java/com/uber/nullaway/libmodel/MethodAnnotationsRecord.java
Outdated
Show resolved
Hide resolved
lib-model/lib-model-consume/src/main/java/com/uber/nullaway/libmodel/StubxFileWriter.java
Outdated
Show resolved
Hide resolved
lib-model/lib-model-consume/src/main/java/com/uber/nullaway/libmodel/LibModelInfoExtractor.java
Outdated
Show resolved
Hide resolved
lib-model/lib-model-consume/src/main/java/com/uber/nullaway/libmodel/LibModelInfoExtractor.java
Outdated
Show resolved
Hide resolved
lib-model/lib-model-consume/src/main/java/com/uber/nullaway/libmodel/LibModelInfoExtractor.java
Outdated
Show resolved
Hide resolved
…bmodel/LibModelInfoExtractor.java Co-authored-by: Manu Sridharan <msridhar@gmail.com>
lib-model/lib-model-consume/src/main/java/com/uber/nullaway/libmodel/LibModelInfoExtractor.java
Outdated
Show resolved
Hide resolved
…o library_model_stub_abhijitk
|
At a high level I'd like to do some renaming of the new modules. Let's use
|
msridhar
left a comment
There was a problem hiding this comment.
Overall this looks great now! I have one more comment which should be an easy fix. I think the remaining high-level question, for me, is whether we should rename classes related to JarInfer inside NullAway (since they now serve a more general purpose), and whether we should add extra command-line flags instead of just having JarInferEnabled. I think we can fix those things in a follow-up PR but we should decide what we want to do.
Otherwise, I'm going to ask @lazaroclapp to review this PR as he originally wrote the stubx generation code.
| if (a.getNameAsString().equalsIgnoreCase(NULL_MARKED)) { | ||
| this.isNullMarked = true; | ||
| } |
There was a problem hiding this comment.
This logic won't handle cases where @NullMarked appears on some nested classes but not others. But, I think for now, we can just say we only support @NullMarked on the top-level class. Can we add a comment to this effect?
There was a problem hiding this comment.
It also needs @NullMarked on the same file, correct? We don't have a way to add this to the package somewhere and have the rest of the traversal be aware of that (that seems more likely than mixed marked/unmarked code in the models, but I am not sure)
There was a problem hiding this comment.
Yes, you're correct, @lazaroclapp, we are assuming an explicit @NullMarked on the top-level class in a source file, and that there is only one top-level class per source file. We won't find anything on package-info.java or module-info.java for now. Both these assumptions hold for jspecify/jdk
There was a problem hiding this comment.
@akulk022 can you update the comment in light of the above?
lazaroclapp
left a comment
There was a problem hiding this comment.
Did a pass. Had a bunch of comments, but they are mostly questions and notes for the future. Overall this looks good to me.
| import com.google.common.collect.ImmutableSet; | ||
|
|
||
| /** A record describing the annotations associated with a java method and its arguments. */ | ||
| @AutoValue |
There was a problem hiding this comment.
Are we otherwise using @AutoValue for something or what's the advantage of it here?
There was a problem hiding this comment.
Sorry this is my fault. I thought this was new code and thought that @AutoValue made it a bit cleaner. I didn't realize it had been moved from somewhere else. @lazaroclapp would you like us to undo the @AutoValue-ization of this file?
There was a problem hiding this comment.
Never mind, seems we have been using @AutoValue in at least some parts of the codebase, since the beginning, so I am fine having it here too, at least until we can do JDK 17 records.
| @@ -0,0 +1,43 @@ | |||
| /* | |||
| * Copyright (C) 2017. Uber Technologies | |||
There was a problem hiding this comment.
Not that I care about this anymore, but I suspect Uber would want these copyright headers to be updated for the right years 😉 (e.g. 2024 here and 20[...]-2024 for the files that were altered, I think)
| */ | ||
| public static void main(String[] args) { | ||
| LibraryModelGenerator libraryModelGenerator = new LibraryModelGenerator(); | ||
| libraryModelGenerator.generateAstubxForLibraryModels(args[0], args[1]); |
There was a problem hiding this comment.
Check the number of args and print a basic usage message? (Not sure if we do that for JarInfer, and I don't think we need a full argument parser here yet, a check on the length of args would suffice for now)
| "-XepOpt:NullAway:JarInferEnabled=true", | ||
| "-XepOpt:NullAway:JarInferUseReturnAnnotations=true")) |
There was a problem hiding this comment.
I agree with @yuxincs that overloading the meaning of the JarInfer[...] CLI options for NullAway for this is bad practice, specially because other than the Android SDK, JarInfer no longer uses .astubx but actually modifies the bytecode of jars to add annotations (so this options aren't used with JarInfer except for that Android SDK jar).
It can certainly be a follow up PR, but I'd be in favor of rethinking these CLI flags. Maybe -XepOpt:NullAway:JarInferEnabled is actually something like -XepOpt:NullAway:ScanClassPathForAstubXModels? Is there a better name? A nicer way to specify which .astubx files we want to load without breaking things in our build internally? (I think there is! We only really care about that Android SDK astubx file right now, so it's not like this changes for each target like it used to, cc: @yuxincs )
There was a problem hiding this comment.
I am in favor of changing these, but maybe in a follow-up PR. I don't think we need to go as far as keeping the old flag but deprecating it, as long as we document properly in release notes. But we should probably check as to what this breaks (if anything) internally at Uber.
@akulk022 can you open an issue on renaming these flags?
| return (annotation.getNameAsString().equalsIgnoreCase(NULLABLE) | ||
| && this.isJspecifyNullableImportPresent) | ||
| || annotation.getNameAsString().equalsIgnoreCase(JSPECIFY_NULLABLE_IMPORT); | ||
| } |
There was a problem hiding this comment.
Do we care about star imports or other nonsense or is it fine here to only count either FQN or an import of the FQN?
There was a problem hiding this comment.
I think this logic is fine but we should add a comment that we don't consider star imports @akulk022
| Map<String, String> importedAnnotations = | ||
| ImmutableMap.of( | ||
| "Nonnull", "javax.annotation.Nonnull", | ||
| "Nullable", "javax.annotation.Nullable"); |
There was a problem hiding this comment.
Why are we using the javax annotations here and not JSpecify? Some limitation with StubxWriter?
There was a problem hiding this comment.
No limitation with StubxWriter 🙂, just used it because the check here considers the javax annotation and I probably didn't consider adding an OR and including the jspecify annotation in the condition. I've updated it.
| if (a.getNameAsString().equalsIgnoreCase(NULL_MARKED)) { | ||
| this.isNullMarked = true; | ||
| } |
There was a problem hiding this comment.
It also needs @NullMarked on the same file, correct? We don't have a way to add this to the package somewhere and have the rest of the traversal be aware of that (that seems more likely than mixed marked/unmarked code in the models, but I am not sure)
| String parentClassName = ""; | ||
| if (parentClassNode.isPresent()) { | ||
| if (parentClassNode.get() instanceof ClassOrInterfaceDeclaration) { | ||
| parentClassName = ((ClassOrInterfaceDeclaration) parentClassNode.get()).getNameAsString(); |
There was a problem hiding this comment.
This is just the simple name, right? So currently this breaks when dealing with nested/inner classes? i.e. the concatenation below:
packageName
+ "."
+ parentClassName
+ ":"
+ getMethodReturnTypeString(md)
[...]
Ends up producing com.example.Bar:String baz() for:
@NullMarked
class Foo {
public class Bar {
@Nullable public String baz() {...}
}
}
correct?
There was a problem hiding this comment.
Yes, that is correct. I'll look into making this work.
There was a problem hiding this comment.
@akulk022 is this easy to fix? Or should it wait for a follow up?
There was a problem hiding this comment.
@msridhar I'm working on a fix since it seems like it should be easy enough.
There was a problem hiding this comment.
@akulk022 be sure to add a test for it as well
There was a problem hiding this comment.
@msridhar @lazaroclapp I've updated the logic to handle this scenario, I've also added a test for it. Does this work and are there more scenarios I should test?
| */ | ||
| private String getMethodReturnTypeString(MethodDeclaration md) { | ||
| if (md.getType() instanceof ClassOrInterfaceType) { | ||
| return md.getType().getChildNodes().get(0).toString(); |
There was a problem hiding this comment.
Note that using simple names is something we inherited from the CF stub format + us not explicitly handling imports, I think. So we might want to change these to FQNs at some point (but probably not in this PR)
Co-authored-by: Lázaro Clapp <lazaro.clapp@gmail.com>
…Handler.java and updating some comments
…erEnabled and JarInferUseReturnAnnotations flags.
…o library_model_stub_abhijitk
… defined in the jar's manifest
| private static class AnnotationCollectionVisitor extends VoidVisitorAdapter<Void> { | ||
|
|
||
| private String packageName = ""; | ||
| private StringBuilder parentName; |
There was a problem hiding this comment.
Probably better to just make this a String? I think making it a StringBuilder is an unnecessary optimization and makes the code more brittle. You can just overwrite it with new String values and I think it'll be fine
…o library_model_stub_abhijitk
lazaroclapp
left a comment
There was a problem hiding this comment.
LGTM! Definitely some potential future improvements are possible (and renaming some stuff so it's less tied to JarInfer), but as far as initial support for building JSpecify JDK astubx files, this is excellent!
|
Going to merge this now, and we will build on it in future PRs |
The newly added
library-modelmodule consists of a CLI process that takes an input directory with annotated java source files as a command line parameter and usescom.github.javaparserAPIS to generatelibmodels.astubxfile containing method stubs for methods that return @nullable. This can be run using the existingJarInferEnabledandJarInferUseReturnAnnotationsflags.This allows us to be able catch issues as shown in the below example from externally annotated source code: