Skip to content

Java: test GeneratedVsManualCoverage query on top 500 JDK APIs#12680

Merged
jcogs33 merged 10 commits intogithub:mainfrom
jcogs33:jcogs33/metrics-query-refactor-top500
Apr 6, 2023
Merged

Java: test GeneratedVsManualCoverage query on top 500 JDK APIs#12680
jcogs33 merged 10 commits intogithub:mainfrom
jcogs33:jcogs33/metrics-query-refactor-top500

Conversation

@jcogs33
Copy link
Copy Markdown
Contributor

@jcogs33 jcogs33 commented Mar 27, 2023

Description

This PR updates the metrics query that was originally added in #11585 to run on a specified subset of APIs and adds a test case for the top JDK APIs.

Consideration

Let me know if there's a better place to put the TopJdkApis.qll and GeneratedVsManualCoverageQuery.qll files. I wanted to put them both in the lib directory, but it seems that I can't import utils.modelgenerator.internal.CaptureModels in that directory, so I've left GeneratedVsManualCoverageQuery.qll in the ../src/Metrics/Summaries directory for now.

@github-actions github-actions bot added the Java label Mar 27, 2023
@jcogs33 jcogs33 added the no-change-note-required This PR does not need a change note label Mar 31, 2023
@jcogs33 jcogs33 force-pushed the jcogs33/metrics-query-refactor-top500 branch from 7a62b62 to 0688fa6 Compare March 31, 2023 22:02
@jcogs33 jcogs33 marked this pull request as ready for review March 31, 2023 22:06
@jcogs33 jcogs33 requested a review from a team as a code owner March 31, 2023 22:06
Copy link
Copy Markdown
Contributor

@atorralba atorralba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks plausible to me.

OOI, why are the stubs in query-tests instead of stubs? And why do we need JDK stubs in the first place?

but it seems that I can't import utils.modelgenerator.internal.CaptureModels in that directory

Yes, that's because utils is under src, and you can import lib modules in src, but not the other way around.

@jcogs33
Copy link
Copy Markdown
Contributor Author

jcogs33 commented Apr 4, 2023

OOI, why are the stubs in query-tests instead of stubs? And why do we need JDK stubs in the first place?

In order to test the query on the top-500 JDK APIs, I needed a database containing those top-500 APIs as callables (since the query is database-dependent and counts APIs as callables). So the stubs are not really being used as stubs here, but rather as the code that the query is being tested against. Let me know if you have a better idea regarding how to set this test case up. 😄
(Note that this is the same structure as the original test case for this query, but at a larger scale since the top-500 JDK API test case is testing against many more models).

Yes, that's because utils is under src, and you can import lib modules in src, but not the other way around.

Ah, sorry, my question wasn't clear; I knew the issue was because utils.modelgenerator.internal.CaptureModels is under src, but I wanted to make sure that where I ended up placing the .qll files due to that issue was okay.
Although, I think this issue will actually be resolved once I change this query to not use DataFlowTargetApi anymore.

@atorralba
Copy link
Copy Markdown
Contributor

So the stubs are not really being used as stubs here, but rather as the code that the query is being tested against.

Ah, I see! Thanks for the clarification. I was a bit misled by the name, but they are stubs, so it makes sense. The alternative would be adding code that makes all those calls so that they are extracted, which seems equally bothersome, so I think it's good as it is 😄.

I wanted to make sure that where I ended up placing the .qll files due to that issue was okay.

It looks okay to me. The only thing we should keep in mind is that TopJdkApis.qll will now be part of our public API, which means that users will be able to use it, any breaking change will need to go through the deprecation process, and so on. But since you're using it both in src and test, I don't think we have many alternatives.

atorralba
atorralba previously approved these changes Apr 5, 2023
@jcogs33
Copy link
Copy Markdown
Contributor Author

jcogs33 commented Apr 6, 2023

The only thing we should keep in mind is that TopJdkApis.qll will now be part of our public API, which means that users will be able to use it, any breaking change will need to go through the deprecation process, and so on.

Would storing TopJdkApis.qll in the ../src/Metrics/Summaries directory instead avoid this situation? It's importable in the test directory when stored there.

Or would it make sense to at least put an INTERNAL use only. comment in the file similar to the comment in ExternalFlow.qll?

@atorralba
Copy link
Copy Markdown
Contributor

It's importable in the test directory when stored there.

Ah, then I think I slightly favor having it in src.

@jcogs33
Copy link
Copy Markdown
Contributor Author

jcogs33 commented Apr 6, 2023

Ah, then I think I slightly favor having it in src.

Moved in b534f40.

Also, I renamed the stubs directory to just TopJdkApis to avoid confusion with the name.

@jcogs33 jcogs33 merged commit c55c9f5 into github:main Apr 6, 2023
@jcogs33 jcogs33 deleted the jcogs33/metrics-query-refactor-top500 branch April 6, 2023 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Java no-change-note-required This PR does not need a change note

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants