Skip to content

Python: add TypeTrackingNode#6218

Merged
codeql-ci merged 6 commits intogithub:mainfrom
tausbn:python-add-typetrackingnode
Jul 15, 2021
Merged

Python: add TypeTrackingNode#6218
codeql-ci merged 6 commits intogithub:mainfrom
tausbn:python-add-typetrackingnode

Conversation

@tausbn
Copy link
Copy Markdown
Contributor

@tausbn tausbn commented Jul 2, 2021

Splits ModuleVariableNode away from LocalSourceNode, instead
creating a class TypeTrackingNode that encapsulates both of these.

This means we no longer have module variable nodes as part of
LocalSourceNode (which is good, since they have no "local" aspect to
them), and hence we can have LocalSourceNode inherit directly from
ExprNode (which makes the API a bit nicer).

Unfortunately these are breaking changes, so we can't actually fulfil
the above two desiderata until the track and backtrack methods on
LocalSourceNode have been fully deprecated. For this reason, we
preserve the present implementation of LocalSourceNode, and instead
lay the foundation for switching over in the future, by deprecating
track and backtrack on LocalSourceNode.


Deprecates two public methods, so needs a change note.

Splits `ModuleVariableNode` away from `LocalSourceNode`, instead
creating a class `TypeTrackingNode` that encapsulates both of these.

This means we no longer have module variable nodes as part of
`LocalSourceNode` (which is good, since they have no "local" aspect to
them), and hence we can have `LocalSourceNode` inherit directly from
`ExprNode` (which makes the API a bit nicer).

Unfortunately these are breaking changes, so we can't actually fulfil
the above two desiderata until the `track` and `backtrack` methods on
`LocalSourceNode` have been fully deprecated. For this reason, we
preserve the present implementation of `LocalSourceNode`, and instead
lay the foundation for switching over in the future, by deprecating
`track` and `backtrack` on `LocalSourceNode`.
@github-actions github-actions bot added the Python label Jul 2, 2021
@tausbn tausbn marked this pull request as ready for review July 12, 2021 10:00
@tausbn tausbn requested a review from a team as a code owner July 12, 2021 10:00
@tausbn
Copy link
Copy Markdown
Contributor Author

tausbn commented Jul 12, 2021

I should probably run a performance test on this just in case. I don't expect it to change anything, but who knows how it may influence the compiler.

Copy link
Copy Markdown
Member

@RasmusWL RasmusWL left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good to me 👍 (as long as performance is good)

I think the major risk from this PR is the extra complexity of having another class of data-flow nodes that you need to know about, and a potential problem in that it's still possible to write type-tracking code using LocalSourceNode and not TypeTrackingNode (that just won't work properly).

Since we probably have some open PRs that use type-tracking right now, fixing this up is something we need to keep in mind for the near future (and possibly do a grep through our codebase on a regular cadence through the next few months)

For the grep part, I developed grep --files-with-match -Pazo 'LocalSourceNode[^({}]+\([^)]*(TypeTracker|TypeBackTracker)' **.{ql,qll} which handles predicate definition spanning multiple-lines... but obviously using CodeQL would be better 😉

@tausbn
Copy link
Copy Markdown
Contributor Author

tausbn commented Jul 13, 2021

Performance looks to be mostly unchanged: https://github.com/dsp-testing/tausbn-dca/issues/46

I'm not sure I understand your point about type-tracking using LocalSourceNode not working properly. With this PR, LocalSourceNode and TypeTrackingNode contain the exact same nodes. The inclusion of the former into the latter is immediate, and the other way round follows from the fact that TypeTrackingNode only adds ModuleVariableNode, but these are already (still) contained in LocalSourceNode. Therefore, I don't think anything is broken as such. You'll just get a big "deprecation warning" if you write a type tracker based on LocalSourceNode (and presumably use track or backtrack).

Just to verify my intuition above, I tried changing the pathlibPath type tracker to return a LocalSourceNode instead, and all the relevant tests still passed. (I also tried breaking it by adding and none() to the base case, and this did break some tests, so it at least does something. 😅)

I agree that this does add a bit more complexity, but my hope was that this would actually make things more clear (at least in the long run). To me, TypeTrackingNode more clearly signals that it's relevant to type tracking than LocalSourceNode does.

@tausbn
Copy link
Copy Markdown
Contributor Author

tausbn commented Jul 13, 2021

Oh, on that note, I must remember to update the documentation correspondingly.

@RasmusWL
Copy link
Copy Markdown
Member

I'm not sure I understand your point about type-tracking using LocalSourceNode not working properly. With this PR, LocalSourceNode and TypeTrackingNode contain the exact same nodes. The inclusion of the former into the latter is immediate, and the other way round follows from the fact that TypeTrackingNode only adds ModuleVariableNode, but these are already (still) contained in LocalSourceNode. Therefore, I don't think anything is broken as such. You'll just get a big "deprecation warning" if you write a type tracker based on LocalSourceNode (and presumably use track or backtrack).

Very good point. The problem I'm talking about would actually never occur. The case I had in mind was when we were able to change the char-pred of LocalSourceNode to not include ModuleVariableNode... but then we would also remove the track backtrack member-predicates, so you wouldn't even be able to compile a type-tracker that used these...

I somehow didn't add 2 and 2 together on the fact that these are now deprecated, so we would start getting warnings if we merged anything that used these, which would in turn make our CI angry... so we probably won't need that grep I created at all 😅

@tausbn
Copy link
Copy Markdown
Contributor Author

tausbn commented Jul 13, 2021

Ah, good. 🙂

(I should also add that actually testing that things behaved the way I expected them to was probably a good idea. It would have been unfortunate if LocalSourceNode-based type trackers started giving different results. 😬)

@tausbn
Copy link
Copy Markdown
Contributor Author

tausbn commented Jul 15, 2021

Oh, on that note, I must remember to update the documentation correspondingly.

It seems I misremembered, and there is no Python-specific documentation of type trackers. While we should probably add this at some point, I don't think this needs to be in the present PR.

@codeql-ci codeql-ci merged commit d282f6a into github:main Jul 15, 2021
@tausbn tausbn deleted the python-add-typetrackingnode branch July 19, 2021 09:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants