Skip to content

Fix the python_paths table to skip unnecessary code paths when filtering by directory #8544

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
directionless merged 1 commit into from
Feb 8, 2025
Merged

Fix the python_paths table to skip unnecessary code paths when filtering by directory #8544

directionless merged 1 commit into from
Feb 8, 2025

Conversation

@ksykulev
Copy link
Contributor

@ksykulev ksykulev commented Feb 5, 2025

where directory = '....' was added in #4407
When specifying a directory to filter by, the query still executes all the code paths unnecessarily.

osquery> select name, version, path from users cross join python_packages using(uid) where python_packages.directory = "/opt/homebrew/lib";
+-----------+---------+----------------------------------------------------------------------+
| name      | version | path                                                                 |
+-----------+---------+----------------------------------------------------------------------+
| packaging | 24.2    | /opt/homebrew/lib/python3.12/site-packages/packaging-24.2.dist-info  |
| numpy     | 2.2.2   | /opt/homebrew/lib/python3.12/site-packages/numpy-2.2.2.dist-info     |
| pip       | 24.3.1  | /opt/homebrew/lib/python3.13/site-packages/pip-24.3.1.dist-info      |
| packaging | 24.2    | /opt/homebrew/lib/python3.13/site-packages/packaging-24.2.dist-info  |
| wheel     | 0.45.1  | /opt/homebrew/lib/python3.13/site-packages/wheel-0.45.1.dist-info    |
| numpy     | 2.2.2   | /opt/homebrew/lib/python3.13/site-packages/numpy-2.2.2.dist-info     |
| mercurial | 6.8.2   | /opt/homebrew/lib/python3.13/site-packages/mercurial-6.8.2.dist-info |
+-----------+---------+----------------------------------------------------------------------+
Run Time: real 18.694 user 4.362583 sys 14.125906
osquery> select name, version, path from python_packages where python_packages.directory = "/opt/homebrew/lib";
W0121 18:26:17.506508 -245185984 virtual_table.cpp:1005] The python_packages table returns data based on the current user by default, consider JOINing against the users table
W0121 18:26:17.506549 -245185984 virtual_table.cpp:1022] Please see the table documentation: https://osquery.io/schema/#python_packages
+-----------+---------+----------------------------------------------------------------------+
| name      | version | path                                                                 |
+-----------+---------+----------------------------------------------------------------------+
| packaging | 24.2    | /opt/homebrew/lib/python3.12/site-packages/packaging-24.2.dist-info  |
| numpy     | 2.2.2   | /opt/homebrew/lib/python3.12/site-packages/numpy-2.2.2.dist-info     |
| pip       | 24.3.1  | /opt/homebrew/lib/python3.13/site-packages/pip-24.3.1.dist-info      |
| packaging | 24.2    | /opt/homebrew/lib/python3.13/site-packages/packaging-24.2.dist-info  |
| wheel     | 0.45.1  | /opt/homebrew/lib/python3.13/site-packages/wheel-0.45.1.dist-info    |
| numpy     | 2.2.2   | /opt/homebrew/lib/python3.13/site-packages/numpy-2.2.2.dist-info     |
| mercurial | 6.8.2   | /opt/homebrew/lib/python3.13/site-packages/mercurial-6.8.2.dist-info |
+-----------+---------+----------------------------------------------------------------------+
Run Time: real 0.175 user 0.037518 sys 0.134921

Exiting early yields a slightly more performant result.

osquery> select name, version, path from users cross join python_packages using(uid) where python_packages.directory = "/opt/homebrew/lib";
+-----------+---------+----------------------------------------------------------------------+
| name      | version | path                                                                 |
+-----------+---------+----------------------------------------------------------------------+
| packaging | 24.2    | /opt/homebrew/lib/python3.12/site-packages/packaging-24.2.dist-info  |
| numpy     | 2.2.2   | /opt/homebrew/lib/python3.12/site-packages/numpy-2.2.2.dist-info     |
| pip       | 24.3.1  | /opt/homebrew/lib/python3.13/site-packages/pip-24.3.1.dist-info      |
| packaging | 24.2    | /opt/homebrew/lib/python3.13/site-packages/packaging-24.2.dist-info  |
| wheel     | 0.45.1  | /opt/homebrew/lib/python3.13/site-packages/wheel-0.45.1.dist-info    |
| numpy     | 2.2.2   | /opt/homebrew/lib/python3.13/site-packages/numpy-2.2.2.dist-info     |
| mercurial | 6.8.2   | /opt/homebrew/lib/python3.13/site-packages/mercurial-6.8.2.dist-info |
+-----------+---------+----------------------------------------------------------------------+
Run Time: real 16.561 user 4.024763 sys 12.490508
osquery> select name, version, path from python_packages where python_packages.directory = "/opt/homebrew/lib";
W0121 18:47:53.716681 -245185984 virtual_table.cpp:1005] The python_packages table returns data based on the current user by default, consider JOINing against the users table
W0121 18:47:53.716734 -245185984 virtual_table.cpp:1022] Please see the table documentation: https://osquery.io/schema/#python_packages
+-----------+---------+----------------------------------------------------------------------+
| name      | version | path                                                                 |
+-----------+---------+----------------------------------------------------------------------+
| packaging | 24.2    | /opt/homebrew/lib/python3.12/site-packages/packaging-24.2.dist-info  |
| numpy     | 2.2.2   | /opt/homebrew/lib/python3.12/site-packages/numpy-2.2.2.dist-info     |
| pip       | 24.3.1  | /opt/homebrew/lib/python3.13/site-packages/pip-24.3.1.dist-info      |
| packaging | 24.2    | /opt/homebrew/lib/python3.13/site-packages/packaging-24.2.dist-info  |
| wheel     | 0.45.1  | /opt/homebrew/lib/python3.13/site-packages/wheel-0.45.1.dist-info    |
| numpy     | 2.2.2   | /opt/homebrew/lib/python3.13/site-packages/numpy-2.2.2.dist-info     |
| mercurial | 6.8.2   | /opt/homebrew/lib/python3.13/site-packages/mercurial-6.8.2.dist-info |
+-----------+---------+----------------------------------------------------------------------+
Run Time: real 0.161 user 0.037412 sys 0.123799

@ksykulev ksykulev requested review from a team as code owners February 5, 2025 21:53
directionless
directionless approved these changes Feb 8, 2025
View reviewed changes
Copy link
Member

@directionless directionless left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense!

@directionless directionless changed the title When filtering by directory, avoid running unnecessary code paths Fix the python_paths table to skip unnecessary code paths when filtering by directory Feb 8, 2025
@directionless directionless merged commit 16bb015 into osquery:master Feb 8, 2025
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@directionless directionless directionless approved these changes

Assignees

No one assigned

Labels

virtual tables

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ksykulev @directionless