Skip to content

Optimize rpm_packages and rpm_package_files use of query context #8537

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
zwass merged 2 commits into from
Jan 28, 2025
Merged

Optimize rpm_packages and rpm_package_files use of query context #8537

zwass merged 2 commits into from
Jan 28, 2025

Conversation

@Micah-Kolide
Copy link
Contributor

@Micah-Kolide Micah-Kolide commented Jan 24, 2025 •
edited
Loading

Currently the RPM table is being generated by initializing the RPM db with all packages, or a single package by name with query context. I've changed this table to start using rpmdbSetIteratorRE with string compare, by iterating through all values given in query context, and adding those regex patterns to the RPM iterator selector.

RPM API rpmdbSetIteratorRE: https://ftp.osuosl.org/pub/rpm/api/4.4.2.2/group__rpmdb.html#g9f84dbc47d1d432f4ffe83b713fb47c9

This allows the table to generate results with all values from query context at once.

I've confirmed that the columns can support these changes by querying the tables with an IN constraint on the optimized columns. I validated the expected results by comparing returned values from osquery 5.13.1 (before IN optimization existed), 5.14.1, and 5.14.1 containing these spec file changes.

With each query I included a NULL, '' (empty string), and some non-existent values in my IN constraint.

Tests were ran on CentOS Linux release 7.9.2009 (Core)

@Micah-Kolide Micah-Kolide requested review from a team as code owners January 24, 2025 21:50
zwass
zwass approved these changes Jan 28, 2025
View reviewed changes
Copy link
Member

@zwass zwass left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. @Micah-Kolide can you please confirm that you tested multiple valid package names with IN and saw that rows were returned for each?

@Micah-Kolide
Copy link
Contributor Author

Micah-Kolide commented Jan 28, 2025

LGTM. @Micah-Kolide can you please confirm that you tested multiple valid package names with IN and saw that rows were returned for each?

@zwass sure thing!

Here's some of the test output I was working with:

osquery> SELECT * FROM rpm_packages WHERE name IN ('zlib', 'yum', 'which', '', NULL, 'gwnviwjiworjnv');
...
osquery planner: xFilter Adding constraint to cursor (1): name =
osquery planner: xFilter Adding constraint to cursor (1): name = gwnviwjiworjnv
osquery planner: xFilter Adding constraint to cursor (1): name = which
osquery planner: xFilter Adding constraint to cursor (1): name = yum
osquery planner: xFilter Adding constraint to cursor (1): name = zlib
...
+-------+---------+----------------+----------------------------------+---------+------------------------------------------+--------+-------+--------------+--------+------------------------------+
| name  | version | release        | source                           | size    | sha1                                     | arch   | epoch | install_time | vendor | package_group                |
+-------+---------+----------------+----------------------------------+---------+------------------------------------------+--------+-------+--------------+--------+------------------------------+
| which | 2.20    | 7.el7          | which-2.20-7.el7.src.rpm         | 76962   | 01530fd2bd615864e31bb31c1fbd0fd590f3c702 | x86_64 |       | 1707852703   | CentOS | Applications/System          |
| yum   | 3.4.3   | 168.el7.centos | yum-3.4.3-168.el7.centos.src.rpm | 5829237 | 946a3d8da1cd710a343f0cdc8bf7b50efca93822 | noarch |       | 1707852753   | CentOS | System Environment/Base      |
| zlib  | 1.2.7   | 21.el7_9       | zlib-1.2.7-21.el7_9.src.rpm      | 185206  | 90bcb1700b0b5e68692f0075dbf8c8bfa2f32d21 | x86_64 |       | 1707852700   | CentOS | System Environment/Libraries |
+-------+---------+----------------+----------------------------------+---------+------------------------------------------+--------+-------+--------------+--------+------------------------------+

osquery> SELECT * FROM rpm_package_files WHERE package IN ('zlib', 'yum', 'which', '', NULL, 'gwnviwjiworjnv');
...
osquery planner: xFilter Adding constraint to cursor (1): package =
osquery planner: xFilter Adding constraint to cursor (1): package = gwnviwjiworjnv
osquery planner: xFilter Adding constraint to cursor (1): package = which
osquery planner: xFilter Adding constraint to cursor (1): package = yum
osquery planner: xFilter Adding constraint to cursor (1): package = zlib
...
+---------+-----------------------------------------------------------+----------+-----------+------+--------+------------------------------------------------------------------+
| package | path                                                      | username | groupname | mode | size   | sha256                                                           |
+---------+-----------------------------------------------------------+----------+-----------+------+--------+------------------------------------------------------------------+
| which   | /usr/share/doc/which-2.20/README                          | root     | root      | 0644 | 4433   | 680debfe517546b48b9d76943eecb47e3ce1360f516496741e1a6b1852e4c53b |
...
| yum     | /usr/share/doc/yum-3.4.3/README                           | root     | root      | 0644 | 1078   | 3d52ed91258700fa5cb7396fe6f60086a94e3daa80afc5a30ebab2508a5f88b7 |
...
| zlib    | /usr/share/doc/zlib-1.2.7/README                          | root     | root      | 0644 | 5185   | a4dfc7f4225a68315763aafb536428852aff6dd3135ab4e0cdc9e3e6a65fd796 |
+---------+-----------------------------------------------------------+----------+-----------+------+--------+------------------------------------------------------------------+

@zwass
Copy link
Member

zwass commented Jan 28, 2025

Love it, thanks!!

@zwass zwass merged commit 5fe4bd4 into osquery:master Jan 28, 2025
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zwass zwass zwass approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Micah-Kolide @zwass