Skip to content

Conversation

@skurpad7
Copy link
Contributor

Fixes #8611

As described in the above issue, Osquery stores the state of the scheduled queries in the kQueries domain in RocksDB. The Config::purge() function is invoked on every config refresh. It is responsible for purging the stored state of queries that are no longer in the schedule and haven't been executed in the last 7 days. Whenever a query is scheduled, the timestamp for it is updated as timestamp.<name> in the kPersistentSettings domain. The Config::purge() function however, sets the timestamps for all metadata keys in the kQueries domain such as <name>counter, <name>epoch, query.<name> as well if not already present. The Osquery scheduler never updates the timestamp for these metadata keys. Every 7 days, Config::purge() checks the timestamps of these keys, and because they always have week old timestamps, they are purged from the store. This causes the counter metadata key of queries to reset but only differential results are reported leading to the issue above.
With this change,
1. Metadata keys are ignored from the purging logic.
2. Users will not be allowed to add queries with query.,cache. and config_views. as a prefix or epoch and counter as a suffix.

Config module changes

  • The queryExists function returns false for queries even if they are present in the schedule. Consider a query query_name in a pack pack_name with / as the pack delimiter. When queryExists("pack/pack_name/query_name") is called, it iterates over the schedule and compares if "query_name" == "pack/pack_name/query_name". This always returns false even if the query is present in the schedule. This function is updated to use fully qualified query names to fix this behavior.

  • The timestamp has the final say in determining if a key is purged from the store. The Config::purge() function ages and purges reserved keys such as <name>counter, <name>epoch and query.<name> as described above. This leads to the counter (and other query metadata) being purged inconsistently. With this change, such keys are not considered for purge. The unit test test_config_results_purge is updated to validate the same.

  • Currently, there are no restrictions on the names that can be set for a query. Query names can contain metadata prefixes such as query. and cache. or suffixes such as epoch and counter. This leads to the following issues:

    • The updated purge function skips over keys which have metadata prefixes and suffixes. Queries having such names are not purged even if out of schedule.
    • When a query name conflicts with the metadata of another query, osquery crashes with an error. For example, in a pack pack1, if there are two queries with names query1 and query1counter, osquery crashes with the following error: E0624 15:50:45.187048 119689 shutdown.cpp:80] Error adding new results to database for query pack/pack1/query1counter: JSON object was not an array.
  • This change prevents queries from having such reserved prefixes and suffixes in their names to address the above issues and adds a new unit test test_queryname_validation for the same.

Database module changes

  • Vectors kReservedPrefixes and kReservedSuffixes are defined to store the reserved prefixes and suffixes that have a special meaning in the RocksDB store.

Util Module Changes

  • Helper functions bool hasAnyPrefix(const std::string& s, const std::vector<std::string>& prefixes), bool hasAnySuffix(const std::string& s, const std::vector<std::string>& suffixes) and their corresponding unit test test_affixes are added. These functions are used in the config module to check if a string has any of the reserved database prefixes or suffixes.

@skurpad7 skurpad7 requested review from a team as code owners June 25, 2025 20:38
@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Jun 25, 2025

CLA Signed

The committers listed above are authorized under a signed CLA.

@skurpad7
Copy link
Contributor Author

The new files created in this change contain Akamai Technologies Inc. in the header. Please let me know if something else would be preferred there.

@skurpad7 skurpad7 force-pushed the fix-counter-resets branch 2 times, most recently from 8fa93c7 to 4d50d1a Compare June 30, 2025 13:49
@skurpad7 skurpad7 force-pushed the fix-counter-resets branch from 4d50d1a to ad79c33 Compare June 30, 2025 15:26
@@ -0,0 +1,31 @@
/**
* Copyright (c) 2025-present, Akamai Technologies Inc.
Copy link
Member

@zwass zwass Jul 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@skurpad7 thank you for asking about this! We will need the standard osquery copyright header which says

/**
 * Copyright (c) 2014-present, The osquery authors
 *
 * This source code is licensed as defined by the LICENSE file found in the
 * root directory of this source tree.
 *
 * SPDX-License-Identifier: (Apache-2.0 OR GPL-2.0-only)
 */

You will also need to sign the CLA.

Note that the "check code style" action (https://github.com/osquery/osquery/actions/runs/15977093557/job/45232527575?pr=8635) is failing because of this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, thank you! I have updated the header accordingly.

@zwass
Copy link
Member

zwass commented Aug 6, 2025

@skurpad7 we need you to sign the CLA in order to merge any code to the project. Thank you!

@skurpad7
Copy link
Contributor Author

skurpad7 commented Aug 8, 2025

Hi @zwass! the CLA has been signed.

@zwass zwass closed this Aug 13, 2025
@zwass zwass reopened this Aug 13, 2025
@zwass
Copy link
Member

zwass commented Aug 13, 2025

Reopened PR to kick off CI.

Copy link
Member

@zwass zwass left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@skurpad7 thank you so much for the thoroughly documented issue, PR, and code. I'm very pleased to review something with such attention to detail.

Thank you for your patience awaiting this review.

I intend to merge this once CI passes.

@zwass zwass merged commit 95b48d6 into osquery:master Aug 13, 2025
22 checks passed
@skurpad7
Copy link
Contributor Author

Awesome! thank you so much for the review and the guidance through the PR, @zwass!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Inconsistent counter resets due to Config Purge

2 participants