Skip to content

Reduce log noise for hash table #8626

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
zwass merged 1 commit into from
Jun 11, 2025
Merged

Reduce log noise for hash table #8626

zwass merged 1 commit into from
Jun 11, 2025

Conversation

@lucasmrod
Copy link
Contributor

@lucasmrod lucasmrod commented Jun 11, 2025 •
edited
Loading

Fixes fleetdm/fleet#28577.

TL;DR: When hash table is used as scheduled query, the generated WARNING logs fill up the disk space.

Sample:
Without --verbose:

sudo ./osquery/osqueryi

osquery> SELECT * FROM hash WHERE path = '/Users/lucas/Downloads/Win11_24H2_English_Arm64.iso';
+-----------------------------------------------------+------------------------+-----+------+--------+
| path                                                | directory              | md5 | sha1 | sha256 |
+-----------------------------------------------------+------------------------+-----+------+--------+
| /Users/lucas/Downloads/Win11_24H2_English_Arm64.iso | /Users/lucas/Downloads |     |      |        |
+-----------------------------------------------------+------------------------+-----+------+--------+

With --verbose:

sudo ./osquery/osqueryi --verbose

osquery> SELECT * FROM hash WHERE path = '/Users/lucas/Downloads/Win11_24H2_English_Arm64.iso';
I0611 10:23:41.937412 -289448192 hashing.cpp:133] Failed to hash /Users/lucas/Downloads/Win11_24H2_English_Arm64.iso: Cannot read /Users/lucas/Downloads/Win11_24H2_English_Arm64.iso size exceeds limit: 5460387840 > 52428800
+-----------------------------------------------------+------------------------+-----+------+--------+
| path                                                | directory              | md5 | sha1 | sha256 |
+-----------------------------------------------------+------------------------+-----+------+--------+
| /Users/lucas/Downloads/Win11_24H2_English_Arm64.iso | /Users/lucas/Downloads |     |      |        |
+-----------------------------------------------------+------------------------+-----+------+--------+

@lucasmrod lucasmrod requested review from a team as code owners June 11, 2025 13:29
@lucasmrod lucasmrod mentioned this pull request Jun 11, 2025
Closed
@zwass zwass merged commit 8458718 into master Jun 11, 2025
27 checks passed
@zwass zwass deleted the 28577-reduce-hash-table-log-noise branch June 11, 2025 22:26
directionless
directionless reviewed Jun 13, 2025
View reviewed changes
osquery/hashing/hashing.cpp

if (!status.ok()) {
LOG(WARNING) << "Failed to hash " << path;
VLOG(1) << "Failed to hash " << path << ": " << status.getMessage();
Copy link
Member

@directionless directionless Jun 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this merged, but I think this might not be the right thing here.

You're example, size exceeds limit feels routine, and thus maybe does belong down on VLOG, but what if there's a different error? Should that be on WARNING?

@lucasmrod lucasmrod mentioned this pull request Jul 16, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@directionless directionless directionless left review comments

@zwass zwass zwass approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

osquery: Errors from hash table should only be logged when DEBUGing

4 participants

@lucasmrod @directionless @zwass