Skip to content

Conversation

@Smjert
Copy link
Member

@Smjert Smjert commented Oct 19, 2024

The log entries timestamp is a double, we need to ensure that it's cast to an integer when returning it as a result to sqlite and using a timestamp constraint, otherwise sqlite will filter out all the entries that have the decimal part in them.

The log entries timestamp is a double, we need to ensure
that it's cast to an integer when returning it as a result
to sqlite and using a timestamp constraint,
otherwise sqlite will filter out all the entries
that have the decimal part in them.

r["timestamp"] = BIGINT([[entry date] timeIntervalSince1970]);
r["timestamp"] = BIGINT(
static_cast<std::uint32_t>([[entry date] timeIntervalSince1970]));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the spirit of not losing information, do you think it makes sense to also pass a float through? (timestamp_float or something)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure (I would call it timestamp_double to respect the true type).

@Smjert
Copy link
Member Author

Smjert commented Oct 19, 2024

There's an additional problem I did not consider. When writing a constraint like timestamp > 123, the predicate passed to the unified log is actually timestamp 123.0 which then means that entries with timestamp 123.1, 123.2 and so on are for all intents and purposes higher than that, but then when casted back to an integer they are still 123.
So these will still get filtered by sqlite and at the same time count towards the default 100 rows returned limit.

The solution here is to actually pass the timestamp in the constraint + 1, if the constraint is using >. If it was any other logging system, a log message with time 123.1 would've had 123 as a timestamp.
The table has predicate that can be used to do more precise constraints in theory.

@Smjert
Copy link
Member Author

Smjert commented Oct 19, 2024

This should do, although there are still some interesting questions around the > and >= priorities on the timestamp column due to how they are processed (where >= is "stronger" if the constraint value is the same between > and >=, but whoever has the value that's higher, wins). This can wait though.

@Smjert Smjert closed this Oct 19, 2024
@Smjert Smjert reopened this Oct 19, 2024
@directionless directionless changed the title unified_log: Fix sqlite filtering incorrect timestamp format Fix unified_log handling of timestamp formats Oct 20, 2024
@directionless directionless merged commit b3b3595 into osquery:master Oct 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants