Description

This issue tracks the initial implementation of the Search Analytics logging infrastructure. This feature allows administrators to view historical query performance and latency for executed queries to validate user experience and monitor usage trends. This implementation focuses on the backend capture logic within Elasticsearch.

Functional Requirements

• US1A: The log must capture:
• Execution time (took).
• Status (success vs failure) and error messages if applicable.
• The exact query (request parameters and body).
• Support for DSL (_search), ES|QL (_query), EQL, SQL..
• US1B: Users must be able to enable logging based on a query execution time threshold (e.g., log all queries > 0ms, or only those taking > 5 min).
• US1C: The output must allow ECH and self-managed users to ship logs from the generating deployment to a destination of their choice (via standard file-based log collectors initially).
• US1F: Configuration of the log must be restricted to administrators or users with the manage Elasticsearch privilege.

Non-Functional Requirements

• US1D: The logging process must be asynchronous to the search execution to prevent impact on query latency.
• US1E: If the system is overloaded, the implementation must drop log entries rather than slowing down the query response or causing node instability.
• US1M: The log format must adhere to Elastic Common Schema (ECS) conventions where possible.