• Ensure the logger’s worker guard remains alive for the duration of the server.
• Add request-handling trace logs to improve debugging and monitoring.
• Set the default log level to INFO.
• Format log events in JSON and with the source information included.

Description

The existing API server logging setup is broken and this PR is to fix the following problems:

• The logger's worker guard is discarded inside the logger setup function, causing no log will be actually written to the log stream when the server is running (bug in this line). This PR fixes this problem by returning the guard to the server's main and hold it for the entire server execution.
• Even with the worker guard fix, the log will not contain any request handling log. This PR adds logs to track request handling status, making it easier to debug issues (I was debugging whether fetching query results would reject non-existing queries and found nothing was showing up in the log).
• The DEBUG log would only contain sqlx's log, which is very noisy as we probably don't need the actual SQL executed for debugging. This PR sets the log level to INFO to make the log concise.
• Format log events in JSON with the source information included (file name and line number).

Checklist

• The PR satisfies the contribution guidelines.
• This is a breaking change and that has been indicated in the PR title, OR this isn't a
  breaking change.
• Necessary docs have been updated, OR no docs need to be updated.

Validation performed

• Ensure all workflows pass.
• Build a package locally and test the following step by step:
  • Compress some JSON logs.
  • Submit a query.
  • Fetch results and ensure the results are successfully received.
  • Fetch results from a non-existing query and ensure it's rejected.
    Logs:

{"timestamp":"2025-11-29T04:46:27.504843Z","level":"INFO","fields":{"message":"Server started at 0.0.0.0:3001"},"filename":"components/api-server/src/bin/api_server.rs","line_number":101}
{"timestamp":"2025-11-29T04:46:40.423889Z","level":"INFO","fields":{"message":"Submitting query: QueryConfig { query_string: \"*\", dataset: None, max_num_results: 0, begin_timestamp: None, end_timestamp: None, ignore_case: false, write_to_file: false }"},"filename":"components/api-server/src/routes.rs","line_number":101}
{"timestamp":"2025-11-29T04:46:40.427186Z","level":"INFO","fields":{"message":"Submitted query with search job ID: 1"},"filename":"components/api-server/src/routes.rs","line_number":104}
{"timestamp":"2025-11-29T04:46:47.498577Z","level":"INFO","fields":{"message":"Fetching results for search job ID: 1"},"filename":"components/api-server/src/routes.rs","line_number":146}
{"timestamp":"2025-11-29T04:46:47.502079Z","level":"INFO","fields":{"message":"Successfully initiated result stream for search job ID 1"},"filename":"components/api-server/src/routes.rs","line_number":149}
{"timestamp":"2025-11-29T04:46:49.770245Z","level":"INFO","fields":{"message":"Fetching results for search job ID: 100"},"filename":"components/api-server/src/routes.rs","line_number":146}
{"timestamp":"2025-11-29T04:46:49.770603Z","level":"ERROR","fields":{"message":"Failed to fetch results for search job ID 100: Sql(RowNotFound)"},"filename":"components/api-server/src/routes.rs","line_number":156}

Summary by CodeRabbit

• Chores

  • Improved logging setup to use a non-blocking writer guard so logging stays active for the program duration.
  • Switched default runtime log level from DEBUG to INFO.
  • Enabled Debug formatting for query configuration objects.

• Bug Fixes

  • Added diagnostic logs around query submission and result streaming and explicit handling/logging for malformed SSE lines.

_{✏️ Tip: You can customize this high-level summary in your review settings.}