Skip to content

clo: Add orig_file_id and log_event_ix to search results.#424

Merged
haiqi96 merged 9 commits into
y-scope:mainfrom
haiqi96:ExtraFields
Jun 12, 2024
Merged

clo: Add orig_file_id and log_event_ix to search results.#424
haiqi96 merged 9 commits into
y-scope:mainfrom
haiqi96:ExtraFields

Conversation

@haiqi96

@haiqi96 haiqi96 commented Jun 4, 2024

Copy link
Copy Markdown
Contributor

References

Description

Current CLP's search results contain the original message, timestamp and original file path. However, the original file path is not sufficient to identify the file if multiple file shares the same path (if they are from different file system, or they are both rotated log with the same file name).

In this PR, we add file's original id and the log event index to every search result, which allows us to locate the specific file split that contains the matching result.

The change can be viewed as two smaller change:

  1. We add a new member variable "log_event_ix" in the Message.hpp that tracks the index in the original file. The original "message_number" is renamed to "ix_in_split" to improve clarity.
  2. We updated the add_results methods in the output handler to add log_event_ix and original_file_id to the search results output by clo executable.

Note that the new fields are not used in the code yet.

Validation performed

Built the package and compress a log file locally, Verified that

  1. The results can be returned as normal by running search.sh
  2. There's no issue when performing a search from webui.
  3. Verifiy that extra fields are properly inserted into the results cache

@haiqi96 haiqi96 changed the title Draft: Add log_event core: Add original_file_id and log_event_ix to clp search results Jun 4, 2024
@haiqi96 haiqi96 changed the title core: Add original_file_id and log_event_ix to clp search results clp: Add original_file_id and log_event_ix to clp search results Jun 10, 2024
Comment thread components/core/src/clp/streaming_archive/reader/Message.hpp Outdated
Comment thread components/core/src/clp/streaming_archive/reader/Message.hpp Outdated
Comment thread components/core/src/clp/clo/OutputHandler.hpp Outdated
Comment thread components/core/src/clp/clo/OutputHandler.hpp Outdated
Comment thread components/core/src/clp/clo/OutputHandler.hpp Outdated
Comment thread components/core/src/clp/clo/OutputHandler.cpp
Comment thread components/core/src/clp/clo/OutputHandler.cpp Outdated
@haiqi96 haiqi96 requested a review from kirkrodrigues June 12, 2024 18:38

@kirkrodrigues kirkrodrigues left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the PR title, how about:

clo: Add orig_file_id and log_event_ix to search results.

@haiqi96 haiqi96 changed the title clp: Add original_file_id and log_event_ix to clp search results clo: Add orig_file_id and log_event_ix to search results. Jun 12, 2024
@haiqi96 haiqi96 merged commit 7116bdf into y-scope:main Jun 12, 2024
@haiqi96 haiqi96 deleted the ExtraFields branch June 28, 2024 14:43
jackluo923 pushed a commit to jackluo923/clp that referenced this pull request Dec 4, 2024
junhaoliao pushed a commit to junhaoliao/clp that referenced this pull request May 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants