Skip to content

clo: Add CLI command to extract a compressed file as IR.#437

Merged
haiqi96 merged 106 commits into
y-scope:mainfrom
haiqi96:clo-change
Jun 22, 2024
Merged

clo: Add CLI command to extract a compressed file as IR.#437
haiqi96 merged 106 commits into
y-scope:mainfrom
haiqi96:clo-change

Conversation

@haiqi96

@haiqi96 haiqi96 commented Jun 12, 2024

Copy link
Copy Markdown
Contributor

Description

The change refactored clo so it takes in a command and can extract a file split into one or more IR chunks.
clo extracts the IRs to a local file system path, and sends metadata to a mongodb specified by the user input.

Validation performed

  1. Verified that CLP package still properly return search results.
    The following validation are all performed directly via commandline (as opposed to in a clp package)
  2. Compressed a hadoop64MB log into an archive, resulting in a single file split.
  3. Extract the file split as IR with target-size = 16MB (resulting in 5 IR chunks), with results sent to a mongodb running in container
  • Verified that metadata are properly written to the mongodb collection
  • Verified that is_last_ir_chunk is set properly in the metadata.
  • Verified that IR chunks are successfully written to the disk

@LinZhihao-723 LinZhihao-723 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ask Haiqi asked, this is a review on CommandLinArguments.hpp/cpp and clo.cpp.
Note that the review is brief, mostly catching micro-level issues. Didn't review it from an architecture level, so the variable naming + API design review might be required (please let me know if I should do a further as well).

Comment thread components/core/src/clp/clo/CommandLineArguments.hpp
Comment thread components/core/src/clp/clo/CommandLineArguments.hpp Outdated
Comment thread components/core/src/clp/clo/CommandLineArguments.hpp Outdated
Comment thread components/core/src/clp/clo/CommandLineArguments.hpp Outdated
Comment thread components/core/src/clp/clo/CommandLineArguments.cpp Outdated
Comment thread components/core/src/clp/clo/clo.cpp Outdated
Comment thread components/core/src/clp/clo/clo.cpp Outdated
Comment thread components/core/src/clp/clo/clo.cpp Outdated
Comment thread components/core/src/clp/clo/clo.cpp Outdated
Comment thread components/core/src/clp/clo/clo.cpp

@kirkrodrigues kirkrodrigues left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the PR title, how about:

clo: Add CLI command to extract a compressed file as IR.

@haiqi96 haiqi96 changed the title core-clp: Add CLI command in clo to extract a compressed file as IR. clo: Add CLI command to extract a compressed file as IR. Jun 22, 2024
@haiqi96 haiqi96 merged commit ea99f50 into y-scope:main Jun 22, 2024
jackluo923 pushed a commit to jackluo923/clp that referenced this pull request Dec 4, 2024
junhaoliao pushed a commit to junhaoliao/clp that referenced this pull request May 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants