Skip to content
This repository was archived by the owner on May 14, 2026. It is now read-only.

feat: store dir structure compatible with pnpm#166

Merged
zkochan merged 110 commits into
mainfrom
compatible-store-dir-structure
Nov 2, 2023
Merged

feat: store dir structure compatible with pnpm#166
zkochan merged 110 commits into
mainfrom
compatible-store-dir-structure

Conversation

@KSXGitHub

@KSXGitHub KSXGitHub commented Oct 30, 2023

Copy link
Copy Markdown
Contributor

Resolves #165

Other changes:

  • A new crate named pacquet-store-dir has been created.
  • The field store-dir in Npmrc has been changed from PathBuf to StoreDir.
  • The function write_sync in pacquet-cafs has been replaced by 2 functions (8f98e73).
  • The pacquet-cafs crate has been merged into pacquet-store-dir (db673ef).
  • The pacquet-tarball crate now also writes index files (3b6e68a..8f98e73).
  • The command pacquet prune now panics on todo!() for being incomplete (c8526d0).

TODO:

  • Test:
    • cas_file_path.
    • index_file_path.
    • Snapshot of the store.
    • Use pnpm install --offline to test store dir compatibility.
    • Executable bits.
  • Use sha2::Sha512.
  • Executable files (31148d3).
  • Fix executable bits
    • Use 111 as bit mask.
    • All executables must be 755.
  • Index files (3b6e68a..8f98e73).
  • Disable prune (c8526d0).
  • checkedAt.
  • Refactor:
    • Remove unnecessary items from cafs.
    • Create StoreDir::new and switch all .to_path_buf().into() to using it.
    • Move cafs into store-dir (db673ef).
    • Add From to TaskError.
    • Separate write_sync into 2 functions, one for non-index, one for index (8f98e73).
    • Move TarballIndex to store-dir and reuse it in write_tarball_index_file.
    • file_path_by_content_address should take exec: bool instead of suffix: FileSuffix.
    • Remove FileSuffix and strum.
    • pacquet_fs::make_file_executable.

@KSXGitHub KSXGitHub requested review from anonrig and zkochan October 30, 2023 08:02
@codecov

codecov Bot commented Oct 30, 2023

Copy link
Copy Markdown

Codecov Report

Attention: 17 lines in your changes are missing coverage. Please review.

Comparison is base (a97eb4b) 86.08% compared to head (4d72247) 86.58%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #166      +/-   ##
==========================================
+ Coverage   86.08%   86.58%   +0.49%     
==========================================
  Files          53       56       +3     
  Lines        2710     2863     +153     
==========================================
+ Hits         2333     2479     +146     
- Misses        377      384       +7     
Files Coverage Δ
crates/npmrc/src/lib.rs 98.01% <100.00%> (+0.04%) ⬆️
crates/package-manager/src/install.rs 81.81% <100.00%> (ø)
...ckage-manager/src/install_package_from_registry.rs 98.36% <100.00%> (ø)
crates/store-dir/src/store_dir.rs 100.00% <100.00%> (ø)
crates/testing-utils/src/fs.rs 100.00% <100.00%> (ø)
crates/cli/src/cli_args/store.rs 72.72% <0.00%> (ø)
crates/npmrc/src/custom_deserializer.rs 98.13% <95.65%> (+0.17%) ⬆️
crates/store-dir/src/cas_file.rs 97.56% <97.56%> (ø)
crates/testing-utils/src/bin.rs 96.42% <94.11%> (-3.58%) ⬇️
crates/store-dir/src/index_file.rs 93.54% <93.54%> (ø)
... and 3 more

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@github-actions

github-actions Bot commented Oct 30, 2023

Copy link
Copy Markdown

Micro-Benchmark Results

Linux

group                          main                                   pr
-----                          ----                                   --
tarball/download_dependency    1.09      8.3±7.34ms   522.8 KB/sec    1.00      7.6±0.18ms   568.7 KB/sec

@github-actions

github-actions Bot commented Oct 30, 2023

Copy link
Copy Markdown

Integrated-Benchmark Report (Linux)

Scenario: Frozen Lockfile

Command Mean [ms] Min [ms] Max [ms] Relative
pacquet@HEAD 139.8 ± 6.0 128.6 148.5 1.05 ± 0.07
pacquet@main 132.7 ± 6.2 125.6 148.2 1.00

@KSXGitHub KSXGitHub requested a review from zkochan November 1, 2023 08:46
@KSXGitHub KSXGitHub marked this pull request as ready for review November 1, 2023 18:42
@KSXGitHub KSXGitHub mentioned this pull request Nov 1, 2023
Comment thread crates/store-dir/src/index_file.rs Outdated
/// Path to an index file of a tarball.
pub fn tarball_index_file_path(&self, tarball_integrity: &Integrity) -> PathBuf {
let (algorithm, hex) = tarball_integrity.to_hex();
assert_eq!(algorithm, Algorithm::Sha512, "Only Sha512 is supported"); // TODO: propagate this error

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is handled by the ssri library, which can accept any SHA version and convert it to hex. I assume it is just a shorter hex because sha1 is shorter.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need so many files? Isn't it enough to test on a package that has two files? One executable and one non-executable?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you can suggest a package that has only a few files with one executable, I will use it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There relevant code is here:

let package_json_content = serde_json::json!({
"dependencies": {
"pretty-exec": "0.3.10",
},
});

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a registry-mock with testing packages. It has a package with one executable: https://github.com/pnpm/registry-mock/tree/main/packages/hello-world-js-bin%401.0.0

We just need to start using the registry-mock

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We just need to start using the registry-mock

This should be in another PR.

@zkochan

zkochan commented Nov 2, 2023

Copy link
Copy Markdown
Member

There's one important nuance to linking from the store. This shouldn't be done in this PR but we should remember to do it. When a package that has postinstall scripts is "imported" from the store, it is only imported using reflinks or copying. This is important because we build the package and during build the files could get modified. We don't want files in the store become modified.

Also, we will have to implement side-effects caching. But that's also for a separate PR and not high priority at the moment.

@KSXGitHub KSXGitHub requested a review from zkochan November 2, 2023 04:37
@zkochan zkochan merged commit 782a1ab into main Nov 2, 2023
@zkochan zkochan deleted the compatible-store-dir-structure branch November 2, 2023 16:04
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Compatible store dir structure.

2 participants