Skip to content

fix: reduce the side-effects cache key length#7563

Merged
zkochan merged 6 commits intomainfrom
fix/5056
Jan 25, 2024
Merged

fix: reduce the side-effects cache key length#7563
zkochan merged 6 commits intomainfrom
fix/5056

Conversation

@zkochan
Copy link
Copy Markdown
Member

@zkochan zkochan commented Jan 23, 2024

close #5056

@zkochan zkochan requested a review from a team January 24, 2024 01:20
jakebailey
jakebailey approved these changes Jan 24, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@jakebailey jakebailey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW it seems like https://www.npmjs.com/package/node-object-hash is 10x faster (if you can believe their benchmarks).

@jakebailey
Copy link
Copy Markdown
Member

jakebailey commented Jan 24, 2024

I can confirm that this PR fixes the test case I had for #5056.

image

Which is roughly:

$ git clone https://github.com/DefinitelyTyped/DefinitelyTyped.git
$ git switch --detach c76c419153bf70858ab82843483657ae1bf11f17
$ pnpm install --filter . --filter '...[HEAD^1]...'
$ pnpm ls --depth Infinity --parseable --filter '...@types/**[HEAD^1]...'

(This goofy command is part of a workaround for #7283; unfortunately it can stil OOM but this PR has fixed more typical runs.)

@zkochan
Copy link
Copy Markdown
Member Author

zkochan commented Jan 24, 2024

oh, ok

I'll use node-object-hash instead.

@zkochan zkochan changed the title fix: reduce the side-effects cache key fix: reduce the side-effects cache key length Jan 24, 2024
BlackHole1
BlackHole1 approved these changes Jan 25, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@BlackHole1 BlackHole1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zkochan zkochan merged commit a3f90c2 into main Jan 25, 2024
@zkochan zkochan deleted the fix/5056 branch January 25, 2024 02:17
zkochan added a commit that referenced this pull request Jan 25, 2024
@zkochan
fix: reduce the side-effects cache key length (#7563)
0c38332 
close #5056
@jakebailey jakebailey mentioned this pull request Jan 27, 2024
Merged
@samtsai
Copy link
Copy Markdown

samtsai commented Jan 29, 2024 •
edited
Loading

I'm still seeing this:

[2024-01-29T17:26:39.979Z] Removing intermediate container 69c5274b8d3e
[2024-01-29T17:26:39.979Z]  ---> d64ca7992607
[2024-01-29T17:26:39.979Z] Step 11/12 : COPY package.json ./
[2024-01-29T17:26:39.979Z]  ---> 2a9e8f66a8ea
[2024-01-29T17:26:39.979Z] Step 12/12 : RUN pnpm install -r --offline
[2024-01-29T17:26:39.979Z]  ---> Running in 2806c668ddea
[2024-01-29T17:26:47.860Z] .../esbuild@0.20.0/node_modules/esbuild postinstall$ node install.js
[2024-01-29T17:26:47.860Z] .../esbuild@0.19.12/node_modules/esbuild postinstall$ node install.js
[2024-01-29T17:26:47.860Z] .../esbuild@0.20.0/node_modules/esbuild postinstall: Done
[2024-01-29T17:26:47.860Z] .../esbuild@0.19.12/node_modules/esbuild postinstall: Done
[2024-01-29T17:28:02.158Z]  ERROR  Invalid string length
[2024-01-29T17:28:02.158Z] 
[2024-01-29T17:28:02.158Z] RangeError: Invalid string length
[2024-01-29T17:28:02.158Z]     at Array.join (<anonymous>)
[2024-01-29T17:28:02.158Z]     at Array.toString (<anonymous>)
[2024-01-29T17:28:02.158Z]     at Object._object (/home/builder/.cache/node/corepack/pnpm/8.15.0/dist/pnpm.cjs:124428:29)
[2024-01-29T17:28:02.158Z]     at objectToString (/home/builder/.cache/node/corepack/pnpm/8.15.0/dist/pnpm.cjs:124574:61)
[2024-01-29T17:28:02.158Z]     at hash (/home/builder/.cache/node/corepack/pnpm/8.15.0/dist/pnpm.cjs:124597:24)
[2024-01-29T17:28:02.158Z]     at calcDepState (/home/builder/.cache/node/corepack/pnpm/8.15.0/dist/pnpm.cjs:124639:76)
[2024-01-29T17:28:02.158Z]     at /home/builder/.cache/node/corepack/pnpm/8.15.0/dist/pnpm.cjs:126132:69
[2024-01-29T17:28:02.263Z] The command '/bin/sh -c pnpm install -r --offline' returned a non-zero code: 1

Is there another step I need to do?

@jakebailey
Copy link
Copy Markdown
Member

jakebailey commented Jan 29, 2024 •
edited
Loading

Reading https://github.com/SkeLLLa/node-object-hash/blob/master/src/stringifiers.ts#L392, it looks like this library builds up a hash by building a huge string representation and then hashing it... which was what we were already attempting to avoid... (Related is SkeLLLa/node-object-hash#63 (comment), which is the same root cause; building up huge strings).

I guess my suggestion for a "faster" library was not a good one, because it looks like object-hash avoids this problem by streaming the hash updates: https://github.com/puleos/object-hash/blob/master/index.js#L173

(In the same way, https://www.npmjs.com/package/hash-object cannot be used because it just deep sorts the object and JSONifies it.)

@zkochan
Copy link
Copy Markdown
Member Author

zkochan commented Jan 29, 2024

Ok, let's switch to object-hash then

@jakebailey
Copy link
Copy Markdown
Member

jakebailey commented Jan 29, 2024

Will send a PR shortly; was already working on it.

@jakebailey jakebailey mentioned this pull request Jan 29, 2024
Merged
@jakebailey
Copy link
Copy Markdown
Member

jakebailey commented Jan 29, 2024

Sent #7591.

@zkochan zkochan mentioned this pull request Jan 31, 2024
Merged
zkochan added a commit to teambit/bit that referenced this pull request Jan 31, 2024
@zkochan
fix: update pnpm (#8481)
96522bf 
## Proposed Changes

- pnpm/pnpm#7583
- pnpm/pnpm#7563
- pnpm/pnpm#7591
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jakebailey jakebailey jakebailey approved these changes

@BlackHole1 BlackHole1 BlackHole1 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Invalid string length

4 participants

@zkochan @jakebailey @samtsai @BlackHole1