Conversation
There was a problem hiding this comment.
Pull request overview
This pull request reverts PR #9971, which introduced v8 binary serialization for storing package metadata and index files. The revert changes back to JSON serialization throughout the codebase.
Changes:
- Reverted from v8.serialize/deserialize to JSON.stringify/parse for all package metadata storage
- Changed data structures from Map-based (PackageFiles, SideEffects) to Record-based (PackageFilesRaw, SideEffectsRaw)
- Updated file extensions from
.v8to.jsonthroughout the codebase - Removed the
@pnpm/fs.v8-filepackage entirely
Reviewed changes
Copilot reviewed 70 out of 71 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| worker/src/start.ts | Core serialization logic reverted to JSON with critical bug in temp file calculation |
| store/cafs-types/src/index.ts | Added new Raw types for JSON-serializable versions of Map-based types |
| store/cafs/src/checkPkgFilesIntegrity.ts | Updated to use Record/object access instead of Map methods |
| store/cafs/src/getFilePathInCafs.ts | Changed file extensions from .v8 to .json |
| resolving/npm-resolver/src/pickPackage.ts | Updated serialization to JSON.stringify |
| exec/pkg-requires-build/src/index.ts | Enhanced to accept both Map and Record for backward compatibility |
| reviewing/license-scanner/src/getPkgInfo.ts | Updated file access patterns from Map to Record |
| modules-mounter/daemon/src/cafsExplorer.ts | Updated to use Object.keys() instead of Map.keys() |
| fs/v8-file/* | Complete removal of the v8-file package |
| **/package.json | Removed @pnpm/fs.v8-file dependency, added load-json-file |
| **/test/*.ts | Updated all tests to use JSON serialization and Record access patterns |
| pnpm-lock.yaml | Updated lockfile with new dependencies |
| .changeset/*.md | Removed changesets for the reverted feature |
Files not reviewed (1)
- pnpm-lock.yaml: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // in order to avoid ENAMETOOLONG errors | ||
| const temp = `${filePath.slice(0, -9)}${process.pid}` | ||
| gfs.writeFileSync(temp, v8.serialize(data)) | ||
| const temp = `${filePath.slice(0, -11)}${process.pid}` |
There was a problem hiding this comment.
The temp file name calculation is incorrect. The slice is using -11 which would remove 11 characters (the length of "-index.json"), but the files passed to this function actually end with ".json" (5 characters), not "-index.json". This will result in incorrect temporary file names that include part of the package name/identifier instead of just the hash portion. The slice should be filePath.slice(0, -5) to correctly remove ".json".
There was a problem hiding this comment.
no, it is correct. We should remove "-index.json"
| res.statusCode = 404 | ||
| const error = { error: `${req.url!} does not match any route` } | ||
| res.end(v8.serialize(error)) | ||
| res.end(JSON.stringify(error, replacer)) |
Check failure
Code scanning / CodeQL
Reflected cross-site scripting High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 2 months ago
In general, to prevent reflected XSS when echoing user input, you must either (1) avoid reflecting the raw value entirely, or (2) encode or escape it so that it cannot break out of its intended context (HTML, JSON, JavaScript, etc.) when a browser interprets the response. For JSON APIs, this typically means either not including raw user‑controlled strings in error messages, or ensuring the entire response is treated strictly as JSON by clients with the correct Content-Type, and optionally sanitizing especially dangerous fields.
For this file, the problematic flow is: req.url → template literal for error → JSON.stringify → res.end. The least intrusive, behavior‑preserving fix is to prevent any unexpected control characters in req.url (notably newlines) from influencing how the HTTP response is parsed or displayed. A common mitigation is to encode the URL using encodeURIComponent before embedding it, or, more simply, to strip dangerous characters so only a safe representation of the URL is included. Since we must not assume additional project utilities, we can use built‑in JavaScript functions.
Concretely, in store/server/src/createServer.ts around line 185, replace:
const error = { error: `${req.url!} does not match any route` }
res.end(JSON.stringify(error, replacer))with code that ensures the user‑controlled URL is encoded before it is included in the error message. For example:
const safeUrl = encodeURIComponent(req.url ?? '')
const error = { error: `${safeUrl} does not match any route` }
res.end(JSON.stringify(error, replacer))This preserves the information content (the client can still see which URL failed) while ensuring that special characters are percent‑encoded and cannot be interpreted by a browser as HTML or script content. No new imports are required because encodeURIComponent is globally available in Node.js.
| @@ -182,7 +182,8 @@ | ||
| break | ||
| default: { | ||
| res.statusCode = 404 | ||
| const error = { error: `${req.url!} does not match any route` } | ||
| const safeUrl = encodeURIComponent(req.url ?? '') | ||
| const error = { error: `${safeUrl} does not match any route` } | ||
| res.end(JSON.stringify(error, replacer)) | ||
| } | ||
| } |
| res.end(JSON.stringify({ | ||
| error: { | ||
| message: err.message, | ||
| ...JSON.parse(JSON.stringify(err)), | ||
| }, | ||
| })) | ||
| }, replacer)) |
Check warning
Code scanning / CodeQL
Information exposure through a stack trace Medium
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 2 months ago
In general, the fix is to avoid sending detailed error objects (including potential stack traces) to clients. Instead, log the full error server-side and return a minimal, generic error response body, such as an error code and a simple message that does not leak implementation details.
Concretely, in store/server/src/createServer.ts, within the /requestPackage handler’s catch block (lines 114–121), replace the current logic that serializes the entire err object:
res.end(JSON.stringify({
error: {
message: err.message,
...JSON.parse(JSON.stringify(err)),
},
}, replacer))with a safer pattern that (a) logs the full error, including stack, using globalInfo, and (b) responds with a generic error payload that does not include stack trace or arbitrary error properties. For example:
globalInfo('store-server', { msg: 'requestPackage failed', err })
res.statusCode = 500
res.end(JSON.stringify({
error: {
message: 'Internal server error',
},
}, replacer))This preserves existing behavior in terms of using JSON and replacer, but removes the exposure of internal error details. No new imports are needed because globalInfo is already imported. The rest of the file remains unchanged; we only modify the catch (err: unknown) block inside the /requestPackage case.
| @@ -113,10 +113,11 @@ | ||
| res.end(JSON.stringify(pkgResponse.body, replacer)) | ||
| } catch (err: unknown) { | ||
| assert(util.types.isNativeError(err)) | ||
| globalInfo('store-server', { msg: 'requestPackage failed', err }) | ||
| res.statusCode = 500 | ||
| res.end(JSON.stringify({ | ||
| error: { | ||
| message: err.message, | ||
| ...JSON.parse(JSON.stringify(err)), | ||
| message: 'Internal server error', | ||
| }, | ||
| }, replacer)) | ||
| } |
| res.end(JSON.stringify({ | ||
| error: { | ||
| message: err.message, | ||
| ...JSON.parse(JSON.stringify(err)), | ||
| }, | ||
| })) | ||
| }, replacer)) |
Check warning
Code scanning / CodeQL
Information exposure through a stack trace Medium
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 2 months ago
To fix the problem, the server should stop returning the full serialized error object to the client and instead respond with a minimal, generic error payload. Any detailed information (including stack traces) should be logged server-side only. This keeps internal structure, file paths, and potential stack traces away from untrusted users while preserving debuggability for developers.
Concretely, in store/server/src/createServer.ts:
- In the
/requestPackagehandlercatchblock (lines 114–122), replace the currentres.end(JSON.stringify({ error: { message: err.message, ...JSON.parse(JSON.stringify(err)), }, }, replacer))with a simpler response such asres.end(JSON.stringify({ error: { message: err.message }, }, replacer)). Optionally log the full error to the console or a logger, but do not send it back. - In the
/fetchPackagehandlercatchblock (lines 131–139), perform the same simplification: only send a minimal error structure (e.g., message) and avoid spreading the entire error into the response. - In the outermost
catchblock at lines 189–193, similarly avoid sending the full serialized error (jsonErr) and instead send a minimal generic error object. You may still logeserver-side.
We do not need new imports or dependencies; Node’s built-in console.error or the existing globalInfo logger can be used to log detailed errors. The functional behaviour from the client’s perspective changes only in the shape of error responses (less detail), while success paths and control flow remain intact.
| @@ -113,10 +113,11 @@ | ||
| res.end(JSON.stringify(pkgResponse.body, replacer)) | ||
| } catch (err: unknown) { | ||
| assert(util.types.isNativeError(err)) | ||
| // Log full error details server-side, but do not expose them to the client | ||
| console.error('Error in /requestPackage handler:', err) | ||
| res.end(JSON.stringify({ | ||
| error: { | ||
| message: err.message, | ||
| ...JSON.parse(JSON.stringify(err)), | ||
| }, | ||
| }, replacer)) | ||
| } | ||
| @@ -130,10 +128,11 @@ | ||
| res.end(JSON.stringify({ filesIndexFile: pkgResponse.filesIndexFile }, replacer)) | ||
| } catch (err: unknown) { | ||
| assert(util.types.isNativeError(err)) | ||
| // Log full error details server-side, but do not expose them to the client | ||
| console.error('Error in /fetchPackage handler:', err) | ||
| res.end(JSON.stringify({ | ||
| error: { | ||
| message: err.message, | ||
| ...JSON.parse(JSON.stringify(err)), | ||
| }, | ||
| }, replacer)) | ||
| } | ||
| @@ -188,9 +184,13 @@ | ||
| } | ||
| } catch (e: any) { // eslint-disable-line | ||
| res.statusCode = 503 | ||
| const jsonErr = JSON.parse(JSON.stringify(e)) | ||
| jsonErr.message = e.message | ||
| res.end(JSON.stringify(jsonErr, replacer)) | ||
| // Log full error details server-side, but return only a generic error to the client | ||
| console.error('Unhandled error in store server request handler:', e) | ||
| res.end(JSON.stringify({ | ||
| error: { | ||
| message: e?.message ?? 'Service temporarily unavailable', | ||
| }, | ||
| }, replacer)) | ||
| } | ||
| }) | ||
|
|
| const jsonErr = JSON.parse(JSON.stringify(e)) | ||
| jsonErr.message = e.message | ||
| res.end(v8.serialize(jsonErr)) | ||
| res.end(JSON.stringify(jsonErr, replacer)) |
Check warning
Code scanning / CodeQL
Information exposure through a stack trace Medium
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 2 months ago
In general, the fix is to avoid returning raw or semi-raw error objects to the client. Instead, send a minimal, generic error response (for example, an HTTP 5xx status with a simple error code or message) and log the detailed error, including stack trace, on the server side only.
For this specific code, the best fix with minimal functional change is:
- Keep the 503 status code to indicate a server error.
- Log the full error
e(ore.stack) locally using an existing logging mechanism from this file (globalInfofrom@pnpm/loggeris already imported). - Replace the current construction of
jsonErr(which cloneseand exposes its fields) with a small, explicit object that contains only non-sensitive information, such as a generic message like"Service unavailable"and optionally a stable error code. - Continue to use
JSON.stringify(..., replacer)so the rest of the serialization behavior is preserved, but now over a controlled, minimal object.
Concretely, in store/server/src/createServer.ts, update the catch block at lines 189–194 to:
- Add a call like
globalInfo(util.inspect(e, { depth: null }))or similar to log details. - Replace
const jsonErr = JSON.parse(JSON.stringify(e)); jsonErr.message = e.message; res.end(JSON.stringify(jsonErr, replacer));with something like:
const jsonErr = { error: 'Internal server error' }
res.end(JSON.stringify(jsonErr, replacer))This keeps the external behavior of returning JSON on error and a 503 status, but no longer exposes the internals of the thrown error or any stack information. No new imports are strictly required; util and globalInfo are already imported if we want to log more detail.
| @@ -188,8 +188,9 @@ | ||
| } | ||
| } catch (e: any) { // eslint-disable-line | ||
| res.statusCode = 503 | ||
| const jsonErr = JSON.parse(JSON.stringify(e)) | ||
| jsonErr.message = e.message | ||
| // Log detailed error information on the server, but do not expose it to the client | ||
| globalInfo(`Request handling failed: ${util.inspect(e, { depth: null })}`) | ||
| const jsonErr = { error: 'Internal server error' } | ||
| res.end(JSON.stringify(jsonErr, replacer)) | ||
| } | ||
| }) |
close #10409