Skip to content

revert: "perf: use v8 serialize/deserialize instead of JSON (#9971)"#10420

Merged
zkochan merged 6 commits intomainfrom
fix/10409
Jan 13, 2026
Merged

revert: "perf: use v8 serialize/deserialize instead of JSON (#9971)"#10420
zkochan merged 6 commits intomainfrom
fix/10409

Conversation

@zkochan
Copy link
Copy Markdown
Member

@zkochan zkochan commented Jan 7, 2026

close #10409

@zkochan zkochan marked this pull request as ready for review January 13, 2026 11:50
@zkochan zkochan requested a review from weyert as a code owner January 13, 2026 11:50
Copilot AI review requested due to automatic review settings January 13, 2026 11:50
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request reverts PR #9971, which introduced v8 binary serialization for storing package metadata and index files. The revert changes back to JSON serialization throughout the codebase.

Changes:

  • Reverted from v8.serialize/deserialize to JSON.stringify/parse for all package metadata storage
  • Changed data structures from Map-based (PackageFiles, SideEffects) to Record-based (PackageFilesRaw, SideEffectsRaw)
  • Updated file extensions from .v8 to .json throughout the codebase
  • Removed the @pnpm/fs.v8-file package entirely

Reviewed changes

Copilot reviewed 70 out of 71 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
worker/src/start.ts Core serialization logic reverted to JSON with critical bug in temp file calculation
store/cafs-types/src/index.ts Added new Raw types for JSON-serializable versions of Map-based types
store/cafs/src/checkPkgFilesIntegrity.ts Updated to use Record/object access instead of Map methods
store/cafs/src/getFilePathInCafs.ts Changed file extensions from .v8 to .json
resolving/npm-resolver/src/pickPackage.ts Updated serialization to JSON.stringify
exec/pkg-requires-build/src/index.ts Enhanced to accept both Map and Record for backward compatibility
reviewing/license-scanner/src/getPkgInfo.ts Updated file access patterns from Map to Record
modules-mounter/daemon/src/cafsExplorer.ts Updated to use Object.keys() instead of Map.keys()
fs/v8-file/* Complete removal of the v8-file package
**/package.json Removed @pnpm/fs.v8-file dependency, added load-json-file
**/test/*.ts Updated all tests to use JSON serialization and Record access patterns
pnpm-lock.yaml Updated lockfile with new dependencies
.changeset/*.md Removed changesets for the reverted feature
Files not reviewed (1)
  • pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

// in order to avoid ENAMETOOLONG errors
const temp = `${filePath.slice(0, -9)}${process.pid}`
gfs.writeFileSync(temp, v8.serialize(data))
const temp = `${filePath.slice(0, -11)}${process.pid}`
Copy link

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The temp file name calculation is incorrect. The slice is using -11 which would remove 11 characters (the length of "-index.json"), but the files passed to this function actually end with ".json" (5 characters), not "-index.json". This will result in incorrect temporary file names that include part of the package name/identifier instead of just the hash portion. The slice should be filePath.slice(0, -5) to correctly remove ".json".

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, it is correct. We should remove "-index.json"

res.statusCode = 404
const error = { error: `${req.url!} does not match any route` }
res.end(v8.serialize(error))
res.end(JSON.stringify(error, replacer))

Check failure

Code scanning / CodeQL

Reflected cross-site scripting High

Cross-site scripting vulnerability due to a
user-provided value
.

Copilot Autofix

AI 2 months ago

In general, to prevent reflected XSS when echoing user input, you must either (1) avoid reflecting the raw value entirely, or (2) encode or escape it so that it cannot break out of its intended context (HTML, JSON, JavaScript, etc.) when a browser interprets the response. For JSON APIs, this typically means either not including raw user‑controlled strings in error messages, or ensuring the entire response is treated strictly as JSON by clients with the correct Content-Type, and optionally sanitizing especially dangerous fields.

For this file, the problematic flow is: req.url → template literal for errorJSON.stringifyres.end. The least intrusive, behavior‑preserving fix is to prevent any unexpected control characters in req.url (notably newlines) from influencing how the HTTP response is parsed or displayed. A common mitigation is to encode the URL using encodeURIComponent before embedding it, or, more simply, to strip dangerous characters so only a safe representation of the URL is included. Since we must not assume additional project utilities, we can use built‑in JavaScript functions.

Concretely, in store/server/src/createServer.ts around line 185, replace:

const error = { error: `${req.url!} does not match any route` }
res.end(JSON.stringify(error, replacer))

with code that ensures the user‑controlled URL is encoded before it is included in the error message. For example:

const safeUrl = encodeURIComponent(req.url ?? '')
const error = { error: `${safeUrl} does not match any route` }
res.end(JSON.stringify(error, replacer))

This preserves the information content (the client can still see which URL failed) while ensuring that special characters are percent‑encoded and cannot be interpreted by a browser as HTML or script content. No new imports are required because encodeURIComponent is globally available in Node.js.

Suggested changeset 1
store/server/src/createServer.ts

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/store/server/src/createServer.ts b/store/server/src/createServer.ts
--- a/store/server/src/createServer.ts
+++ b/store/server/src/createServer.ts
@@ -182,7 +182,8 @@
         break
       default: {
         res.statusCode = 404
-        const error = { error: `${req.url!} does not match any route` }
+        const safeUrl = encodeURIComponent(req.url ?? '')
+        const error = { error: `${safeUrl} does not match any route` }
         res.end(JSON.stringify(error, replacer))
       }
       }
EOF
@@ -182,7 +182,8 @@
break
default: {
res.statusCode = 404
const error = { error: `${req.url!} does not match any route` }
const safeUrl = encodeURIComponent(req.url ?? '')
const error = { error: `${safeUrl} does not match any route` }
res.end(JSON.stringify(error, replacer))
}
}
Copilot is powered by AI and may make mistakes. Always verify output.
Comment on lines +116 to +121
res.end(JSON.stringify({
error: {
message: err.message,
...JSON.parse(JSON.stringify(err)),
},
}))
}, replacer))

Check warning

Code scanning / CodeQL

Information exposure through a stack trace Medium

This information exposed to the user depends on
stack trace information
.
This information exposed to the user depends on
stack trace information
.

Copilot Autofix

AI 2 months ago

In general, the fix is to avoid sending detailed error objects (including potential stack traces) to clients. Instead, log the full error server-side and return a minimal, generic error response body, such as an error code and a simple message that does not leak implementation details.

Concretely, in store/server/src/createServer.ts, within the /requestPackage handler’s catch block (lines 114–121), replace the current logic that serializes the entire err object:

res.end(JSON.stringify({
  error: {
    message: err.message,
    ...JSON.parse(JSON.stringify(err)),
  },
}, replacer))

with a safer pattern that (a) logs the full error, including stack, using globalInfo, and (b) responds with a generic error payload that does not include stack trace or arbitrary error properties. For example:

globalInfo('store-server', { msg: 'requestPackage failed', err })
res.statusCode = 500
res.end(JSON.stringify({
  error: {
    message: 'Internal server error',
  },
}, replacer))

This preserves existing behavior in terms of using JSON and replacer, but removes the exposure of internal error details. No new imports are needed because globalInfo is already imported. The rest of the file remains unchanged; we only modify the catch (err: unknown) block inside the /requestPackage case.

Suggested changeset 1
store/server/src/createServer.ts

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/store/server/src/createServer.ts b/store/server/src/createServer.ts
--- a/store/server/src/createServer.ts
+++ b/store/server/src/createServer.ts
@@ -113,10 +113,11 @@
           res.end(JSON.stringify(pkgResponse.body, replacer))
         } catch (err: unknown) {
           assert(util.types.isNativeError(err))
+          globalInfo('store-server', { msg: 'requestPackage failed', err })
+          res.statusCode = 500
           res.end(JSON.stringify({
             error: {
-              message: err.message,
-              ...JSON.parse(JSON.stringify(err)),
+              message: 'Internal server error',
             },
           }, replacer))
         }
EOF
@@ -113,10 +113,11 @@
res.end(JSON.stringify(pkgResponse.body, replacer))
} catch (err: unknown) {
assert(util.types.isNativeError(err))
globalInfo('store-server', { msg: 'requestPackage failed', err })
res.statusCode = 500
res.end(JSON.stringify({
error: {
message: err.message,
...JSON.parse(JSON.stringify(err)),
message: 'Internal server error',
},
}, replacer))
}
Copilot is powered by AI and may make mistakes. Always verify output.
Comment on lines +133 to +138
res.end(JSON.stringify({
error: {
message: err.message,
...JSON.parse(JSON.stringify(err)),
},
}))
}, replacer))

Check warning

Code scanning / CodeQL

Information exposure through a stack trace Medium

This information exposed to the user depends on
stack trace information
.
This information exposed to the user depends on
stack trace information
.

Copilot Autofix

AI 2 months ago

To fix the problem, the server should stop returning the full serialized error object to the client and instead respond with a minimal, generic error payload. Any detailed information (including stack traces) should be logged server-side only. This keeps internal structure, file paths, and potential stack traces away from untrusted users while preserving debuggability for developers.

Concretely, in store/server/src/createServer.ts:

  • In the /requestPackage handler catch block (lines 114–122), replace the current res.end(JSON.stringify({ error: { message: err.message, ...JSON.parse(JSON.stringify(err)), }, }, replacer)) with a simpler response such as res.end(JSON.stringify({ error: { message: err.message }, }, replacer)). Optionally log the full error to the console or a logger, but do not send it back.
  • In the /fetchPackage handler catch block (lines 131–139), perform the same simplification: only send a minimal error structure (e.g., message) and avoid spreading the entire error into the response.
  • In the outermost catch block at lines 189–193, similarly avoid sending the full serialized error (jsonErr) and instead send a minimal generic error object. You may still log e server-side.

We do not need new imports or dependencies; Node’s built-in console.error or the existing globalInfo logger can be used to log detailed errors. The functional behaviour from the client’s perspective changes only in the shape of error responses (less detail), while success paths and control flow remain intact.

Suggested changeset 1
store/server/src/createServer.ts

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/store/server/src/createServer.ts b/store/server/src/createServer.ts
--- a/store/server/src/createServer.ts
+++ b/store/server/src/createServer.ts
@@ -113,10 +113,11 @@
           res.end(JSON.stringify(pkgResponse.body, replacer))
         } catch (err: unknown) {
           assert(util.types.isNativeError(err))
+          // Log full error details server-side, but do not expose them to the client
+          console.error('Error in /requestPackage handler:', err)
           res.end(JSON.stringify({
             error: {
               message: err.message,
-              ...JSON.parse(JSON.stringify(err)),
             },
           }, replacer))
         }
@@ -130,10 +128,11 @@
           res.end(JSON.stringify({ filesIndexFile: pkgResponse.filesIndexFile }, replacer))
         } catch (err: unknown) {
           assert(util.types.isNativeError(err))
+          // Log full error details server-side, but do not expose them to the client
+          console.error('Error in /fetchPackage handler:', err)
           res.end(JSON.stringify({
             error: {
               message: err.message,
-              ...JSON.parse(JSON.stringify(err)),
             },
           }, replacer))
         }
@@ -188,9 +184,13 @@
       }
     } catch (e: any) { // eslint-disable-line
       res.statusCode = 503
-      const jsonErr = JSON.parse(JSON.stringify(e))
-      jsonErr.message = e.message
-      res.end(JSON.stringify(jsonErr, replacer))
+      // Log full error details server-side, but return only a generic error to the client
+      console.error('Unhandled error in store server request handler:', e)
+      res.end(JSON.stringify({
+        error: {
+          message: e?.message ?? 'Service temporarily unavailable',
+        },
+      }, replacer))
     }
   })
 
EOF
@@ -113,10 +113,11 @@
res.end(JSON.stringify(pkgResponse.body, replacer))
} catch (err: unknown) {
assert(util.types.isNativeError(err))
// Log full error details server-side, but do not expose them to the client
console.error('Error in /requestPackage handler:', err)
res.end(JSON.stringify({
error: {
message: err.message,
...JSON.parse(JSON.stringify(err)),
},
}, replacer))
}
@@ -130,10 +128,11 @@
res.end(JSON.stringify({ filesIndexFile: pkgResponse.filesIndexFile }, replacer))
} catch (err: unknown) {
assert(util.types.isNativeError(err))
// Log full error details server-side, but do not expose them to the client
console.error('Error in /fetchPackage handler:', err)
res.end(JSON.stringify({
error: {
message: err.message,
...JSON.parse(JSON.stringify(err)),
},
}, replacer))
}
@@ -188,9 +184,13 @@
}
} catch (e: any) { // eslint-disable-line
res.statusCode = 503
const jsonErr = JSON.parse(JSON.stringify(e))
jsonErr.message = e.message
res.end(JSON.stringify(jsonErr, replacer))
// Log full error details server-side, but return only a generic error to the client
console.error('Unhandled error in store server request handler:', e)
res.end(JSON.stringify({
error: {
message: e?.message ?? 'Service temporarily unavailable',
},
}, replacer))
}
})

Copilot is powered by AI and may make mistakes. Always verify output.
const jsonErr = JSON.parse(JSON.stringify(e))
jsonErr.message = e.message
res.end(v8.serialize(jsonErr))
res.end(JSON.stringify(jsonErr, replacer))

Check warning

Code scanning / CodeQL

Information exposure through a stack trace Medium

This information exposed to the user depends on
stack trace information
.
This information exposed to the user depends on
stack trace information
.

Copilot Autofix

AI 2 months ago

In general, the fix is to avoid returning raw or semi-raw error objects to the client. Instead, send a minimal, generic error response (for example, an HTTP 5xx status with a simple error code or message) and log the detailed error, including stack trace, on the server side only.

For this specific code, the best fix with minimal functional change is:

  • Keep the 503 status code to indicate a server error.
  • Log the full error e (or e.stack) locally using an existing logging mechanism from this file (globalInfo from @pnpm/logger is already imported).
  • Replace the current construction of jsonErr (which clones e and exposes its fields) with a small, explicit object that contains only non-sensitive information, such as a generic message like "Service unavailable" and optionally a stable error code.
  • Continue to use JSON.stringify(..., replacer) so the rest of the serialization behavior is preserved, but now over a controlled, minimal object.

Concretely, in store/server/src/createServer.ts, update the catch block at lines 189–194 to:

  • Add a call like globalInfo(util.inspect(e, { depth: null })) or similar to log details.
  • Replace const jsonErr = JSON.parse(JSON.stringify(e)); jsonErr.message = e.message; res.end(JSON.stringify(jsonErr, replacer)); with something like:
const jsonErr = { error: 'Internal server error' }
res.end(JSON.stringify(jsonErr, replacer))

This keeps the external behavior of returning JSON on error and a 503 status, but no longer exposes the internals of the thrown error or any stack information. No new imports are strictly required; util and globalInfo are already imported if we want to log more detail.

Suggested changeset 1
store/server/src/createServer.ts

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/store/server/src/createServer.ts b/store/server/src/createServer.ts
--- a/store/server/src/createServer.ts
+++ b/store/server/src/createServer.ts
@@ -188,8 +188,9 @@
       }
     } catch (e: any) { // eslint-disable-line
       res.statusCode = 503
-      const jsonErr = JSON.parse(JSON.stringify(e))
-      jsonErr.message = e.message
+      // Log detailed error information on the server, but do not expose it to the client
+      globalInfo(`Request handling failed: ${util.inspect(e, { depth: null })}`)
+      const jsonErr = { error: 'Internal server error' }
       res.end(JSON.stringify(jsonErr, replacer))
     }
   })
EOF
@@ -188,8 +188,9 @@
}
} catch (e: any) { // eslint-disable-line
res.statusCode = 503
const jsonErr = JSON.parse(JSON.stringify(e))
jsonErr.message = e.message
// Log detailed error information on the server, but do not expose it to the client
globalInfo(`Request handling failed: ${util.inspect(e, { depth: null })}`)
const jsonErr = { error: 'Internal server error' }
res.end(JSON.stringify(jsonErr, replacer))
}
})
Copilot is powered by AI and may make mistakes. Always verify output.
@zkochan zkochan merged commit da112f7 into main Jan 13, 2026
11 of 13 checks passed
@zkochan zkochan deleted the fix/10409 branch January 13, 2026 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Performance regression after switching to v8 serialization format

2 participants