Skip to content

Commit 34875b2

Browse files
zkochanzkochan
committed
fix: infer missing platform fields of optional deps from the package name
Some registries strip the os/cpu/libc fields (or just libc) from the version objects of the packuments they serve. Resolution then saw every platform-specific optional dependency as platform-unrestricted, so pnpm downloaded and installed the binaries of every platform regardless of supportedArchitectures, and wrote lockfile entries without the platform fields, which broke installs from that lockfile on every machine. Platform-specific binary packages encode their platform in the package name (e.g. @nx/nx-win32-arm64-msvc), so packageIsInstallable now fills the missing platform fields of an optional dependency from the name's tokens. Since every install path decides installability through that check before fetching, foreign-platform binaries are skipped without even downloading them, in fresh resolution and in headless installs with both node linkers alike. A package that declares no platform fields at all is treated as platform-specific only when an operating system is recognized in its name, so a generic name segment (such as 'arm' on its own) never gets a package skipped. Fixes #11702 Fixes #9940
1 parent d2125b8 commit 34875b2

6 files changed

Lines changed: 373 additions & 1 deletion

File tree

‎.changeset/spicy-pots-wonder.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
"@pnpm/config.package-is-installable": patch
3+
"pnpm": patch
4+
---
5+
6+
Platform-specific optional dependencies are now skipped even when their `os`/`cpu`/`libc` fields are missing from the registry metadata or the lockfile. Some registries strip these fields from the package metadata, which made pnpm download and install the binaries of every platform regardless of `supportedArchitectures`. The missing platform fields of an optional dependency are now inferred from its name (e.g. `@nx/nx-win32-arm64-msvc` → `os: win32`, `cpu: arm64`), so foreign-platform binaries are skipped without even downloading them [#11702](https://github.com/pnpm/pnpm/issues/11702).

‎config/package-is-installable/src/index.ts‎

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,11 @@ import type { SupportedArchitectures } from '@pnpm/types'
77

88
import { checkEngine, UnsupportedEngineError, type WantedEngine } from './checkEngine.js'
99
import { checkPlatform, UnsupportedPlatformError } from './checkPlatform.js'
10+
import { inferPlatformFromPackageName } from './inferPlatformFromPackageName.js'
1011

1112
export type { Engine } from './checkEngine.js'
1213
export type { Platform, WantedPlatform } from './checkPlatform.js'
14+
export { inferPlatformFromPackageName } from './inferPlatformFromPackageName.js'
1315

1416
export {
1517
UnsupportedEngineError,
@@ -36,7 +38,7 @@ export function packageIsInstallable (
3638
supportedArchitectures?: SupportedArchitectures
3739
}
3840
): boolean | null {
39-
const warn = checkPackage(pkgId, pkg, options)
41+
const warn = checkPackage(pkgId, { engines: pkg.engines, ...effectivePlatform(pkg, options.optional) }, options)
4042

4143
if (warn == null) return true
4244

@@ -65,6 +67,36 @@ export function packageIsInstallable (
6567
return null
6668
}
6769

70+
interface PlatformFields {
71+
cpu?: string[]
72+
os?: string[]
73+
libc?: string[]
74+
}
75+
76+
/**
77+
* The platform fields of an optional dependency may be incomplete: some
78+
* registries strip os/cpu/libc (or just libc) from the metadata they serve,
79+
* and lockfile entries written from such metadata lack them too. For a
80+
* platform-specific binary the package name carries the same information, so
81+
* each missing field is filled from the name's tokens. A package that
82+
* declares no platform fields at all is treated as platform-specific only
83+
* when an operating system is recognized in its name — a generic name
84+
* segment (e.g. `arm` on its own) never marks it as such.
85+
* https://github.com/pnpm/pnpm/issues/11702
86+
*/
87+
function effectivePlatform (pkg: PlatformFields & { name: string }, optional: boolean): PlatformFields {
88+
if (!optional || (pkg.os != null && pkg.cpu != null && pkg.libc != null)) return pkg
89+
const inferred = inferPlatformFromPackageName(pkg.name)
90+
if (inferred == null) return pkg
91+
const pkgDeclaresPlatform = pkg.os != null || pkg.cpu != null || pkg.libc != null
92+
if (!pkgDeclaresPlatform && inferred.os == null) return pkg
93+
return {
94+
os: pkg.os ?? inferred.os,
95+
cpu: pkg.cpu ?? inferred.cpu,
96+
libc: pkg.libc ?? inferred.libc,
97+
}
98+
}
99+
68100
export function checkPackage (
69101
pkgId: string,
70102
manifest: {

‎config/package-is-installable/src/inferPlatformFromPackageName.ts‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
const OS_BY_TOKEN = new Map([
2+
['aix', 'aix'],
3+
['android', 'android'],
4+
['darwin', 'darwin'],
5+
['macos', 'darwin'],
6+
['osx', 'darwin'],
7+
['freebsd', 'freebsd'],
8+
['linux', 'linux'],
9+
['netbsd', 'netbsd'],
10+
['openbsd', 'openbsd'],
11+
['openharmony', 'openharmony'],
12+
['sunos', 'sunos'],
13+
['win32', 'win32'],
14+
['windows', 'win32'],
15+
])
16+
17+
const CPU_BY_TOKEN = new Map([
18+
['arm', 'arm'],
19+
['armv6', 'arm'],
20+
['armv7', 'arm'],
21+
['arm64', 'arm64'],
22+
['aarch64', 'arm64'],
23+
['ia32', 'ia32'],
24+
['loong64', 'loong64'],
25+
['mips64el', 'mips64el'],
26+
['ppc64', 'ppc64'],
27+
['ppc64le', 'ppc64'],
28+
['riscv64', 'riscv64'],
29+
['s390x', 's390x'],
30+
['x64', 'x64'],
31+
['amd64', 'x64'],
32+
['wasm32', 'wasm32'],
33+
])
34+
35+
const LIBC_BY_TOKEN = new Map([
36+
['glibc', 'glibc'],
37+
['gnu', 'glibc'],
38+
['gnueabihf', 'glibc'],
39+
['musl', 'musl'],
40+
['musleabihf', 'musl'],
41+
])
42+
43+
export interface PlatformInferredFromPackageName {
44+
os?: string[]
45+
cpu?: string[]
46+
libc?: string[]
47+
}
48+
49+
/**
50+
* Infers the supported platforms of a package from the tokens of its name,
51+
* e.g. `@nx/nx-win32-arm64-msvc` → `{ os: ['win32'], cpu: ['arm64'] }`.
52+
* Platform-specific binary packages follow this naming convention, which is
53+
* the only platform signal left when their os/cpu/libc manifest fields are
54+
* absent. Returns null when no platform token is recognized in the name.
55+
*/
56+
export function inferPlatformFromPackageName (name: string): PlatformInferredFromPackageName | null {
57+
const nameWithoutScope = name.includes('/') ? name.slice(name.indexOf('/') + 1) : name
58+
const tokens = nameWithoutScope.toLowerCase().split(/[-_.]/)
59+
const os = pickTokenValues(tokens, OS_BY_TOKEN)
60+
const cpu = pickTokenValues(tokens, CPU_BY_TOKEN)
61+
const libc = pickTokenValues(tokens, LIBC_BY_TOKEN)
62+
if (os == null && cpu == null && libc == null) return null
63+
return {
64+
...(os != null ? { os } : {}),
65+
...(cpu != null ? { cpu } : {}),
66+
...(libc != null ? { libc } : {}),
67+
}
68+
}
69+
70+
function pickTokenValues (tokens: string[], valueByToken: Map<string, string>): string[] | undefined {
71+
const values = new Set<string>()
72+
for (const token of tokens) {
73+
const value = valueByToken.get(token)
74+
if (value != null) {
75+
values.add(value)
76+
}
77+
}
78+
return values.size > 0 ? Array.from(values) : undefined
79+
}

‎config/package-is-installable/test/inferPlatformFromPackageName.ts‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
import { expect, jest, test } from '@jest/globals'
2+
import type * as DetectLibc from 'detect-libc'
3+
4+
jest.mock('detect-libc', () => {
5+
const original = jest.requireActual<typeof DetectLibc>('detect-libc')
6+
return {
7+
...original,
8+
familySync: () => 'glibc',
9+
}
10+
})
11+
12+
const { inferPlatformFromPackageName } = await import('../lib/inferPlatformFromPackageName.js')
13+
const { packageIsInstallable } = await import('../lib/index.js')
14+
15+
test.each([
16+
['@nx/nx-win32-arm64-msvc', { os: ['win32'], cpu: ['arm64'] }],
17+
['@nx/nx-linux-arm-gnueabihf', { os: ['linux'], cpu: ['arm'], libc: ['glibc'] }],
18+
['@nx/nx-linux-x64-gnu', { os: ['linux'], cpu: ['x64'], libc: ['glibc'] }],
19+
['@esbuild/aix-ppc64', { os: ['aix'], cpu: ['ppc64'] }],
20+
['@esbuild/openharmony-arm64', { os: ['openharmony'], cpu: ['arm64'] }],
21+
['@biomejs/cli-linux-x64-musl', { os: ['linux'], cpu: ['x64'], libc: ['musl'] }],
22+
['@typescript/native-preview-darwin-arm64', { os: ['darwin'], cpu: ['arm64'] }],
23+
['turbo-windows-64', { os: ['win32'] }],
24+
['esbuild-darwin-64', { os: ['darwin'] }],
25+
['bun-linux-aarch64', { os: ['linux'], cpu: ['arm64'] }],
26+
['sharp-linux-armv7', { os: ['linux'], cpu: ['arm'] }],
27+
['is-arm', { cpu: ['arm'] }],
28+
['fsevents', null],
29+
['lodash', null],
30+
['@pnpm.e2e/not-compatible-with-any-os', null],
31+
])('inferPlatformFromPackageName(%s)', (name, inferred) => {
32+
expect(inferPlatformFromPackageName(name)).toStrictEqual(inferred)
33+
})
34+
35+
test('an optional dependency without platform fields is not installable when its name declares an unsupported platform', () => {
36+
expect(packageIsInstallable('@nx/nx-win32-arm64-msvc@1.0.0', {
37+
name: '@nx/nx-win32-arm64-msvc',
38+
version: '1.0.0',
39+
}, {
40+
optional: true,
41+
lockfileDir: process.cwd(),
42+
supportedArchitectures: { os: ['linux'], cpu: ['x64'] },
43+
})).toBe(false)
44+
})
45+
46+
test('a missing libc field is taken from the package name even when the other platform fields are declared', () => {
47+
const options = {
48+
optional: true,
49+
lockfileDir: process.cwd(),
50+
supportedArchitectures: { os: ['linux'], cpu: ['x64'], libc: ['glibc'] },
51+
}
52+
expect(packageIsInstallable('@nx/nx-linux-x64-musl@1.0.0', {
53+
name: '@nx/nx-linux-x64-musl',
54+
version: '1.0.0',
55+
os: ['linux'],
56+
cpu: ['x64'],
57+
}, options)).toBe(false)
58+
expect(packageIsInstallable('@nx/nx-linux-x64-gnu@1.0.0', {
59+
name: '@nx/nx-linux-x64-gnu',
60+
version: '1.0.0',
61+
os: ['linux'],
62+
cpu: ['x64'],
63+
}, options)).toBe(true)
64+
})
65+
66+
test('a missing cpu field is taken from the name of a package that declares its platform', () => {
67+
expect(packageIsInstallable('@pnpm.e2e/some-pkg-arm64@1.0.0', {
68+
name: '@pnpm.e2e/some-pkg-arm64',
69+
version: '1.0.0',
70+
os: ['linux'],
71+
}, {
72+
optional: true,
73+
lockfileDir: process.cwd(),
74+
supportedArchitectures: { os: ['linux'], cpu: ['x64'] },
75+
})).toBe(false)
76+
})
77+
78+
test('the platform fields of the manifest take precedence over the package name', () => {
79+
expect(packageIsInstallable('@pnpm.e2e/win32-binary@1.0.0', {
80+
name: '@pnpm.e2e/win32-binary',
81+
version: '1.0.0',
82+
os: ['linux'],
83+
cpu: ['x64'],
84+
libc: ['glibc'],
85+
}, {
86+
optional: true,
87+
lockfileDir: process.cwd(),
88+
supportedArchitectures: { os: ['linux'], cpu: ['x64'], libc: ['glibc'] },
89+
})).toBe(true)
90+
})
91+
92+
test('a package without any declared platform field is not skipped when its name has no operating system token', () => {
93+
expect(packageIsInstallable('is-arm@1.0.0', {
94+
name: 'is-arm',
95+
version: '1.0.0',
96+
}, {
97+
optional: true,
98+
lockfileDir: process.cwd(),
99+
supportedArchitectures: { os: ['linux'], cpu: ['x64'] },
100+
})).toBe(true)
101+
})
102+
103+
test('the platform is not inferred from the name of a non-optional dependency', () => {
104+
expect(packageIsInstallable('@nx/nx-win32-arm64-msvc@1.0.0', {
105+
name: '@nx/nx-win32-arm64-msvc',
106+
version: '1.0.0',
107+
}, {
108+
optional: false,
109+
lockfileDir: process.cwd(),
110+
supportedArchitectures: { os: ['linux'], cpu: ['x64'] },
111+
})).toBe(true)
112+
})

‎installing/deps-installer/test/install/optionalDependencies.ts‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,10 @@
11
import fs from 'node:fs'
2+
import http from 'node:http'
3+
import type { AddressInfo } from 'node:net'
24
import path from 'node:path'
35

46
import { describe, expect, jest, test } from '@jest/globals'
7+
import { WANTED_LOCKFILE } from '@pnpm/constants'
58
import {
69
addDependenciesToPackage,
710
install,
@@ -11,10 +14,12 @@ import {
1114
} from '@pnpm/installing.deps-installer'
1215
import type { LockfileFile } from '@pnpm/lockfile.fs'
1316
import { prepareEmpty, preparePackages } from '@pnpm/prepare'
17+
import { REGISTRY_MOCK_PORT } from '@pnpm/testing.registry-mock'
1418
import type { ProjectRootDir } from '@pnpm/types'
1519
import { rimrafSync } from '@zkochan/rimraf'
1620
import deepRequireCwd from 'deep-require-cwd'
1721
import { readYamlFileSync } from 'read-yaml-file'
22+
import { writeYamlFileSync } from 'write-yaml-file'
1823

1924
import { testDefaults } from '../utils/index.js'
2025

@@ -140,6 +145,70 @@ test('skip optional dependency that does not support the current OS', async () =
140145
}
141146
})
142147

148+
// Test case for https://github.com/pnpm/pnpm/issues/11702
149+
test('skip optional dependencies whose names declare unsupported platforms when the registry metadata has no platform fields', async () => {
150+
const project = prepareEmpty()
151+
const server = createMetadataStrippingRegistryProxy()
152+
await new Promise<void>((resolve) => {
153+
server.listen(0, resolve)
154+
})
155+
const registryProxy = `http://localhost:${(server.address() as AddressInfo).port}/`
156+
try {
157+
await install({
158+
dependencies: {
159+
'@pnpm.e2e/has-many-optional-deps': '1.0.0',
160+
},
161+
}, testDefaults({
162+
registries: { default: registryProxy },
163+
supportedArchitectures: { os: ['darwin'], cpu: ['arm64'] },
164+
}))
165+
} finally {
166+
server.close()
167+
}
168+
169+
expect(deepRequireCwd(['@pnpm.e2e/has-many-optional-deps', '@pnpm.e2e/darwin-arm64', './package.json']).version).toBe('1.0.0')
170+
171+
// The platforms of the other binaries are inferred from their names, so
172+
// they are skipped without even downloading their tarballs.
173+
project.storeHasNot('@pnpm.e2e/linux-x64', '1.0.0')
174+
project.storeHasNot('@pnpm.e2e/windows-x64', '1.0.0')
175+
expect(fs.existsSync(path.resolve('node_modules/.pnpm/@pnpm.e2e+linux-x64@1.0.0'))).toBeFalsy()
176+
expect(fs.existsSync(path.resolve('node_modules/.pnpm/@pnpm.e2e+windows-x64@1.0.0'))).toBeFalsy()
177+
178+
const modulesInfo = readYamlFileSync<{ skipped: string[] }>(path.join('node_modules', '.modules.yaml'))
179+
expect(modulesInfo.skipped).toContain('@pnpm.e2e/linux-x64@1.0.0')
180+
expect(modulesInfo.skipped).toContain('@pnpm.e2e/windows-x64@1.0.0')
181+
})
182+
183+
// Simulates a registry that strips os/cpu/libc from packument version objects
184+
// (some registry proxies do this), forwarding everything else to the registry mock.
185+
function createMetadataStrippingRegistryProxy (): http.Server {
186+
return http.createServer((req, res) => {
187+
(async () => {
188+
const upstream = await fetch(`http://localhost:${REGISTRY_MOCK_PORT}${req.url}`, {
189+
headers: { accept: req.headers.accept ?? '*/*' },
190+
})
191+
const contentType = upstream.headers.get('content-type') ?? ''
192+
if (contentType.includes('json')) {
193+
const doc = await upstream.json() as { versions?: Record<string, Record<string, unknown>> }
194+
for (const versionMeta of Object.values(doc.versions ?? {})) {
195+
delete versionMeta.os
196+
delete versionMeta.cpu
197+
delete versionMeta.libc
198+
}
199+
res.writeHead(upstream.status, { 'content-type': 'application/json' })
200+
res.end(JSON.stringify(doc))
201+
} else {
202+
res.writeHead(upstream.status, { 'content-type': contentType })
203+
res.end(Buffer.from(await upstream.arrayBuffer()))
204+
}
205+
})().catch((err) => {
206+
res.writeHead(500)
207+
res.end(String(err))
208+
})
209+
})
210+
}
211+
143212
test('skip optional dependency that does not support the current Node version', async () => {
144213
const project = prepareEmpty()
145214
const reporter = jest.fn()
@@ -615,6 +684,43 @@ describe('supported architectures', () => {
615684
})
616685
expect(deepRequireCwd(['@pnpm.e2e/has-many-optional-deps', '@pnpm.e2e/linux-x64', './package.json']).version).toBe('1.0.0')
617686
})
687+
// Test case for https://github.com/pnpm/pnpm/issues/11702
688+
test.each(['isolated', 'hoisted'])('skip optional dependencies that do not support the target architecture when their lockfile entries have no platform fields (nodeLinker=%s)', async (nodeLinker) => {
689+
const project = prepareEmpty()
690+
const supportedArchitectures = { os: ['darwin'], cpu: ['arm64'] }
691+
692+
const { updatedManifest: manifest } = await addDependenciesToPackage({}, ['@pnpm.e2e/has-many-optional-deps@1.0.0'], {
693+
...testDefaults({ nodeLinker }),
694+
supportedArchitectures,
695+
})
696+
697+
// Simulate a lockfile resolved from registry metadata that lacks
698+
// the platform fields.
699+
const lockfile = project.readLockfile()
700+
for (const pkgSnapshot of Object.values(lockfile.packages)) {
701+
delete pkgSnapshot.os
702+
delete pkgSnapshot.cpu
703+
delete pkgSnapshot.libc
704+
}
705+
writeYamlFileSync(WANTED_LOCKFILE, lockfile, { lineWidth: 1000 })
706+
rimrafSync('node_modules')
707+
708+
await install(manifest, {
709+
...testDefaults({ nodeLinker }),
710+
frozenLockfile: true,
711+
supportedArchitectures,
712+
})
713+
714+
if (nodeLinker === 'hoisted') {
715+
expect(fs.existsSync(path.resolve('node_modules/@pnpm.e2e/darwin-arm64'))).toBeTruthy()
716+
expect(fs.existsSync(path.resolve('node_modules/@pnpm.e2e/darwin-x64'))).toBeFalsy()
717+
expect(fs.existsSync(path.resolve('node_modules/@pnpm.e2e/linux-x64'))).toBeFalsy()
718+
} else {
719+
expect(deepRequireCwd(['@pnpm.e2e/has-many-optional-deps', '@pnpm.e2e/darwin-arm64', './package.json']).version).toBe('1.0.0')
720+
expect(fs.existsSync(path.resolve('node_modules/.pnpm/@pnpm.e2e+darwin-x64@1.0.0'))).toBeFalsy()
721+
expect(fs.existsSync(path.resolve('node_modules/.pnpm/@pnpm.e2e+linux-x64@1.0.0'))).toBeFalsy()
722+
}
723+
})
618724
test('remove optional dependencies that are not used', async () => {
619725
prepareEmpty()
620726
const opts = testDefaults({ modulesCacheMaxAge: 0 })

0 commit comments

Comments
 (0)