Skip to content

Replace relative URLs to absolute URL when freezing the lockfile#241

Merged
ryanking13 merged 11 commits into
pyodide:mainfrom
ryanking13:store-full-url-freeze
Jul 5, 2025
Merged

Replace relative URLs to absolute URL when freezing the lockfile#241
ryanking13 merged 11 commits into
pyodide:mainfrom
ryanking13:store-full-url-freeze

Conversation

@ryanking13

@ryanking13 ryanking13 commented Jul 5, 2025

Copy link
Copy Markdown
Member

This updates micropip.freeze to replace relative URLs (or just file names) to absolute URL.

This was discussed in pyodide/pyodide#5652 (comment) but I forgot to fix it before releasing 0.28.

This also fixes pyodide/pyodide#5736

  • changelog

@hoodmane hoodmane left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@hoodmane

hoodmane commented Jul 5, 2025

Copy link
Copy Markdown
Member

Now we should merge this, release micropip, release the packages, update the package set, and backport the package set update.

@ryanking13 ryanking13 merged commit 0273f37 into pyodide:main Jul 5, 2025
5 checks passed
@ryanking13

Copy link
Copy Markdown
Member Author

Yes.

@ryanking13 ryanking13 deleted the store-full-url-freeze branch July 5, 2025 12:27
@alumni

alumni commented Jul 8, 2025

Copy link
Copy Markdown

The file names are incorrect on NodeJS: /path/file instead of /path/to/file (missing the last path segment of the folder).

@ryanking13

Copy link
Copy Markdown
Member Author

@alumni Oops... thanks.

@ryanking13

Copy link
Copy Markdown
Member Author

@alumni I cannot reproduce the behavior. Could you provide some reproducer?

@ryanking13

Copy link
Copy Markdown
Member Author

I found some other issues with node anyways...

@alumni

alumni commented Jul 12, 2025

Copy link
Copy Markdown

@ryanking13 I think I figured it out when it happens: if packageCacheDir does not end with / (e.g. if it's set to process.cwd() or to the output of path.join) - folders don't end by default with / (unless the users explicitly add it - which they shouldn't if they want Windows compatibility).

Assume the folder .cache/pyodide already exists and .cache/pyodide-lock.json is a copy of the lockfile that Pyodide 0.28.0 ships with.

const { mkdir, writeFile } = require('node:fs/promises');
const { join } = require('node:path');
const { loadPyodide } = require('pyodide');

const packageCacheDir = join(process.cwd(), '.cache', 'pyodide');
const lockFileURL = join(process.cwd(), '.cache', 'pyodide-lock.json');

// file structure setup:
await mkdir(packageCacheDir, { mode: 0o755, recursive: true });
const lockFile = (await import('pyodide/pyodide-lock.json', { with: { type: 'json' } })).default;
await writeFile(lockFileURL, JSON.stringify(lockFile, null, 4), { encoding: 'utf-8', flush: true });

const pyodide = await loadPyodide({ lockFileURL, packageCacheDir });

// install micropip 0.10.1 from PyPi
await pyodide.loadPackage('https://files.pythonhosted.org/packages/6a/79/3dda562c22b4978493adda6a7810f43b513b46cef82cbfd2aef48c3e29e0/micropip-0.10.1-py3-none-any.whl')

const micropip = pyodide.pyimport('micropip');
await micropip.install('simplejson');
// Console output:
//     Loading simplejson
//     Didn't find package simplejson-3.20.1-cp313-cp313-pyodide_2025_0_wasm32.whl locally, attempting to load from https://cdn.jsdelivr.net/pyodide/v0.28.0/full/
//     Package simplejson-3.20.1-cp313-cp313-pyodide_2025_0_wasm32.whl loaded from https://cdn.jsdelivr.net/pyodide/v0.28.0/full/, caching the wheel in node_modules for future use.
//     Loaded simplejson
//
// File <cwd>/.cache/pyodide/simplejson-3.20.1-cp313-cp313-pyodide_2025_0_wasm32.whl was created

JSON.parse(micropip.freeze()).packages.simplejson;
// output:
{
  depends: [],
  file_name: '<cwd>/.cache/simplejson-3.20.1-cp313-cp313-pyodide_2025_0_wasm32.whl',
  imports: [ 'simplejson' ],
  install_dir: 'site',
  name: 'simplejson',
  package_type: 'package',
  sha256: 'c5884ab81866a337e4d797f4f269a9ad26513097d2993d09bc90e80dfa1ccbbe',
  unvendored_tests: true,
  version: '3.20.1'
}

Actual file: <cwd>/.cache/pyodide/simplejson-3.20.1-cp313-cp313-pyodide_2025_0_wasm32.whl.
Entry in pyodide-lock.json: <cwd>/.cache/simplejson-3.20.1-cp313-cp313-pyodide_2025_0_wasm32.whl.

Probably path functions could be used if Pyodide is running in NodeJS, instead of urljoin. Not sure how it would work on Windows. But if something isn't working for me, I can find workarounds in JS, now I'm slowly starting to understand how things work - I was actually planning to edit the lockfile to remove the absolute path.

@ryanking13

Copy link
Copy Markdown
Member Author

Oh, okay, thanks for the detailed reproducer.

So the problem is that urljoin behaves differently when the base path includes trailing slash or not.

>>> urljoin("file:///a/b/", "hello.py")
'file:///a/b/hello.py'
>>> urljoin("file:///a/b", "hello.py")
'file:///a/hello.py'

@ryanking13

Copy link
Copy Markdown
Member Author

pyodide/pyodide#5763 will do the fix. I am not sure whether it will be fully compatible with the Windows path. But as far as I remember, node handles mixed slashes quite well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0.28.0: micropip lockFileURL failing when provided

3 participants