chore(deps): pin chromadb to >=1.0,<2 by jccidc · Pull Request #426 · MemPalace/mempalace

jccidc · 2026-04-09T19:50:31Z

What does this PR do?

Fixes installs for users whose palaces were created with chromadb 1.x (see #445). The current constraint chromadb>=0.5.0,<0.7 forces installs onto the 0.5/0.6 line, which uses an older on-disk format. Palaces built on chromadb 1.x can't be opened by 0.6 — get_collection fails with KeyError: '_type' when reading collection config JSON, and the CLI surfaces it as a misleading "No palace found" message.

Pin to the 1.x line so:

editable installs keep working with palaces built on 1.x
new installs land on the same major as the current chromadb release (1.5.x)
we have a single supported major instead of straddling two on-disk formats

Known break and migration note

This change does break users with existing chromadb 0.5/0.6 palaces. chromadb 1.x does not auto-migrate the 0.6 on-disk format — the chromadb 1.0 migration guide covers a dedicated process. For mempalace specifically, the simplest recovery is to back up ~/.mempalace/palace/ and re-mine the source files into a fresh 1.x palace.

A proper in-tool migration command (mempalace migrate-chroma or similar) is a separate concern and belongs in a follow-up PR. This PR is intentionally scoped to the dependency pin.

API compatibility audit

All chromadb APIs used by mempalace are part of the stable core interface and work identically on 0.5 → 1.x:

API	Callers
`chromadb.PersistentClient(path=...)`	`palace.py`, `mcp_server.py`, `layers.py`, `searcher.py`, `cli.py`
`client.get_collection(name)`	`palace.py`, `mcp_server.py`, `layers.py`, `searcher.py`, `cli.py`
`client.create_collection(name)`	`palace.py`
`client.get_or_create_collection(name)`	`mcp_server.py`
`collection.get(where=..., limit=...)`	`palace.py`, `miner.py`, `mcp_server.py`
`collection.count()`	`mcp_server.py`, `cli.py`, `layers.py`
`collection.query(...)`	`searcher.py`, `layers.py`
`collection.add(...)`	`miner.py`, `convo_miner.py`
`collection.delete(...)`	`mcp_server.py`

No code changes needed.

How to test

Install the branch in a fresh venv: pip install -e .
Confirm chromadb 1.x resolves: python -c "import chromadb; print(chromadb.__version__)" → 1.5.x
Run the test suite: python -m pytest tests/ -v (passes locally against chromadb 1.5.7)
If you have an existing 1.x palace, verify mempalace status opens it without a KeyError: '_type'.

Replaces

This replaces #365, which accumulated stale diffs from being rebased off an older main. This PR is a single clean commit off current main.

Checklist

Single-line dependency change, no new dependencies
All chromadb API calls audited for 1.x compatibility
Conventional commit title
Known migration impact documented
Tests pass in CI (ran locally — need maintainer CI run)

The previous constraint of chromadb>=0.5.0,<0.7 forces installs onto the 0.5/0.6 line, which uses an older on-disk format. Palaces created with chromadb 1.x (the current major) can't be opened by 0.6 — get_collection fails with KeyError: '_type' when reading collection config json. Pin to the 1.x line so: - editable installs keep working with palaces built on 1.x - new installs land on the same major as the current chromadb release (1.5.x) - we have a single supported major instead of straddling two on-disk formats Note: existing 0.5/0.6 palaces are auto-migrated by chromadb 1.x on first open. The migration is one-way — once migrated, the palace cannot be reopened by chromadb 0.5/0.6. All chromadb APIs used by mempalace (PersistentClient, get_collection, create_collection, get_or_create_collection, collection.get, collection.count, collection.query, collection.add, collection.delete) are stable across the 0.5 → 1.x boundary. No code changes required.

iSapozhnik · 2026-04-09T23:08:56Z

Will this also solve this problem?

web3guru888

This goes in the right direction — issue #445 reports that users who installed ChromaDB 1.x (which pip resolves by default now) can't open palaces created under the current >=0.5.0,<0.7 pin, because ChromaDB 1.0 changed its storage format.

But bumping straight to >=1.0,<2 without a migration path will break anyone running 0.5.x or 0.6.x who upgrades mempalace:

Existing palaces won't open — ChromaDB 1.0 can't read 0.5/0.6 SQLite storage directly
No mempalace repair guidance — the CLI repair command re-indexes vectors but doesn't handle the underlying Chroma storage format migration
The current test suite runs against 0.5/0.6 — has anyone verified all tests pass against chromadb 1.x?

Suggestion: consider >=0.5.0,<2 (widen the range rather than bump the floor), and add a note in the upgrade docs that users on Chroma 1.x should run mempalace repair after upgrading. The repair command already does a full re-index which should handle the format gap — but it needs to be tested against the 0.5→1.0 transition.

@iSapozhnik re your question — yes, this should solve #445, but only if mempalace's own code is compatible with the ChromaDB 1.x API. The pin change alone isn't enough if there are breaking API changes upstream.

jccidc · 2026-04-10T02:22:43Z

Thanks @web3guru888 — you're right that my original PR body overclaimed on migration. I said "auto-migrates on first open" but after checking the chromadb 1.0 release notes, that's not actually the case — there's a migration guide, not an automatic migration. I'll strike that line.

On the constraint choice: widening to >=0.5.0,<2 doesn't actually help new installs — pip will resolve to the latest (1.5.x) regardless, and anyone with an existing 0.5/0.6 palace hits the same break. The only user who benefits from widening is one who already has chromadb 0.6 installed from a prior version and doesn't force pip to upgrade. That's a narrow window.

The core problem is that there's no way to both (a) fix #445 for 1.x users and (b) avoid breaking 0.6 users without a code-level compatibility layer or migration command. A constraint change alone can only pick one side.

Given that:

v3.1.0 ChromaDB pin breaks existing palace created with Chroma 1.x #445 is an actively broken user today
mempalace's entire API surface (PersistentClient, get_collection, count, get, query, add, delete) is stable across the 0.5 → 1.x boundary — no code changes needed for the jump
chromadb 1.x is the current major (1.5.7) and what new installs get by default anyway

I think the right scope for this PR is:

Keep the >=1.0,<2 pin so new installs and editable installs from source both land on 1.x
Explicitly document the break in the release notes for the next version
Recommend users re-mine their source files rather than try to migrate chromadb's on-disk format

A proper migration story (detect old format, run mempalace repair against a chromadb 0.6 read-only client, re-mine into 1.x) is a separate PR. I can open a follow-up issue to track it if that'd be useful.

On your other points:

Test coverage: good catch, CI doesn't currently test against chromadb 1.x. I can add a matrix entry in a separate PR, but the existing test suite (tests/test_mcp_server.py, tests/test_miner.py, etc.) does pass against 1.5.7 locally — I ran them on the v2 branch before pushing.
mempalace repair: I looked at cli.py:160 — it calls client.get_collection() first, which hits the same KeyError: '_type' on 0.5/0.6 → 1.x transitions. So repair in its current form can't bridge the format gap. That's the migration-command gap I mentioned above.

Let me know if you'd rather I close this and wait for a proper migration PR, or merge this as-is with a release-notes callout.

director-LAC · 2026-04-10T03:44:44Z

Adding weight to this: on Python 3.14, chromadb 0.6.x doesn't just cause data incompatibility, it fails to import entirely due to pydantic.v1 incompatibility (#487). The <0.7 pin has no viable target on 3.14. Widening to >=1.0 is the only path forward for current Python versions.

web3guru888 · 2026-04-10T04:03:35Z

You've thought this through carefully, and I agree with the analysis. The narrow-window argument is right: widening to >=0.5,<2 only protects users who happen to have chromadb 0.6 already installed and aren't running pip install --upgrade, which is a shrinking population.

Recommendation: merge as >=1.0,<2 with clear release notes. The case for >=1.0 is:

New installs get 1.x anyway — this formalizes what's already true in practice
The 0.5/0.6 → 1.x break is a data format issue, not an API issue — existing users need to re-mine regardless of what the constraint says
mempalace repair being broken against 1.x is a separate bug that should be tracked and fixed independently

The migration command gap is worth a dedicated issue rather than blocking this PR. A proper mempalace migrate --from 0.6 that spins up a temporary chromadb 0.5/0.6 reader client and re-ingests into 1.x would be the complete fix, but it's a multi-week effort.

One thing to add to your release notes callout: recommend users run mempalace status before upgrading to verify their palace is readable, and keep their palace directory backed up (even just cp -r ~/.mempalace/palace ~/.mempalace/palace.backup). Low-tech but effective.

igorls · 2026-05-08T11:01:32Z

Hi, thanks for the contribution.

This PR has merge conflicts with develop, and the branch has not been updated in over 7 days, which puts it before our most recent release. The conflicts are likely against work that landed in that release.

Could you rebase onto develop so we can take another look?

If this change is no longer relevant, feel free to close the PR.

(This message is part of a periodic backlog pass, sent to all open PRs that match this state.)

chromadb 1.5.4..1.5.9 (latest) segfault in chromadb_rust_bindings.abi3.so on macOS 26 / ARM64 the moment a populated PersistentClient collection is opened. The crash is reproducible with a 5-line script and disables mempalace status / migrate / mine / MCP server lifecycle on that platform. Bisect on a 547k-record real palace (mempalace 3.3.4): | chromadb | result | |----------|-----------------| | 1.5.9 | SIGSEGV | | 1.5.8 | SIGSEGV | | 1.5.3 | works | | 1.5.0 | works | | 1.4.0 | works | | 1.3.0 | works | Regression boundary: 1.5.3 -> 1.5.4. Symptom: many tokio-rt-worker threads in a recursive enum walk (HNSW/segment iteration, by shape), one walker dereferences a poisoned pointer inside pthread_mutex_unlock of a node whose backing memory has been freed. Classic use-after-free under concurrent walkers + compaction/GC. Tracking: - MemPalace#1355 (mirror, OPEN) - chroma-core/chroma#6852 (upstream, OPEN) - MemPalace#1340, MemPalace#1274, MemPalace#1329 (related symptoms) This is a temporary cap. Lift once upstream chroma ships a fix and re-pin to >=<fixed-version>,<2. Lower bound bumped to >=1.5.0 (matches what mempalace was actually built and tested against in this 1.5.x line); maintainers may prefer >=1.0,<1.5.4 if older 1.x is also tested. PR MemPalace#426 is in flight on the lower-bound side. Tested locally on macOS 26.3.1 / Apple Silicon: chromadb 1.5.3 opens the existing palace cleanly (count=547,377, peek returns IDs); chromadb 1.5.8 (current install) crashes immediately on the same operation.

jccidc mentioned this pull request Apr 9, 2026

chore(deps): pin chromadb to >=1.0,<2 #365

Closed

3 tasks

web3guru888 reviewed Apr 10, 2026

View reviewed changes

This was referenced Apr 10, 2026

docs: add Python 3.13 and 3.14 classifiers #302

Open

fix: improve Windows test compatibility #277

Open

chromadb<0.7 pin breaks MCP server on Python 3.14 #487

Open

web3guru888 mentioned this pull request Apr 10, 2026

palace data gone after upgrade to v3.1.0 #469

Closed

This was referenced Apr 10, 2026

v3.1.0 ChromaDB pin breaks existing palace created with Chroma 1.x #445

Open

chore: improve agent readiness score — AGENTS.md, dependabot, CODEOWNERS #497

Merged

Readme chromadb version differs from actual #531

Open

bensig changed the base branch from main to develop April 11, 2026 22:22

bensig requested review from bensig, igorls and milla-jovovich as code owners April 11, 2026 22:22

igorls added the area/install pip/uv/pipx/plugin install and packaging label Apr 14, 2026

This was referenced Apr 18, 2026

bug: fresh pip install mempalace silently broken on chromadb 1.5.x — writes queued but never flush, search returns ghosts #1006

Closed

fix: pin chromadb to <1.0 to avoid 1.5.x queue-stall regression (#1006) #1008

Closed

MohamedAbdallah-14 mentioned this pull request May 6, 2026

fix(deps): cap chromadb<1.5.4 — Rust bindings UAF on macOS 26 ARM64 #1376

Closed

igorls added the needs-rebase PR has merge conflicts with develop and needs rebase label May 8, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(deps): pin chromadb to >=1.0,<2#426

chore(deps): pin chromadb to >=1.0,<2#426
jccidc wants to merge 1 commit into
MemPalace:developfrom
jccidc:fix/chromadb-pin-v2

jccidc commented Apr 9, 2026 •

edited

Loading

Uh oh!

iSapozhnik commented Apr 9, 2026

Uh oh!

web3guru888 left a comment

Uh oh!

jccidc commented Apr 10, 2026

Uh oh!

director-LAC commented Apr 10, 2026

Uh oh!

web3guru888 commented Apr 10, 2026

Uh oh!

igorls commented May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

jccidc commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Known break and migration note

API compatibility audit

How to test

Replaces

Related

Checklist

Uh oh!

iSapozhnik commented Apr 9, 2026

Uh oh!

web3guru888 left a comment

Choose a reason for hiding this comment

Uh oh!

jccidc commented Apr 10, 2026

Uh oh!

director-LAC commented Apr 10, 2026

Uh oh!

web3guru888 commented Apr 10, 2026

Uh oh!

igorls commented May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jccidc commented Apr 9, 2026 •

edited

Loading