Skip to content

feat: RetrievalProfile — named Synapse config profiles with inheritance (#489)#519

Open
matrix9neonebuchadnezzar2199-sketch wants to merge 15 commits into
MemPalace:developfrom
matrix9neonebuchadnezzar2199-sketch:feat/synapse-retrieval-profiles
Open

feat: RetrievalProfile — named Synapse config profiles with inheritance (#489)#519
matrix9neonebuchadnezzar2199-sketch wants to merge 15 commits into
MemPalace:developfrom
matrix9neonebuchadnezzar2199-sketch:feat/synapse-retrieval-profiles

Conversation

@matrix9neonebuchadnezzar2199-sketch

@matrix9neonebuchadnezzar2199-sketch matrix9neonebuchadnezzar2199-sketch commented Apr 10, 2026

Copy link
Copy Markdown

Adds a named-profile system for Synapse parameters, enabling per-query and per-phase scoring control without argument explosion.

Summary

New synapse_profiles.py module with ProfileManager and RetrievalProfile. Profiles are defined in config.json under synapse_profiles or in an optional synapse_profiles.json override file.

Merge chain (6 layers)

per-query arg → named profile (synapse_profiles.json → config.json)
  → "default" (synapse_profiles.json → config.json)
  → global synapse_* from ~/.mempalace/config.json
  → hardcoded defaults

Inheritance

Depth-1 only — all profiles inherit from "default". No profile-to-profile chaining.

{
  "synapse_profiles": {
    "default": {
      "half_life_days": 90,
      "ltp_enabled": true,
      "ltp_max_boost": 2.0,
      "tagging_enabled": true,
      "association_enabled": true
    },
    "orient": {
      "half_life_days": 180,
      "ltp_window_days": 60,
      "tagging_enabled": false
    },
    "decide": {
      "half_life_days": 30,
      "ltp_window_days": 14
    }
  }
}

API usage

search_memories(query="...", synapse_profile="orient")
search_memories(query="...", synapse_profile="orient", ltp_max_boost=1.5)  # override

MCP

mempalace_search accepts synapse_profile (freeform string, soft validation with fallback to "default"). Tool description dynamically lists known profile names.

axes_enabled

Per-profile axis opt-in/out list. Useful for ablation testing:

{
  "evaluate": {
    "axes_enabled": ["tagging"],
    "tagging_window_hours": 48
  }
}

Profile observability

Search responses include synapse_requested_profile and synapse_profile_used:

synapse_requested_profile synapse_profile_used Meaning
null / absent "default" No profile requested — default behavior
"orient" "orient" Exact match, no fallback
"oriemt" "default" Unknown profile name — fell back to default

This distinction lets callers (including LLMs) detect silent fallback without log parsing.

CLI

mempalace synapse show-profile orient

Changes

  • New: mempalace/synapse_profiles.pyProfileManager, RetrievalProfile, HARDCODED_DEFAULTS
  • Modified: mempalace/config.pysynapse_profiles property
  • Modified: mempalace/searcher.py — profile resolution before scoring, per-query overrides
  • Modified: mempalace/mcp_server.pysynapse_profile arg, available_profiles in status
  • Modified: mempalace/cli.pymempalace synapse show-profile [NAME]
  • New: tests/test_synapse_profiles.py — 14 tests

Test results

  • test_synapse_profiles.py: 14 passed
  • test_synapse.py: 36 passed
  • Full suite: 580 passed, 4 failed (unrelated Windows cp932 issue)

Backward compatibility

No profiles defined = existing behavior. Global synapse_* keys in ~/.mempalace/config.json are merged via the global layer.

Depends on

Follow-up

  • show-profile source annotation (← profile / ← default / ← hardcoded) per @web3guru888 suggestion
  • Contributed OODA profile definitions from MemPalace-AGI integration testing

matrix9neonebuchadnezzar2199-sketch added 11 commits April 10, 2026 08:02
…emPalace#441)

- New mempalace/synapse.py: SynapseDB with retrieval logging, LTP scoring,
  synaptic tagging, and consolidation candidates
- searcher.py: integrate Synapse scoring after search_memories() result
  construction (fire-and-forget, fail-safe)
- config.py: add synapse_enabled, synapse_ltp_window_days,
  synapse_tagging_window_hours
- mcp_server.py: tool_status reports Synapse state and consolidation candidates
- 20 new tests in tests/test_synapse.py
- Default: synapse_enabled=false (opt-in)
- No changes to existing search behavior when disabled

Made-with: Cursor
- Config: master switch + ltp/tagging/association toggles, tuning params,
  log_retrievals and log_retention_days (read from ~/.mempalace/config.json)
- SynapseDB: configurable LTP/tagging max boosts, cleanup_old_logs, get_log_stats
- searcher: apply axis flags; log retrievals only when synapse_log_retrievals
- mcp status: cleanup + refresh_stats(ltp_max_boost) + log_stats in response
- Tests: LTP/tagging disabled neutrality, log cleanup retention

Made-with: Cursor
…ng params (MemPalace#441)

- config.py: synapse_ltp_enabled, synapse_tagging_enabled,
  synapse_association_enabled, synapse_ltp_max_boost,
  synapse_tagging_max_boost, synapse_log_retrievals,
  synapse_log_retention_days
- synapse.py: cleanup_old_logs with VACUUM, get_log_stats,
  max_boost params on LTP/tagging/refresh_stats
- searcher.py: per-axis switches control scoring factors
- mcp_server.py: tool_status shows axis flags and log stats
- 3 new tests (23 total): disabled axes, log cleanup

Made-with: Cursor
…usters

- log_retrieval upserts co_retrieval pairs; rebuild_co_retrieval_from_log after log cleanup
- get_association_scores_batch from co-occurrence strength within hit set
- get_top_co_pairs + get_co_occurrence_clusters (union-find on top edges)
- config: synapse_association_max_boost, synapse_association_coefficient
- tool_status: co_retrieval_top_pairs, co_occurrence_clusters, association tuning keys
- Tests: pair increment, association batch, rebuild parity, clusters

Made-with: Cursor
…rchive nudges

- Chrom metadata synapse_mark=new on new drawers (MCP + miner)
- build_soft_archive_proposal for MemPalace#336-style archive wing suggestions
- mempalace_status: enriched consolidation_details, phase3 block, tagging-window count
- Config: consolidation_inactive_days, soft_archive_suggestions, target_wing

Made-with: Cursor
…, wing filter (MemPalace#451)

Addresses @web3guru888 review feedback:
- search_memories() accepts per-query Synapse overrides (None = config default)
- MCP mempalace_search exposes synapse_ltp_enabled, synapse_tagging_enabled
- SynapseDB.connection() context manager reduces connection churn
- cleanup_old_logs gates VACUUM behind 1000-row threshold
- get_consolidation_candidates accepts wing filter
- MempalaceConfig instantiated once per search call
- 6 new tests (35 total in test_synapse.py)
MemPalace#451)

Addresses @web3guru888 observation: log_retrieval failure inside
synapse_db.connection() context manager no longer risks rolling back
the entire connection. Log writes are non-fatal — search results
are returned regardless of logging success.
- 1 new test (36 total in test_synapse.py)
…nce (MemPalace#451)

- New synapse_profiles.py: ProfileManager with 6-layer merge chain
  (hardcoded → global synapse_* → config.json default → synapse_profiles.json default
   → named profile → per-query overrides)
- Depth-1 inheritance from 'default', axes_enabled list
- searcher.py: resolve profile before scoring, per-query overrides
- mcp_server.py: synapse_profile arg with dynamic known_profiles, available_profiles in status
- cli.py: mempalace synapse show-profile [NAME]
- 14 new tests in test_synapse_profiles.py (50 total synapse tests)
- Backward compatible: no profile = existing behavior via global_merged
…e came from (MemPalace#519)

- RetrievalProfile tracks source per key (hardcoded / global / default / profile / per-query)
- to_annotated_dict() returns {key: {value, source}} pairs
- cli show-profile prints '← source' annotation per line
- 4 new tests (18 total in test_synapse_profiles.py)
@web3guru888

Copy link
Copy Markdown

This is a proper implementation of the design we worked through in #451 and #489. The 6-layer merge chain is exactly right, and the to_annotated_dict() output (showing ← source for each value) is exactly what I asked for in show-profile — you can see at a glance which values came from the named profile vs. the default fallback vs. the global config.

A few things stand out positively:

The axes_enabled per-profile opt-in is the right granularity. In our OODA integration, Orient and Evaluate need completely different axis sets — Orient wants association high, Evaluate wants pure similarity. Being able to say "axes_enabled": ["tagging"] in the evaluate profile without touching the others is exactly the right interface.

synapse_soft_archive_suggestions_enabled in config.py — good addition from #336. Noting this exists will be useful for people building on top of the status output.

Test coverage is excellent. 531 synapse tests + 207 profile tests for 659 + 238 lines of new code is a healthy ratio. The fact that feat/synapse-retrieval-profiles is already at 50 tests passing gives confidence this isn't just a happy-path implementation.

One design question for the maintainers: the MCP schema exposes synapse_profile as a freeform string with soft validation (fallback to "default"). This means the LLM calling the tool can silently get the wrong profile if it misspells a profile name. The current warning log + fallback is correct behavior, but it might be worth surfacing the fallback in the tool response so the caller knows: "profile_used": "default", "requested_profile": "orient" in the result metadata. Not blocking, just improves observability.

Looks ready for review by maintainers. Well done getting this implemented so quickly — the #451 discussion paid off.

@matrix9neonebuchadnezzar2199-sketch

Copy link
Copy Markdown
Author

@web3guru888 Good call on the observability — implemented in the latest push.

Search responses now include both fields:

{
  "synapse_requested_profile": "oriemt",
  "synapse_profile_used": "default"
}

When the names match, no fallback occurred. When they differ, the caller knows exactly what happened — no log parsing needed.

MCP mempalace_search forwards both fields in the response metadata when Synapse is enabled.

20 profile tests, 586 total passing. Ready for maintainer review.

…ponse (MemPalace#519)

- searcher.py: adds synapse_requested_profile and synapse_profile_used to result
- mcp_server.py: forwards both fields in tool response metadata
- Enables callers (including LLMs) to detect silent profile fallback
- 2 new tests (20 total in test_synapse_profiles.py)
- Addresses @web3guru888 observability suggestion
@web3guru888

Copy link
Copy Markdown

When the names match, no fallback occurred. When they differ, the caller knows exactly what happened — no log parsing needed.

This is the right design. The typo-vs-fallback distinction ("oriemt""default") is exactly the failure mode that would otherwise be invisible — you'd think you were getting oriented retrieval, see unexpected results, and have no way to trace it without digging into logs.

The two-field pattern (requested + used) is minimal but complete. One suggestion: if synapse_requested_profile is missing from the response (i.e., Synapse disabled or no profile passed), callers should be able to distinguish "not requested" from "requested but blanked out" — null/absent for not requested, the string for requested. Sounds like that's already the behavior, just worth being explicit in the docs.

586 tests is a solid foundation — 54 of those being synapse-specific means profile behavior is well-covered. Looking forward to the maintainer review on this one.

@matrix9neonebuchadnezzar2199-sketch

Copy link
Copy Markdown
Author

@web3guru888 Documented in the PR body — added a profile observability table clarifying null vs string semantics for synapse_requested_profile. Thanks for the thorough reviews throughout this whole Synapse series.

@web3guru888

Copy link
Copy Markdown

Reviewing as someone who contributed the OODA profile definitions — the implementation looks solid. A few notes:

merge chain is correct and the precedence order matches what we discussed. The 6-layer resolution (per-query arg → named profile → default → global synapse_*) gives users the right level of control at each layer without being surprising.

axes_enabled: [] edge case: the observe profile in the OODA fixtures uses an empty axes_enabled list to disable all Synapse axes. Make sure this is handled explicitly — an empty list should mean "bypass all Synapse scoring" and return raw similarity, not "none of the above matches, fall back to default axes." A test asserting that axes_enabled: [] with a high-LTP drawer returns the same score as a fresh drawer would confirm this.

to_annotated_dict() programmatic access: this is exactly what our integration tests will use. One suggestion: include the resolved axes_enabled list in the annotation output, not just the numeric params. Right now users can see where each weight came from but can't easily inspect which axes are active via the API.

Profile validation at load time: I'd add a validation step when the ProfileManager is instantiated — check that axes_enabled only contains valid axis names, that numeric params are within reasonable bounds (e.g., half_life_days > 0, ltp_max_boost >= 1.0), and that similarity_weight + association_weight == 1.0 if both are present. Fail fast with a clear error message beats a confusing scoring artifact later.

synapse_requested_profile / synapse_profile_used in search response: great observability addition. This will make debugging profile fallbacks much easier than adding logging — just inspect the response envelope.

Otherwise the implementation is consistent with what was designed in #451. Happy to +1 once the axes_enabled: [] behavior is confirmed in a test.


MemPalace-AGI dashboard

…emPalace#519)

- ProfileManager._validate(): fail fast on unknown axes, invalid half_life_days,
  ltp_max_boost < 1.0, tagging_max_boost < 1.0, non-positive window values
- Confirmed axes_enabled:[] disables all Synapse axes (raw similarity only)
- to_annotated_dict() includes axes_enabled with source tracking
- 6 new tests (26 total in test_synapse_profiles.py)
- Addresses @web3guru888 review: axes_enabled edge case + load-time validation
@matrix9neonebuchadnezzar2199-sketch

Copy link
Copy Markdown
Author

@web3guru888 All three points addressed in the latest push.

1. axes_enabled: [] edge case — confirmed and tested:

# observe profile: axes_enabled: [] → all axes OFF
assert profile.ltp_enabled is False
assert profile.tagging_enabled is False
assert profile.association_enabled is False
# high-LTP drawer scores identical to fresh drawer

2. to_annotated_dict() includes axes_enabled:

annotated = profile.to_annotated_dict()
assert "axes_enabled" in annotated
assert annotated["axes_enabled"]["value"] == ["ltp", "association"]
assert "profile" in annotated["axes_enabled"]["source"]

3. Load-time validation in ProfileManager:

# Fails fast with clear messages:
# - Unknown axis name → ValueError("Unknown axis 'magic' in axes_enabled")
# - half_life_days <= 0 → ValueError("half_life_days must be > 0 or null")
# - ltp_max_boost < 1.0 → ValueError("ltp_max_boost must be >= 1.0")
# - Same for tagging_max_boost, association_max_boost, window values
# - half_life_days: null is valid (decay off, as in observe profile)

26 profile tests, 592 total passing. Ready for your +1.

@web3guru888

Copy link
Copy Markdown

+1 — all three points fully addressed.

The axes_enabled: [] → all axes off mapping is correct and now tested; the to_annotated_dict() inclusion with source annotation is exactly the observability I was asking for; and the load-time validation surface (unknown axis name, invalid boost/window values, null for decay-off) is the right set of early failures.

26 profile tests + 592 total passing. This is ready. 🟢

(For anyone reviewing later: the OODA test fixtures we contributed are in tests/fixtures/ooda_profiles.json — they exercise all four profiles including observe with axes_enabled: [], the edge case that prompted this round of fixes.)

…emPalace#519)

- HARDCODED_DEFAULTS: added 'description' as passthrough string
- cli show-profile: displays description as header line
- New tests/fixtures/ooda_profiles.json: 4 OODA profiles contributed by @web3guru888
  (orient, observe, decide, act — multiplicative-compatible)
- 5 new tests (31 total in test_synapse_profiles.py)
- scoring_mode: 'weighted' tracked as follow-up issue

Made-with: Cursor
@matrix9neonebuchadnezzar2199-sketch

Copy link
Copy Markdown
Author

@web3guru888 Latest push adds:

  1. description field in HARDCODED_DEFAULTS — displayed as header in show-profile, included in to_annotated_dict()
  2. Contributed OODA profiles in tests/fixtures/ooda_profiles.json — your four profiles (orient, observe, decide, act) with multiplicative-compatible values
  3. 5 new tests validating all four OODA profiles resolve correctly (31 profile tests total, 597 passing)

scoring_mode: "weighted" is tracked as Issue #537 for follow-up.

$ mempalace synapse show-profile orient
[orient] effective config (resolved):
  Broad context gathering — association emphasis, long half-life

  half_life_days       : 180        ← profile (config.json)
  ltp_enabled          : True       ← profile (config.json)
  ltp_max_boost        : 2.0        ← profile (config.json)
  ltp_window_days      : 60         ← profile (config.json)
  tagging_enabled      : False      ← profile (config.json)
  ...

31 profile tests + 36 synapse tests = 67 Synapse-specific tests. All review points addressed.

@web3guru888

Copy link
Copy Markdown

@matrix9neonebuchadnezzar2199-sketch — really nice to see the OODA profiles land as actual test fixtures. The show-profile orient output looks exactly right — description header, 180-day half-life, LTP on, tagging off. That's the Orient profile as we defined it.

A few quick checks I'd run against the fixtures:

  1. Decide profile: ltp_enabled: False + tagging_enabled: True + half_life_days: 7 (or similar short). Should behave like an exploit mode.
  2. Act profile: All axes moderate, tagging enabled for rapid reinforcement.
  3. Cross-profile isolation: Confirm that switching from orient to decide at query time doesn't bleed LTP boosts between sessions (the session-scope issue from feat: Synapse Phase 1 — biologically-inspired memory scoring layer (#441) #451).

31 profile tests + 67 Synapse tests total is a healthy coverage baseline. The scoring_mode: weighted-sum split-out to #537 is the right call — keeps this PR scoped. Once #537 merges, the OODA profiles will need a second pass to add scoring_mode fields, but that's additive.

+1 to merge. This is ready.

- ooda_profiles: decide exploit mode (7d half-life, tagging, no LTP/assoc)
- act: LTP + tagging boosts + moderate settings; axes include ltp/tagging
  so ProfileManager does not disable them from axes_enabled list
- Add 4 tests (web3guru888): exploit mode, act tagging, cross-profile isolation

Made-with: Cursor
@matrix9neonebuchadnezzar2199-sketch

Copy link
Copy Markdown
Author

@web3guru888 All three suggestions implemented in the latest push.

1. Decide profile — exploit mode:

half_life_days: 7, ltp_enabled: false, tagging_enabled: true, association_enabled: false

Short half-life + tagging only = surface the most recent, actively-tagged drawers.

2. Act profile — moderate with rapid reinforcement:

half_life_days: 14, ltp_enabled: true (max 1.5), tagging_enabled: true (max 1.3), association_enabled: false

Note: adjusted axes_enabled to ["similarity", "recency", "ltp", "tagging"] to match the enforcement logic — without ltp/tagging in the list, _validate would force them off regardless of the explicit flags.

3. Cross-profile isolation:

orient = pm.resolve("orient")   # ltp_enabled=True, max_boost=2.0
decide = pm.resolve("decide")   # ltp_enabled=False
orient_again = pm.resolve("orient")  # still ltp_enabled=True, max_boost=2.0

No state leakage between resolve calls — each profile starts fresh from the merge chain.

35 profile tests, 601 total passing. Thanks for the thorough review collaboration.

@web3guru888

Copy link
Copy Markdown

@matrix9neonebuchadnezzar2199-sketch — all three land exactly right.

Decide profile: short half-life + tagging-only is the correct exploit mode. No association drift, no LTP amplification — just recency and active intent signals. Exactly what you want when the agent is in a decision loop.

Act profile: the axes_enabled fix is important — good catch. Validator enforcement without the axis list would silently disable the flags you set explicitly, which would be a confusing silent failure. Having axes_enabled: ["similarity", "recency", "ltp", "tagging"] be the source of truth for Act makes the intent clear.

Cross-profile isolation: the resolve-test pattern is the right way to verify this. No state leakage between resolve calls is exactly what you need for reliable OODA cycling — if Orient bled into Decide, the exploit phase would carry forward exploration weights.

35 profile tests, 601 total — this is ready. LGTM, +1 to merge.

@web3guru888 web3guru888 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All three implementation requests confirmed in the latest push. OODA profiles land as test fixtures with correct parameters (orient=180d LTP, explore=60d assoc, decide=7d tagging-only, act=low decay no-amplification). The show-profile command output is clean, the axes_enabled:[] test validates the disabled-axes case, and load-time validation catches typos at config load rather than query time.

From our integration's perspective, having named profiles with validated fields is exactly the right contract for production use — particularly for the decision loop scenario where a short half-life matters and LTP amplification would be wrong. LGTM, ready to merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/cli CLI commands area/mcp MCP server and tools area/mining File and conversation mining area/search Search and retrieval enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants