Skip to content

fix(hooks): replace mapfile with sed for bash 3.2 compat (#1440)#1441

Merged
igorls merged 4 commits into
MemPalace:developfrom
mvalentsev:fix/hooks-bash-3.2-compat-1440
May 17, 2026
Merged

fix(hooks): replace mapfile with sed for bash 3.2 compat (#1440)#1441
igorls merged 4 commits into
MemPalace:developfrom
mvalentsev:fix/hooks-bash-3.2-compat-1440

Conversation

@mvalentsev

@mvalentsev mvalentsev commented May 10, 2026

Copy link
Copy Markdown
Contributor

What this PR does

The legacy hooks/mempal_save_hook.sh and hooks/mempal_precompact_hook.sh use mapfile -t (introduced in #1231) to read sanitized values from the inline Python JSON parser. On stock macOS /bin/bash is GNU 3.2.57 (Apple GPLv3 freeze) and mapfile only landed in bash 4.0 (2009), so the hook silently fails: every parsed value falls back to its default, the log writes Session unknown: 0 exchanges, and the hook returns {} + exit 0. Users see hooks "firing" with zero drawers added (#1440).

This PR replaces the array read with sed -n 'Np' line extraction, keeping the same one-value-per-line contract from the #1231 security hardening but staying within POSIX so /bin/bash 3.2 runs the scripts unmodified.

It also adds a defense-in-depth fail-loud guard. Python now prints a leading __MEMPAL_PARSE_OK__ sentinel; if the sentinel is missing after parsing, the hook writes a WARN line to hook.log and dumps the first 4096 bytes of the raw input to last_input.log plus the Python traceback to last_python_err.log, both at mode 0600. This means the next time parsing silently fails for any reason (sanitizer regex too tight, broken interpreter, future inline-script regression), a maintainer has the raw payload and the Python error instead of a day of "Session unknown" log lines.

How to test

  • New tests: tests/test_hooks_bash_compat.py (26 cases, mirrored across both hooks where applicable) cover (a) mapfile/readarray absent in both scripts, (b) sed -n extraction shape present, (c) hook produces the expected log line under controlled stdin, (d) fail-loud guard fires on malformed JSON but not on empty stdin / unicode-stripped / literal "unknown" session_id, (e) the dump caps at exactly 4096 bytes including under a UTF-8 locale with a multibyte payload, (f) last_input.log and last_python_err.log are mode 0600 on failure and absent on success, (g) hook stdout stays valid JSON.
  • Full suite: uv run pytest tests/ -v --ignore=tests/benchmarks stays green.
  • Manual: echo '{"session_id":"abc","stop_hook_active":false,"transcript_path":""}' | bash hooks/mempal_save_hook.sh and inspect ~/.mempalace/hook_state/hook.log for Session abc:.

Checklist

  • Tests added covering the regression
  • No CHANGELOG.md change (maintainer-managed)
  • No emoji / AI attribution in source or commits

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request replaces Bash 4-specific mapfile with sed in the hook scripts to ensure compatibility with macOS Bash 3.2. It also introduces a sentinel-based error detection mechanism, captures Python stderr, and implements a fail-loud guard that logs malformed input (capped at 4KB) with restricted file permissions. A comprehensive test suite is added to verify these changes. The review feedback suggests using printf '%s' instead of echo when passing input to the Python interpreter to handle special characters more robustly.

Comment thread hooks/mempal_precompact_hook.sh Outdated
Comment thread hooks/mempal_save_hook.sh Outdated
@mvalentsev mvalentsev marked this pull request as ready for review May 11, 2026 00:23
@mvalentsev mvalentsev force-pushed the fix/hooks-bash-3.2-compat-1440 branch from cfd8248 to 0f6cd65 Compare May 13, 2026 07:17
@igorls igorls added bug Something isn't working area/hooks Claude Code hook scripts (Stop, PreCompact, SessionStart) labels May 15, 2026
@mvalentsev mvalentsev force-pushed the fix/hooks-bash-3.2-compat-1440 branch from 0f6cd65 to 09d91a9 Compare May 15, 2026 13:41
)

The legacy hook scripts used `mapfile -t` (introduced in MemPalace#1231) to read
sanitized values from the inline Python JSON parser. On stock macOS
/bin/bash is GNU 3.2.57 (Apple GPLv3 freeze) and `mapfile` only landed
in bash 4.0, so the hook silently failed: every parsed value fell back
to its default, the log wrote "Session unknown: 0 exchanges", and zero
drawers were saved.

This commit replaces the array read with `sed -n 'Np'` line extraction,
which is POSIX and runs on /bin/bash 3.2 unmodified. It also adds a
defense-in-depth fail-loud guard: Python prints a `__MEMPAL_PARSE_OK__`
sentinel and the hook surfaces the raw input plus Python traceback to
0600-mode sibling files when the sentinel is missing, so the next silent
failure has a debuggable trail instead of an unsolved zero-drawer day.
Follow-up on c271e7b after the bot review and an adversarial pass.

- Replace `echo "$INPUT"` with `printf '%s' "$INPUT"` in both hooks
  so payloads with leading `-n`/`-e`/`-E` or backslashes are not
  eaten by echo flag parsing.
- Wrap the `last_python_err.log` and `last_input.log` writes in a
  `( umask 077; ... )` subshell so the files are created at mode
  0600 atomically. The existing `chmod 600` calls stay as
  belt-and-suspenders.
- Refactor the `_run_hook` test helper: internal assert on
  returncode with stderr surfaced in the failure message; new
  `extra_env=` kwarg subsumes the ad-hoc `subprocess.run` in the
  UTF-8 locale test.
- Promote the world-readable, bounded-overwrite, UTF-8, unicode and
  literal-unknown tests to a shared `_BOTH_HOOKS` parametrize with
  readable ids so the symmetry across both hooks is enforced.
- `test_sed_extraction_present` strips line comments before counting
  `sed -n '` occurrences so a regression that swapped sed back to
  mapfile while keeping the prose cannot false-pass.
- Rewrite the precompact hook header docblock: the previous
  "ALWAYS blocks" wording was stale (the hook returns `{}` and
  does not emit a `decision: block` payload to the Stop-hook
  protocol; the "always run" guarantee comes from the synchronous
  mine call).
- Apply ruff 0.4.x format to the new test file (matches CI pin
  `pip install "ruff>=0.4.0,<0.5"`).

Tests: 1749 passed (was 21 cases in c271e7b's test file; now 26 after
parametrize promotions, +5 cases).
Cycle 2 follow-up: harden the permission tests so they actively
falsify the "ambient umask masking a regression" hypothesis raised
during the adversarial review pass.

- `_run_hook` now sets `preexec_fn=lambda: os.umask(0o022)` on the
  bash subprocess. The child starts with a permissive 022 umask so a
  bare file would land at mode 0644. The hook's own `umask 077` in
  the parse subshell remains the only thing protecting
  `last_python_err.log` and `last_input.log`, so `test_dump_is_not_world_readable`
  and `test_python_stderr_log_is_not_world_readable_on_failure` now
  empirically prove that protection rather than passively benefitting
  from a CI runner's restrictive ambient umask.
- Docstring precision: the helper's primary diagnostic surface is
  `hook.log` (a file under `STATE_DIR`), not the subprocess's stderr.
  Subprocess stderr is captured here only to surface unexpected
  interpreter failures (e.g., a bash syntax error introduced by a
  future edit), and the docstring now says so.

No production-code changes. 26/26 tests still pass.
@mvalentsev mvalentsev force-pushed the fix/hooks-bash-3.2-compat-1440 branch from 9e7228a to 1988f3d Compare May 15, 2026 17:51
@igorls igorls added this to the v3.3.6 milestone May 15, 2026
@igorls igorls merged commit fc4d06d into MemPalace:develop May 17, 2026
6 checks passed
arnoldwender pushed a commit to arnoldwender/mempalace that referenced this pull request May 24, 2026
Bumps version 3.3.5 → 3.3.6 across pyproject.toml, version.py, plugin
manifests (.claude-plugin/plugin.json, .claude-plugin/marketplace.json,
.codex-plugin/plugin.json), README badge, and uv.lock. Flips CHANGELOG.md
from ``[Unreleased]`` to ``[3.3.6] — 2026-05-24`` and backfills the
major user-facing entries that landed without changelog entries during
the cycle:

Features:
- MemPalace#1555 office-document mining via --mode extract + virtual line numbers
- MemPalace#1584 surgical closet pointers with date+line locators (Tier 6a)
- MemPalace#1558 + MemPalace#1560 within-wing hallways (entity co-occurrence graph)
- MemPalace#1565 cross-wing tunnels auto-promoted from hallways
- MemPalace#1578 Hebbian potentiation + Ebbinghaus decay on hallways/tunnels
- MemPalace#1236 API-tool transcripts auto-route to wing_api
- MemPalace#711 hooks.auto_save toggle for silent-mode sessions
- MemPalace#1605 COCA content-word filter for entity detection
- MemPalace#1557 case-insensitive entity matching at mine time
- MemPalace#1483 multilingual embeddings (embeddinggemma-300m) by default

Bug Fixes (selected, user-visible):
- MemPalace#1540 silent data loss in three unchunked upsert sites
- MemPalace#1538 paragraph chunker oversized chunks
- MemPalace#1554 per-file chunk cap too low for transcripts
- MemPalace#1562 Windows hook subprocess/ChromaDB deadlock
- MemPalace#1529 create_tunnel corrupted hyphenated wing names
- MemPalace#1424 save-hook truncated hyphenated project folders
- MemPalace#1383 KG cache duplicated graphs for symlinked/cased paths
- MemPalace#1466 silent symlink skip now logged
- MemPalace#1441 macOS stock-bash 3.2 hook compatibility
- MemPalace#1500 / MemPalace#1513 structured JSON-RPC errors on bad MCP input
- MemPalace#1523 VACUUM + FTS5 rebuild after repair
- MemPalace#1548 FTS5 validation at end of mine
- plus MemPalace#1216, MemPalace#1408, MemPalace#1438, MemPalace#1439, MemPalace#1445, MemPalace#1452, MemPalace#1459, MemPalace#1461, MemPalace#1466,
  MemPalace#1470, MemPalace#1477, MemPalace#1485, MemPalace#1500, MemPalace#1513, MemPalace#1528, MemPalace#1532, MemPalace#1543, MemPalace#1546, MemPalace#1585

Performance:
- MemPalace#1474 convo miner pre-fetches mined-set
- MemPalace#1487 rebuild_index progress callback
- MemPalace#1530 MCP cold-start diagnostics + opt-in warmup

Lint passes (ruff 0.15.14); mempalace-mcp entry point alignment
verified per RELEASING.md.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/hooks Claude Code hook scripts (Stop, PreCompact, SessionStart) bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants