Spatial video memory prototype from PRD.md.

Runnable local ingestion shell:

• FastAPI backend
• React browser control surface
• Environment health checks for ffmpeg, colmap, optional mast3r_sfm, and opensplat
• Video upload into data/sessions/
• Frame extraction via ffmpeg
• SfM reconstruction step wired as an explicit checkpoint action with COLMAP and optional MASt3R-SfM engines
• Clickable stage inspector with artifacts, frame previews, and command logs

• backend/, frontend/, tests/ — MemorySplat app
• harness-agent-layer/ — agent runtime (orchestrator, formerly claw-code/). Independent git checkout. See harness-agent-layer/USAGE.md.
• obsidian-wiki/ — KB framework (skills only, no Python deps). Provides wiki-query, daily-update, etc. See obsidian-wiki/AGENTS.md.
• knowledge_base/ — Obsidian vault (gitignored except AGENTS.md). Vault-local conventions for memory-shaped data live in knowledge_base/AGENTS.md.
• docs/ingest_contract.md — §7.5 ↔ §7.6 handoff spec.
• tasks/todo.md — current work plan and slice breakdown.

Backend (single shared venv on Python 3.13):

uv venv --python python3.13 .venv
.venv/bin/pip install -e ".[dev]"
.venv/bin/uvicorn backend.main:app --reload

KB layer — project-scoped only. The harness agent's <cwd>/.claude/skills/ lookup hits memorylane/.claude/skills/, which contains relative symlinks into obsidian-wiki/.skills/<name> — so all 31 framework skills are discoverable when the harness runs anywhere inside memorylane/. Vault path is set in memorylane/.env (gitignored). User-scope skill installations (~/.claude/skills/, ~/.codex/skills/, etc.) and ~/.obsidian-wiki/config are intentionally not used.

Add MemorySplat-specific skills (e.g. memorylane-recap) as real directories alongside the symlinks: memorylane/.claude/skills/memorylane-recap/SKILL.md.

Frontend:

cd frontend
pnpm install
pnpm dev

Open http://127.0.0.1:5173.

Pipeline state is stored locally under data/sessions/<session_id>/. Reloading the page or restarting the backend will reload existing sessions, uploaded videos, extracted frames, COLMAP outputs, and logs. The React inspector also stores the selected session/stage in the URL query string.

Set these in .env to enable the Cloudinary checkpoint:

CLOUDINARY_CLOUD_NAME=
CLOUDINARY_API_KEY=
CLOUDINARY_API_SECRET=
CLOUDINARY_FOLDER=memorysplat

After OpenSplat and Orbs are done, click Sync Cloudinary. The backend uploads splat.ply, memory frame media, and a cloud-ready memories.json under data/sessions/<session_id>/opensplat/cloudinary/.

COLMAP is still the default reconstruction engine. MASt3R-SfM is an optional reconstruction engine for the same SfM checkpoint. The MemorySplat integration expects MASt3R to produce a COLMAP-compatible sparse model under {colmap_dir}/sparse/0, then OpenSplat consumes that model exactly like a normal COLMAP run.

The integration wrapper is:

scripts/mast3r_sfm.py

It runs MASt3R matching, writes a COLMAP database, runs pycolmap mapping, and copies the resulting reconstruction into {colmap_dir}/sparse/0.

Clone MASt3R beside this repository:

cd /Users/itsmarsss/Documents/Projects
git clone --recursive https://github.com/naver/mast3r
cd mast3r

Use Python 3.11. Do not pin cmake=3.14.0 on Apple Silicon; that old build may not exist in default Anaconda channels.

conda create -y -n mast3r python=3.11
conda install -y -n mast3r -c conda-forge cmake ninja

Install MASt3R, DUSt3R, and the COLMAP export dependencies:

cd /Users/itsmarsss/Documents/Projects/mast3r
conda run -n mast3r python -m pip install --no-cache-dir -r requirements.txt -r dust3r/requirements.txt
conda run -n mast3r python -m pip install --no-cache-dir pycolmap kapture kapture-localization

On CUDA machines, install a CUDA-enabled PyTorch build before the requirements if your default pip install torch does not pick one:

conda run -n mast3r python -m pip install --no-cache-dir torch torchvision --index-url https://download.pytorch.org/whl/cu121

On CPU-only machines or Apple Silicon Macs, the normal PyPI torch build is acceptable, but MASt3R will be much slower. The CUDA-only RoPE extension build can be skipped on CPU/Mac. If you try to build it and see CUDA_HOME environment variable is not set, that is expected on CPU/Mac.

Download the main MASt3R checkpoint locally so runtime does not depend on Hugging Face downloads:

cd /Users/itsmarsss/Documents/Projects/mast3r
mkdir -p checkpoints
curl -L --fail --continue-at - \
  --output checkpoints/MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth \
  https://download.europe.naverlabs.com/ComputerVision/MASt3R/MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth

The checkpoint is about 2.6 GB. Keep several GB free for reconstruction outputs.

Set MAST3R_SFM_COMMAND in this repo's .env.

CPU or Apple Silicon fallback:

MAST3R_SFM_COMMAND='/opt/homebrew/anaconda3/envs/mast3r/bin/python -u /Users/itsmarsss/Documents/Projects/memorylane/scripts/mast3r_sfm.py --image_dir {frames_dir} --output_dir {colmap_dir} --mast3r_root /Users/itsmarsss/Documents/Projects/mast3r --device cpu --weights /Users/itsmarsss/Documents/Projects/mast3r/checkpoints/MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth'

CUDA:

MAST3R_SFM_COMMAND='/path/to/conda/envs/mast3r/bin/python -u /path/to/memorylane/scripts/mast3r_sfm.py --image_dir {frames_dir} --output_dir {colmap_dir} --mast3r_root /path/to/mast3r --device cuda --weights /path/to/mast3r/checkpoints/MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth'

Supported placeholders are {frames_dir}, {image_dir}, {colmap_dir}, {output_dir}, and {sparse_dir}.

Use the environment's Python binary directly instead of conda run; conda run can buffer stdout and make the live pipeline console look frozen.

Check imports and device availability:

/opt/homebrew/anaconda3/envs/mast3r/bin/python -c "import torch; print('cuda', torch.cuda.is_available()); print('mps', torch.backends.mps.is_available())"
/opt/homebrew/anaconda3/envs/mast3r/bin/python -c "import pycolmap, kapture; from mast3r.model import AsymmetricMASt3R; print('mast3r deps ok')"

For this Apple Silicon setup, expected output is CPU-only:

cuda False
mps False

When running from the pipeline, the stage console should immediately show lines like:

[mast3r-sfm] Python executable: ...
[mast3r-sfm] Requested device: cpu
[mast3r-sfm] cuda_available=False
[mast3r-sfm] mps_available=False

Restart the backend after changing .env; .env is loaded when the backend process starts.

• No live logs: make sure MAST3R_SFM_COMMAND uses .../envs/mast3r/bin/python -u, not conda run.
• FutureWarning: torch.cuda.amp.autocast: harmless PyTorch deprecation warning from MASt3R/DUSt3R internals.
• mps_built=True but mps_available=False: PyTorch was built with MPS support, but the current runtime cannot use it. Use --device cpu.
• Segfault after importing pycolmap: the wrapper loads the MASt3R model before importing pycolmap to avoid macOS native runtime conflicts.
• Torch not compiled with CUDA enabled: the command or MASt3R internals tried to use CUDA with a CPU-only torch build. Use --device cpu or install CUDA-enabled PyTorch.
• No good initial image pair found: MASt3R produced matches, but pycolmap could not initialize a sparse model. Try more frames, better overlap/motion, or a different frame extraction FPS.
• Disk errors or silent failures: the checkpoint is large and reconstructions write databases, caches, and sparse models. Keep several GB free.

.venv/bin/pytest
cd frontend && pnpm build