Lightweight embedding inference via ggml. No Python runtime, no ONNX. Text, image, and face embeddings in one binary.

Text: 10 architectures (BERT, XLM-R, MPNet, NomicBERT, ModernBERT, GTE v1.5, Qwen3, Gemma3, SPLADE, DeBERTa-v2). Dense, sparse (SPLADE/BGE-M3), ColBERT multi-vector, cross-encoder rerankers, bi-encoder reranking.

NER: Zero-shot Named Entity Recognition via GLiNER. Two backbone options: DeBERTa-v3-base (209M, Apache-2.0, recommended) and LFM2.5-350M (LFM-1.0). Detect arbitrary entity types at inference time — no retraining needed. CLI (--ner), server (POST /ner/extract), Python (CrispNER), Rust, Dart.

Vision: CLIP and SigLIP text-image cross-modal search (encode text and images into the same vector space). YuNet/SCRFD face detection, ArcFace/SFace/AuraFace face recognition. Full detect-align-encode pipeline.

General OCR: Full text detection + recognition pipeline. DBNet (ResNet-18 + FPNC, 7 MB Q4_K) detects text regions, TrOCR-small (DeiT + Transformer decoder, 63 MB Q8_0) recognizes each crop. ~200ms per region. CLI (--ocr-det/--ocr-rec), server (POST /ocr), Python (CrispOcrPipeline), Rust (OcrPipeline), Flutter (CrispOcrPipeline).

Scene Text OCR: PARSeq (ViT encoder + Transformer decoder, 24M params, 94-char ASCII, Apache-2.0). Base 91 MB F32 / 24 MB Q8_0, Tiny 12 MB F16 / 6 MB Q8_0. State-of-the-art accuracy for scene text recognition (ECCV 2022). CLI (--ocr), server (POST /math/ocr), Python (CrispMathOcr), auto-detected.

Math OCR: Seven engines for math-image → LaTeX: PP-FormulaNet-L (printed, SAM-ViT+MBart 181M, Apache-2.0, 122 MB Q4_K), Texo-Distill (printed, HGNetv2+MBart 20M, BLEU 0.90), DeiT+TrOCR (printed, 17 MB Q4_K), PosFormer (handwritten, DenseNet+Transformer+ARM, 57% CROHME), BTTR (handwritten, DenseNet+Transformer, 53% CROHME), HMER (handwritten, DenseNet+GRU attention). All auto-detected from GGUF metadata, ~3-5s decoder time.

9.5x faster than FastEmbed (ONNX) on MiniLM-L6. Python/Rust/Dart APIs. GPU acceleration (CUDA/Vulkan/Metal). iOS + Android + WASM builds. 58 models in registry (text, vision, face), 160+ GGUF variants on HF.

Browser: Math OCR compiles to WebAssembly (1 MB) via build-wasm.sh. Runs entirely client-side — no server, no API key. GGUF models fetched on demand and cached in IndexedDB.

Demo: HuggingFace Space — text embeddings (cosine similarity) + math OCR (image → LaTeX), auto-deployed from hf-space/.

28 embedding models shown below (all pass, cos>=0.965 vs HF), 59 models total in registry (including rerankers, SPLADE, CLIP/SigLIP vision, CLIP text, YuNet/SCRFD face detection, AuraFace/SFace recognition):

Q8_0 = all PASS (cos >= 0.995). Q4_K = most PASS; -- = SwiGLU/GeGLU too sensitive for aggressive quants. Q4_K cosines for the 4B/8B decoder rows are min-cos vs bf16 HF (the native training precision; full f32 would not fit in 16 GB RAM).

• NomicBERT quantization: SwiGLU gate/value projections are sensitive to aggressive quantization. Q5_K (cos0.95) and Q4_K (cos0.85) degrade significantly. Use F32 or Q8_0 only for this model.
• Jina v5 LoRA adapters: Jina v5 models use task-specific LoRA adapters. GGUFs have the retrieval adapter merged. Other tasks (text-matching, clustering, classification) require separate GGUFs. Performance (Apple M1, Metal):

Model: all-MiniLM-L6-v2. See PERFORMANCE.md for full multi-model benchmarks.

Ollama-compatible: All 45 registry models export as Ollama-compatible GGUFs by default (--ollama converter flag). 13 models verified end-to-end in our Ollama fork (adds XLM-R, Viterbi SentencePiece tokenizer, GELU_ERF, multi-tokenizer BERT support). See PERFORMANCE.md for Ollama Q8_0/Q4_K cosine results.

BidirLM-Omni Q4_K verified locally: text, audio, and raw vision-patch embedding all load and emit 2048-d vectors from bidirlm-omni-2.5b-q4_k.gguf. Current CPU smoke benchmark on Apple M1: text batch of 4 = 373.2 ms, JFK audio = 618.7 ms, synthetic 2x2 vision patches = 11.0 ms. Regression gate used for graph changes: cosine >= 0.99999 and max abs diff <= 1e-3 against saved baseline vectors.

# Clone with submodule
git clone --recursive https://github.com/CrispStrobe/CrispEmbed
cd CrispEmbed

# Build (CPU)
cmake -S . -B build
cmake --build build -j

# Encode text
./build/crispembed -m model.gguf "Hello world"

# Matryoshka truncation (e.g. 128 dims from a 384-dim model)
./build/crispembed -m model.gguf -d 128 "Hello world"

# Prefix + capability inspection
./build/crispembed -m model.gguf --prefix "query: " --capabilities

# BidirLM-Omni text/audio/raw-vision patch embedding
# Put local paths in a gitignored `.env.local`, then source it here.
source .env.local
./build/crispembed -m bidirlm-omni-2.5b "Hello world"
./build/crispembed -m bidirlm-omni-2.5b --audio "$CRISPEMBED_BIDIRLM_AUDIO"
./build/crispembed -m bidirlm-omni-2.5b \
    --image-raw "$CRISPEMBED_BIDIRLM_PATCHES" --grid-thw 1,14,14

# Sparse / ColBERT retrieval (BGE-M3)
./build/crispembed -m bge-m3.gguf --sparse "Hello world"
./build/crispembed -m bge-m3.gguf --colbert "Hello world"

# Cross-encoder and bi-encoder reranking
./build/crispembed -m bge-reranker-v2-m3.gguf --rerank "capital of france" \
    "Paris is the capital of France." "Bicycles have two wheels."
./build/crispembed -m model.gguf --biencoder "capital of france" --top-n 2 \
    "Paris is the capital of France." "Berlin is the capital of Germany."

# CLIP text-image search (cross-modal)
./build/crispembed -m clip-text-base "a photo of a cat"
./build/crispembed -m clip-vit-base-patch16 --image photo.jpg

# Face detection (YuNet: 0.2 MB, or SCRFD: 16 MB)
./build/crispembed -m yunet --detect photo.jpg --json

# CLI parity test
python tests/test_cli_parity.py --cli ./build/crispembed \
    --dense-model "$CRISPEMBED_DENSE_MODEL" \
    --retrieval-model "$CRISPEMBED_RETRIEVAL_MODEL" \
    --reranker-model "$CRISPEMBED_RERANKER_MODEL"

# Start server (text + vision + face + CLIP + OCR + NER)
./build/crispembed-server -m model.gguf \
    --vit clip-vit-base-patch16.gguf \
    --clip-text clip-text-base.gguf \
    --det yunet.gguf \
    --ocr ppformulanet-l-q8_0.gguf \
    --ner gliner-lfm-f32.gguf --port 8080
curl -X POST http://localhost:8080/embed -d '{"texts": ["Hello world"]}'
curl -X POST http://localhost:8080/clip/text -d '{"text": "a photo of a cat"}'
curl -X POST http://localhost:8080/vit/encode -d '{"image": "photo.jpg"}'
curl -X POST http://localhost:8080/math/ocr -d '{"image": "formula.png"}'
curl -X POST http://localhost:8080/ner/extract \
  -d '{"text": "Tim Cook at Apple", "labels": ["person", "organization"]}'

Eleven engines for image → text, all auto-detected from GGUF metadata via the unified crispembed_math_ocr_* C API. Available through CLI (--ocr), HTTP server (POST /math/ocr), Python (CrispMathOcr), Rust, and Dart/Flutter.

PP-FormulaNet-L (recommended for printed math): SAM-ViT encoder with windowed + global attention and decomposed relative position bias, full ggml graph compute. Encoder parity cos=0.999962 vs HuggingFace reference.

# CLI
./build/crispembed -m ppformulanet-l --ocr formula.png

# Server
./build/crispembed-server --ocr ppformulanet-l-q8_0.gguf --port 8080
curl -X POST http://localhost:8080/math/ocr -d '{"image": "formula.png"}'

# Python
from crispembed import CrispMathOcr
ocr = CrispMathOcr("ppformulanet-l-q8_0.gguf")
latex = ocr.recognize("formula.png")

# C API
void *ctx = crispembed_math_ocr_init("ppformulanet-l-q8_0.gguf", 4);
const char *latex = crispembed_math_ocr_recognize(ctx, pixels, w, h, ch, &len);

Flutter integration: The flutter/crispembed/ plugin provides CrispEmbedOcr for Dart FFI access. Used by CrispCalc for camera-based math input. Platform dirs (Linux/Windows/macOS/iOS/Android) with CI-built native libraries.

Document layout analysis via RT-DETRv2 (ResNet-50 + HybridEncoder + deformable cross-attention decoder). Detects 17 region types: text, title, table, figure, formula, caption, section_header, list_item, footnote, page_header, page_footer, code, document_index, checkbox_selected, checkbox_unselected, form, key_value_region.

Auto-detected from GGUF metadata (general.architecture = layout). Available through CLI (--layout), HTTP server (POST /layout/detect), Python (CrispLayout), Rust (CrispLayout), and Dart/Flutter.

# CLI
./build/crispembed -m layout-heron --layout document.png --json

# Server
./build/crispembed-server --layout layout-heron-f32.gguf --port 8080
curl -X POST http://localhost:8080/layout/detect -d '{"image": "page.png"}'

# Python
from crispembed import CrispLayout
layout = CrispLayout("layout-heron-f32.gguf")
regions = layout.detect("page.png")
for r in regions:
    print(f"{r['label']} ({r['score']:.2f})")

Encoder parity: all stages cos=1.0 vs HF reference. Detection score 0.93 (HF: 0.95). Performance: 21s with BLAS (F32), Q8_0 model 43 MB. Source: docling-project/docling-layout-heron (Apache-2.0). Models: cstr/layout-heron-gguf

Zero-shot NER via GLiNER with an LFM2.5-350M bidirectional backbone (16 layers: 10 ShortConv + 6 GQA attention, SwiGLU FFN). Detects arbitrary entity types specified at inference time — no retraining needed. Ported from CrispASR's LFM2 backbone with bidirectional attention and symmetric convolutions.

Architecture: BPE tokenize → LFM2.5-bi backbone → layer fusion (squeeze-and-excitation) → BiLSTM → GLiNER span-label matching head (SpanMarkerV1 + dot-product scorer).

# CLI
./build/crispembed -m gliner-lfm-f32.gguf --ner "Maria Schmidt arbeitet bei Siemens in München"

# Server
./build/crispembed-server --ner gliner-lfm-f32.gguf --port 8080
curl -X POST http://localhost:8080/ner/extract \
  -d '{"text": "Maria Schmidt arbeitet bei Siemens", "labels": ["person", "organization"], "threshold": 0.5}'

# Python
from crispembed import CrispNER
ner = CrispNER("gliner-lfm-f32.gguf")
entities = ner.extract("Maria Schmidt arbeitet bei Siemens in München",
                       labels=["person", "organization", "location"])
for e in entities:
    print(f"{e['text']} => {e['label']} ({e['score']:.2f})")

Parity: all 16 backbone layers cos=1.000000 vs HF reference. Layer fusion and BiLSTM cos=1.000000. 17/17 entities match across 5 test texts. Source: VAGOsolutions/SauerkrautLM-LFM2.5-GLiNER (LFM Open License v1.0).

The auto-download registry (-m <name>) covers models under multiple licenses. CrispEmbed itself is permissive, but the model you download is governed by its upstream license — converting a checkpoint to GGUF does not relicense it. Always check --list-models (the License column) or the upstream model card before using a model commercially.

Restricted-license entries (NC + Gemma) are marked with * in --list-models. Auto-download for them requires explicit consent:

# Interactive (TTY): you'll be shown the license + model card URL and
# prompted to accept.
./build/crispembed -m jina-v5-nano "hello"

# Non-interactive (CI, scripts): pass the SPDX tag.
./build/crispembed -m jina-v5-nano --accept-license cc-by-nc-4.0 "hello"

# Or via env var (useful for shared scripts).
export CRISPEMBED_ACCEPT_LICENSE=cc-by-nc-4.0
./build/crispembed -m jina-v5-nano "hello"

# Special value "all" accepts every license tag — intended for trusted
# environments where you've already vetted the registry.
export CRISPEMBED_ACCEPT_LICENSE=all

--accept-license is an affirmative acknowledgement that the caller accepts the upstream terms; it does not grant rights you don't otherwise have. For commercial use of a CC BY-NC model you still need a separate license from the model author.

To audit what the registry actually carries (cross-checks the upstream license, the cstr/*-GGUF re-host's declared license, and the claim in models/upload_to_hf.py):

python tests/check_registry_licenses.py            # human-readable table
python tests/check_registry_licenses.py --json     # for CI

# CPU only (default)
cmake -S . -B build && cmake --build build -j

# With OpenBLAS acceleration
cmake -S . -B build -DGGML_BLAS=ON && cmake --build build -j

# With Intel MKL
cmake -S . -B build -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Intel10_64lp

# With CUDA (NVIDIA GPU)
cmake -S . -B build -DGGML_CUDA=ON && cmake --build build -j

# With Vulkan (cross-platform GPU)
cmake -S . -B build -DGGML_VULKAN=ON && cmake --build build -j

# macOS with Metal (recommended)
./build-macos.sh              # Metal + Accelerate + embedded shaders
./build-macos.sh --cpu        # CPU only, no Metal
./build-macos.sh --shared     # Also build shared lib for Python

Requires Visual Studio 2022 Build Tools + Ninja.

:: CPU build
build-windows.bat

:: Vulkan GPU build (needs Vulkan SDK)
build-vulkan.bat

:: CUDA GPU build (needs CUDA Toolkit)
build-cuda.bat

If you get "ggml does not contain a CMakeLists.txt", run:

git submodule update --init --recursive

• Required: C++17 compiler, CMake 3.14+
• Optional: OpenBLAS (apt install libopenblas-dev), Intel MKL, CUDA Toolkit, Vulkan SDK

cmake --install build --prefix /usr/local (or any other prefix) lays out a standard distro tree:

<prefix>/
  bin/{crispembed, crispembed-server, crispembed-quantize}
  lib/
    libcrispembed.so.0.3.0        (real file)
    libcrispembed.so.0            (SONAME symlink — SOVERSION 0)
    libcrispembed.so              (linker symlink)
    libggml*.so*                  (ggml backend siblings)
    cmake/crispembed/             (find_package(crispembed) plumbing)
    pkgconfig/crispembed.pc       (pkg-config --cflags --libs crispembed)
  include/
    crispembed.h
    ggml*.h

The installed .so/.dylib carries RPATH=$ORIGIN (Linux) / @loader_path (macOS) so it finds its libggml* siblings without LD_LIBRARY_PATH. The installed binaries carry RPATH=$ORIGIN/../lib / @loader_path/../lib.

Downstream CMake consumers:

find_package(crispembed REQUIRED)
target_link_libraries(my_app PRIVATE crispembed::crispembed)

Downstream pkg-config consumers:

$ pkg-config --cflags --libs crispembed
-I/usr/local/include -L/usr/local/lib -lcrispembed

The pkg-config file is relocatable (prefix=${pcfiledir}/../..), so extracting a release tarball into /opt/foo and pointing PKG_CONFIG_PATH=/opt/foo/lib/pkgconfig Just Works without editing the .pc file.

# BERT / XLM-R encoder models
pip install torch transformers gguf
python models/convert-bert-to-gguf.py \
    --model sentence-transformers/all-MiniLM-L6-v2 \
    --output all-MiniLM-L6-v2.gguf

# Qwen3 / Gemma3 decoder models
python models/convert-decoder-embed-to-gguf.py \
    --model Octen/Octen-Embedding-0.6B \
    --output octen-0.6b.gguf

# Quantize (Q8_0 recommended, Q4_K for max compression)
./build/crispembed-quantize model.gguf model-q8_0.gguf q8_0
./build/crispembed-quantize model.gguf model-q4_k.gguf q4_k

Pre-converted models: HuggingFace cstr/

Embedding tables quantized to Q8_0 even in Q4_K mode (quality-sensitive).

CrispEmbed supports all three BGE-M3 retrieval modalities plus cross-encoder rerankers.

# Convert BGE-M3 (writes sparse_linear.weight + colbert_linear.weight into GGUF)
pip install torch transformers gguf FlagEmbedding
python models/convert-bert-to-gguf.py --model BAAI/bge-m3 --output bge-m3.gguf --crisp

# Validate all three heads against FlagEmbedding ground truth
python tests/test_bgem3.py --gguf bge-m3.gguf --lib build/libcrispembed.so

from crispembed import CrispEmbed

model = CrispEmbed("bge-m3.gguf")

# Dense (L2-normalised)
vec = model.encode("Hello world")                   # Vec<f32> len 1024

# Sparse (SPLADE-style term weights)
if model.has_sparse():
    sparse = model.encode_sparse("Hello world")     # {token_id: weight}

# ColBERT multi-vector
if model.has_colbert():
    multi = model.encode_multivec("Hello world")    # [[f32; 128]; n_tokens]

Cross-encoder rerankers:

reranker = CrispEmbed("bge-reranker-v2-m3.gguf")
score = reranker.rerank("query text", "document text")   # raw logit

Requires the shared library (--shared flag or -DCRISPEMBED_BUILD_SHARED=ON).

from crispembed import CrispEmbed

model = CrispEmbed("all-MiniLM-L6-v2.gguf")

# Single text
vec = model.encode("Hello world")      # shape (384,)

# Batch — single C call, true batched Metal/GPU inference
vectors = model.encode(["Hello world", "Goodbye world"])
print(vectors.shape)  # (2, 384)

# Matryoshka dimension truncation
model.set_dim(128)
vec128 = model.encode("Hello world")   # shape (128,)

# Prompt prefix (for models that need it)
model.set_prefix("query: ")           # auto-prepended before tokenization

# Sparse (BGE-M3)
model = CrispEmbed("bge-m3.gguf")
if model.has_sparse:
    sparse = model.encode_sparse("Hello world")   # {token_id: weight}

# ColBERT multi-vector
if model.has_colbert:
    multi = model.encode_multivec("Hello world")   # (n_tokens, 128)

# Cross-encoder reranking
reranker = CrispEmbed("bge-reranker-v2-m3.gguf")
score = reranker.rerank("query", "document")       # raw logit

# Bi-encoder reranking (any embedding model, cosine similarity)
results = model.rerank_biencoder("query", ["doc1", "doc2", "doc3"], top_n=2)
for r in results:
    print(f"  [{r['index']}] {r['score']:.4f}: {r['document']}")

# BidirLM-Omni: text, audio, image, and image-conditioned text in one shared 2048-d space
omni = CrispEmbed("bidirlm-omni-2.5b")
text_vec  = omni.encode("a small cat on a chair")
if omni.has_audio:
    audio_vec = omni.encode_audio(pcm_f32, sr=16000)             # 1-D float32 PCM
if omni.has_vision:
    # Two preprocessing paths:
    #   - encode_image(...)      uses HF Qwen2VL processor (tight parity with HF, requires `transformers`)
    #   - encode_image_file(...) uses the in-process C++ preprocessor (no transformers dep, ~0.97 cos vs HF)
    img_vec   = omni.encode_image("cat.jpg")
    img_vec_native = omni.encode_image_file("cat.jpg")
    img_raw, deepstack = omni.encode_image_raw("cat.jpg")        # un-pooled (n_merged, 2048)
    # Image-conditioned text — text must contain image_token_id placeholders
    text_with_img        = omni.encode_text_with_image(prompt, "cat.jpg")
    text_with_img_native = omni.encode_text_with_image_file(prompt, "cat.jpg")

Math OCR:

from crispembed import CrispMathOcr

ocr = CrispMathOcr("ppformulanet-l-q8_0.gguf")
latex = ocr.recognize("formula.png")           # file path
latex = ocr.recognize(pil_image)               # PIL Image
latex = ocr.recognize(numpy_uint8_array)       # (H, W, C) uint8
latex = ocr.recognize_gray(float32_array)      # (H, W) float32 [0..1]

Wrapper parity script:

python tests/feature_parity.py \
  --dense-model "$CRISPEMBED_DENSE_MODEL" \
  --retrieval-model "$CRISPEMBED_RETRIEVAL_MODEL" \
  --reranker-model "$CRISPEMBED_RERANKER_MODEL"

[dependencies]
crispembed = { git = "https://github.com/CrispStrobe/CrispEmbed" }

use crispembed::CrispEmbed;

let mut model = CrispEmbed::new("model.gguf", 0)?;
let vec = model.encode("Hello world");

// Prompt prefix
model.set_prefix("query: ");

// Sparse + ColBERT (BGE-M3)
if model.has_sparse() {
    let sparse = model.encode_sparse("query");   // Vec<(i32, f32)>
}
if model.has_colbert() {
    let multi = model.encode_multivec("query");  // Vec<Vec<f32>>
}

// Bi-encoder reranking (cosine similarity)
let ranked = model.rerank_biencoder("query", &["doc1", "doc2"], Some(2));
for (idx, score) in &ranked {
    println!("  doc {} score {:.4}", idx, score);
}

Wrapper parity script:

cargo run -p crispembed --example feature_parity -- \
  "$CRISPEMBED_DENSE_MODEL" \
  "$CRISPEMBED_RETRIEVAL_MODEL" \
  "$CRISPEMBED_RERANKER_MODEL"

# pubspec.yaml
dependencies:
  crispembed:
    path: <local Flutter plugin path>

import 'package:crispembed/crispembed.dart';

final model = CrispEmbed('model.gguf');

// Dense encoding
final vec = model.encode('Hello world');           // Float32List(384)
final batch = model.encodeBatch(['Hello', 'World']); // List<Float32List>

// Matryoshka truncation + prefix
model.setDim(128);
model.setPrefix('query: ');

// Bi-encoder reranking
final ranked = model.rerankBiencoder('query', ['doc1', 'doc2']);

// Sparse / ColBERT / cross-encoder (BGE-M3, rerankers)
if (model.hasSparse) {
  final sparse = model.encodeSparse('text');  // Map<int, double>
}

model.dispose();

Wrapper parity script:

cd flutter/crispembed
dart run example/feature_parity.dart \
  "$CRISPEMBED_DENSE_MODEL" \
  "$CRISPEMBED_RETRIEVAL_MODEL" \
  "$CRISPEMBED_RERANKER_MODEL" \
  "$CRISPEMBED_LIB"

Works on iOS (Metal GPU), Android (Vulkan/NEON), macOS, Linux, Windows.

Python, Rust, Dart, and the crispembed CLI now cover the same core inference features from the shared C API: dense encode, batch encode, Matryoshka truncation, prefix control, sparse retrieval, ColBERT multi-vector retrieval, cross-encoder reranking, and bi-encoder reranking.

The CLI still keeps some convenience-only UX that the wrappers do not mirror directly:

• CLI-only conveniences: --list-models, model-name auto-download, --cache-dir, -f FILE, --json, --dim, --capabilities.
• Wrapper-only convenience helpers: prefix getters and in-process ranking helper return types.

Inference capability parity is now aligned across all four entry points.

# iOS — xcframework with Metal GPU acceleration
./build-ios.sh                    # arm64 device + simulator
./build-ios.sh --device           # device only

# Android — NDK cross-compilation
./build-android.sh                # arm64-v8a + armeabi-v7a + x86_64
./build-android.sh --abi arm64-v8a --vulkan  # single ABI with Vulkan GPU

Output:

• iOS: build-ios/CrispEmbed.xcframework/
• Android: build-android/<abi>/libcrispembed.so

./benchmark.sh                          # single model, all engines
./benchmark.sh --multi                  # 6 models, all engines
./benchmark.sh -n 100 --skip-fastembed  # CrispEmbed + HF only, 100 runs

# RAG retrieval quality benchmark
python tests/bench_rag.py --lib build/libcrispembed.so --gguf model.gguf

# Reranking benchmark
python tests/bench_rerank.py --lib build/libcrispembed.so \
    --embed-gguf model.gguf --reranker-gguf reranker.gguf

# BidirLM-Omni text/audio/raw-vision benchmark with cosine regression check
PYTHONPATH=python python tests/benchmark_bidirlm.py \
    --model "$CRISPEMBED_MODEL" \
    --lib "$CRISPEMBED_LIB" \
    --save-baseline "$CRISPEMBED_BIDIRLM_BASELINE"
PYTHONPATH=python python tests/benchmark_bidirlm.py \
    --model "$CRISPEMBED_MODEL" \
    --lib "$CRISPEMBED_LIB" \
    --compare-baseline "$CRISPEMBED_BIDIRLM_BASELINE"

Compares CrispEmbed (CLI, Python ctypes, HTTP server) against HuggingFace sentence-transformers, FastEmbed (ONNX), and fastembed-rs (Rust ONNX). Auto-creates a .bench-venv for Python dependencies.

The model type is auto-detected from GGUF metadata at load time:

• Encoder models (BERT/XLM-R/MPNet/NomicBERT/ModernBERT/GTE-v1.5/DeBERTa-v2/SPLADE) → src/crispembed.cpp → encode_tokens() / encode_tokens_batch(). Encoder variants auto-detect from tensor names: no pos_embd ⇒ RoPE (NomicBERT/ModernBERT/GTE-v1.5), rel_attn_bias ⇒ relative position bias (MPNet), pre_ln ⇒ pre-LayerNorm (ModernBERT/GTE-v1.5), ffn_up_gate ⇒ fused ggml_geglu.
• Decoder models (Qwen3/Gemma3/BidirLM-Omni text and image-conditioned text) → src/decoder_embed.cpp → decoder_encode_tokens(). Detection heuristic: presence of blk.0.ffn_gate ⇒ decoder path.
• Vision (BidirLM-Omni) → src/bidirlm_vision.cpp, opens lazily on the first crispembed_encode_image* call when visual.* tensors are present.
• Audio (BidirLM-Omni) → src/bidirlm_audio.cpp wrapping the shared crisp_audio library, opens lazily on the first crispembed_encode_audio call.

Tokenizer dispatch reads tokenizer.ggml.type: 0=WordPiece, 1=BPE, 2=SentencePiece. Heuristic fallback: vocab > 100K ⇒ SentencePiece.

Server (examples/server/server.cpp) exposes text embedding (4 API dialects), face detection/recognition, ViT/CLIP vision, math OCR, and NER:

• POST /embed — native {"texts": [...]}
• POST /v1/embeddings — OpenAI-compatible
• POST /api/embed — Ollama batch
• POST /api/embeddings — Ollama legacy single
• POST /detect, POST /face — face detection/recognition
• POST /vit/encode, POST /clip/text — image/text encoding
• POST /math/ocr — formula recognition {"image": "path"} → {"latex": "..."}
• POST /ner/extract — NER {"text": "...", "labels": [...]} → {"entities": [...]}

BERT encoder (all-MiniLM, gte, arctic-embed-xs, paraphrase-multilingual-MiniLM-L12-v2):

• Token + Position + Type embeddings → Post-LN transformer → Mean/CLS pooling
• Tokenizer is WordPiece by default; model_type=bert with vocab > 100K (paraphrase-multilingual, multilingual-e5-small) loads the XLM-R SentencePiece-Unigram vocab via Viterbi DP, still with pos_offset=0.

XLM-R encoder (PIXIE-Rune, multilingual-e5-base/large, arctic-embed-l-v2):

• Token + Position(+offset) embeddings → Post-LN transformer → CLS/Mean pooling
• SentencePiece Unigram tokenizer (Viterbi DP), pos_offset=2, model_type=xlm-roberta

BGE-M3 multi-modal (BAAI/bge-m3):

• Same BERT encoder trunk with three output heads:
  • Dense: mean-pool → L2 normalize → float[1024]
  • Sparse: Linear(H,1) + ReLU → scatter via input_ids → {token_id: weight}
  • ColBERT: Linear(H,128) → per-token L2 normalize → float[n_tokens][128]

MPNet encoder (all-mpnet-base-v2):

• Token + Position(+offset) embeddings → Post-LN transformer with relative position bias → Mean pooling
• T5-style logarithmic bucket relative attention bias (32 buckets × n_heads)

NomicBERT encoder (nomic-embed-text-v1.5):

• Token embeddings (no position) + RoPE → Post-LN transformer + SwiGLU FFN → Mean pooling
• Rotary position embeddings (same as decoder path), no absolute position embeddings

NomicBERT MoE encoder (nomic-embed-text-v2-moe):

• Token + Type embeddings + emb_ln + RoPE → Post-LN transformer with mixed MoE/dense FFN → Mean pooling
• 8 experts, top-2 routing, GELU activation; MoE on odd layers, dense GELU FFN on even layers
• Fully in-graph routing: ggml_top_k + ggml_get_rows + ggml_mul_mat_id (no CPU-side partial compute)

Cross-encoder reranker (BGE-reranker-v2-m3, ms-marco-MiniLM, mxbai-rerank, etc.):

• [CLS] query [SEP] document [SEP] pair tokenization → CLS hidden state → Linear(H,1) → scalar score

Qwen3 decoder (Octen, F2LLM, Jina v5, Harrier-0.6B, Qwen3-Embed):

• Token embeddings + RoPE → RMSNorm + GQA with causal mask + SwiGLU → Last-token pooling

Gemma3 decoder (Harrier-270M):

• Token embeddings * sqrt(H) + RoPE → Gemma3 RMSNorm(1+w) + GQA + GeGLU → Last-token pooling

BidirLM-Omni (BidirLM-Omni-2.5B-Embedding) — text + audio + image, shared 2048-d space:

• Text: bidirectional Qwen3 body (RoPE, GQA, RMSNorm, q_norm/k_norm, SwiGLU) → Mean pooling → 2048-d.
• Audio (when CrispAudio is available): Whisper-shape encoder (Conv2D stem + 24-layer pre-LN encoder + proj1/GELU/proj2) → Mean pooling → same 2048-d shared space. Built on the shared crisp_audio library from the configured CrispASR checkout (CMake auto-discovers it; override with -DCRISP_AUDIO_DIR=…).
• Vision: BidirLM/Qwen2VL-style vision tower in ggml (patch embed, interpolated position embedding, rotary attention, patch merger, DeepStack hooks at layers 8/16/24). Two preprocessing paths: the Python binding's encode_image(image) uses HF Qwen2VLImageProcessorFast (byte-tight HF parity, requires transformers); crispembed -m … --image FILE and model.encode_image_file(path) use CrispEmbed's in-process C++ preprocessor (smart_resize + Catmull-Rom bicubic with antialias + OpenAI CLIP normalize + Qwen2VL patchify, via stb_image) — no transformers runtime dep, cosine ≈ 0.97 vs HF on photographs (the gap is JPEG decoder differences PIL/libjpeg-turbo vs stb).
• Image-conditioned text: crispembed_encode_text_with_image() runs the vision tower, splices image_embeds into inputs_embeds at every image_token_id placeholder, adds deepstack[k] features at the first 3 decoder layers, and uses 3D interleaved-MRoPE position ids derived from grid_thw. A lower-level crispembed_encode_with_image_ids() accepts pre-tokenized ids for clean parity with external tokenizers. See tests/test_bidirlm_image_text.py for the parity test against HF BidirLMOmniModel.forward(input_ids, pixel_values, image_grid_thw) (cosine ≥ 0.99).
• Pooled-only image path: crispembed_encode_image() skips DeepStack materialization since the mean-pooled image vector doesn't use it; encode_image_raw and encode_text_with_image keep DeepStack on.
• Decoder text/text+image embedding both run through ggml_backend_sched, matching the encoder and vision execution paths.

Cache convention: point CRISPEMBED_CACHE_DIR at your backing store to keep large GGUF/cache files out of the repo tree (default: ~/.cache/crispembed/).

All via ggml graphs with GPU dispatch (ggml_backend_sched). See PLAN.md for architecture and roadmap, HISTORY.md for completed milestones, LEARNINGS.md for technical detail (3D MRoPE workaround, DeepStack splice via mask+add, decoder scheduler init), and PERFORMANCE.md for benchmarks.

• ggml -- inference engine
• CrispASR -- shared core (gguf_loader, bpe.h)
• sentence-transformers -- ground-truth validation