Spectra is a systematic RAG evaluation toolkit that implements 12 retrieval strategies spanning dense, sparse, hybrid, generative, iterative, and graph-based approaches. It provides automated A/B testing with statistical rigor (Welch's t-test, Mann-Whitney U, bootstrap), 8 quality metrics aligned with the RAGAS framework, and Pareto-optimal pipeline selection powered by Optuna -- everything needed to move from prototype to production RAG.

• 12 Retrieval Strategies -- Dense, BM25, SPLADE, Hybrid RRF, HyDE, Self-RAG, ColBERT, Multi-hop, Graph-RAG, IRCoT, Chain-of-Note, Cross-encoder Reranker
• 8 Evaluation Metrics -- Faithfulness, Answer Relevance, Context Relevance, Coherence, Context Recall, Context Precision, Answer Correctness, Latency Score
• Statistical A/B Testing -- Welch's t-test, Mann-Whitney U, paired t-test, bootstrap permutation with Bonferroni correction
• Optuna-Powered Optimization -- Automated hyperparameter search across chunking, embedding, retrieval, and re-ranking stages
• Pareto Frontier Analysis -- Multi-objective selection with NSGA-II crowding distance for quality vs. latency trade-offs
• 4 Chunking Strategies -- Fixed, Semantic, Recursive, and Document-Aware splitting
• 3 Vector Store Backends -- FAISS, Qdrant, pgvector
• Streamlit Dashboard -- Interactive visualization of metrics, comparisons, and optimization results
• Synthetic Data Generation -- Automatic evaluation dataset creation for rapid iteration

graph LR
    A[Documents] --> B[Chunking]
    B --> C[Embedding]
    C --> D[VectorStore]
    D --> E[Retrieval]
    E --> F[Evaluation]
    F --> G[Optimization]

    subgraph Chunking
        B1[Fixed]
        B2[Semantic]
        B3[Recursive]
        B4[Document-Aware]
    end

    subgraph VectorStore
        D1[FAISS]
        D2[Qdrant]
        D3[pgvector]
    end

    subgraph "Retrieval Strategies x12"
        E1[Dense / Sparse / Hybrid]
        E2[HyDE / Self-RAG / ColBERT]
        E3[Multi-hop / Graph-RAG / IRCoT]
        E4[Chain-of-Note / SPLADE / Reranker]
    end

    subgraph Evaluation
        F1[8 Quality Metrics]
        F2[A/B Testing]
        F3[Synthetic Data]
    end

    subgraph Optimization
        G1[Optuna HPO]
        G2[Pareto Frontier]
    end

    B --> B1 & B2 & B3 & B4
    D --> D1 & D2 & D3
    E --> E1 & E2 & E3 & E4
    F --> F1 & F2 & F3
    G --> G1 & G2

# Core
pip install spectra-rag

# With all vector store backends + dashboard
pip install spectra-rag[all]

# Development
pip install spectra-rag[dev]

from spectra.retrieval.base import Document
from spectra.retrieval.dense import DenseRetriever
from spectra.optimization.pipeline import RAGPipeline, PipelineConfig

docs = [
    Document(id="1", content="RAG combines retrieval with generation..."),
    Document(id="2", content="Dense retrieval uses learned embeddings..."),
]

pipeline = RAGPipeline(
    DenseRetriever(),
    PipelineConfig(chunk_size=256, top_k=3),
)
pipeline.ingest(docs)

result = pipeline.query("What is RAG?")
for doc, score in zip(result.retrieved_documents, result.scores):
    print(f"[{score:.3f}] {doc.content[:80]}...")

from spectra.evaluation.metrics import EvaluationSample, compute_all_metrics

sample = EvaluationSample(
    query="What is RAG?",
    answer=result.retrieved_documents[0].content,
    contexts=[d.content for d in result.retrieved_documents],
    ground_truth="RAG combines retrieval with generation.",
    latency_seconds=result.latency_seconds,
)
metrics = compute_all_metrics([sample])
print(metrics.scores)

from spectra.evaluation.ab_testing import ABTest, ABTestConfig

ab = ABTest(ABTestConfig(significance_level=0.05))
results = ab.compare(samples_dense, samples_hybrid, "Dense", "Hybrid")

for r in results:
    print(f"{r.metric}: p={r.p_value:.4f}, winner={r.winner}")

from spectra.optimization.optimizer import PipelineOptimizer, OptimizerConfig

optimizer = PipelineOptimizer(
    documents=docs,
    eval_queries=queries,
    eval_ground_truths=ground_truths,
    config=OptimizerConfig(n_trials=100),
)
result = optimizer.optimize()
print(f"Best config: {result.best_config}")
print(f"Best score:  {result.best_score:.4f}")

spectra
# or
streamlit run src/spectra/dashboard/app.py

All metrics return scores in [0, 1]:

Benchmark results on a synthetic corpus of 1,000 documents (avg 500 tokens):

HyDE, Self-RAG, IRCoT, and Chain-of-Note latencies are dominated by LLM calls. Actual throughput depends on the LLM provider and concurrency settings.

Reproduce locally:

python benchmarks/strategy_benchmark.py

spectra/
├── src/spectra/
│   ├── retrieval/          # 12 retrieval strategy implementations
│   ├── chunking/           # Fixed, Semantic, Recursive, Document-Aware
│   ├── evaluation/         # 8 metrics, A/B testing, RAGAS, synthetic data
│   ├── optimization/       # Optuna optimizer, Pareto frontier analysis
│   ├── vectorstores/       # FAISS, Qdrant, pgvector backends
│   ├── dashboard/          # Streamlit interactive UI
│   └── utils/              # Embedding models, LLM client
├── examples/               # quickstart, compare_strategies, optimize_pipeline
├── tests/                  # Unit & integration tests
├── benchmarks/             # Performance benchmark harness
└── pyproject.toml

Contributions are welcome. Please open an issue or submit a pull request.

git clone https://github.com/JiwaniZakir/spectra.git
cd spectra
pip install -e ".[dev]"
pytest tests/ -v

Released under the Apache License 2.0.