Summary

This PR implements Memory Compression and Deduplication Optimization for PowerMem as requested in issue #141.

Changes

Core Features

Deduplication (deduplicate())

• Detect duplicate memories using semantic similarity (embedding-based)
• Configurable similarity threshold (default: 0.95)
• Intelligent merge: keep most important memory, combine metadata
• Dry-run mode to preview changes without applying
• Detailed report of operations performed

Compression (compress())

• Consolidate similar memories using LLM summarization
• Three compression strategies:
  • conservative: Keep original wording, remove only obvious duplicates
  • moderate: Combine similar points, remove redundancy
  • aggressive: Very concise summary, keep only essential information
• Preserve key information while reducing redundancy
• Maintain memory relationships and metadata

Optimization (optimize())

• Convenience wrapper for deduplication and compression
• Strategy options: deduplicate, compress, all
• Unified interface for memory optimization

API Examples

# Deduplication
results = memory.deduplicate(
    user_id="user123",
    threshold=0.95,  # Similarity threshold
    dry_run=False,   # Apply changes
)
# Returns: {"duplicates_found": 10, "merged": 10, "deleted": 10, "saved_space": "5.00 KB"}

# Compression
results = memory.compress(
    user_id="user123",
    strategy="conservative",  # conservative, moderate, aggressive
    dry_run=False,
)
# Returns: {"memories_analyzed": 20, "compressed": 5, "original_count": 20, "compressed_count": 15}

# Combined optimization
results = memory.optimize(
    user_id="user123",
    strategy="all",
    threshold=0.95,
)

Technical Implementation

• Embedding-based detection: Uses batch embedding for efficient similarity computation
• Cosine similarity: Calculates similarity between memory embeddings
• LLM-powered compression: Uses LLM to create intelligent summaries
• Safety features:
  • Backup metadata before merging
  • Preserve important information
  • Detailed error handling
  • Progress logging

Testing

• Added comprehensive unit tests in tests/unit/test_memory_optimization.py
• Tests cover:
  • Empty memory handling
  • Duplicate detection
  • Different compression strategies
  • Dry-run mode
  • Embedding similarity calculation
  • Edge cases

Documentation

• Clear docstrings for all new methods
• Usage examples in code comments
• Strategy parameter explanations

Related Issues

• Primary: [Feature]: Memory Compression and Deduplication Optimization #141
• Context: Developer Activities: Call for Participation! seekdb#123 (OceanBase Developer Challenge)

Closes #141