• 🚀 Native Apple Silicon — Optimized for M1/M2/M3/M4 chips using MLX
• 📝 Text Prompts — Segment objects by describing them ("car", "person", "dog")
• 📦 Box Prompts — Draw bounding boxes to include or exclude regions
• 🎨 Interactive Studio — Beautiful web interface for real-time segmentation
• 🐍 Python API — Simple programmatic access for scripting and integration
• ⬇️ Auto Model Download — Weights automatically fetched from HuggingFace

Object detection with "car" prompt	Semantic segmentation with "coat" prompt

SAM3 Studio — Interactive segmentation with text and box prompts

⚠️ Apple Silicon Only: This project uses MLX, Apple's machine learning framework optimized exclusively for Apple Silicon.

If you have uv installed:

# Clone the repository
git clone https://github.com/Deekshith-Dade/mlx-sam3.git
cd mlx-sam3

# Install project dependencies
uv sync

# Launch the app (backend + frontend)
cd app && ./run.sh

The first run will automatically download MLX weights from mlx-community/sam3-image (~3.5GB).

Access the app:

• 🌐 Frontend: http://localhost:3000
• 🔌 API: http://localhost:8000
• 📚 API Docs: http://localhost:8000/docs

Press Ctrl+C to stop all servers.

# Clone the repository
git clone https://github.com/your-username/mlx-sam3.git
cd mlx-sam3

# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install the package
pip install -e .

cd app/backend
pip install -r requirements.txt
python main.py

cd app/frontend
npm install
npm run dev

Use SAM3 directly in your Python scripts:

from PIL import Image
from sam3 import build_sam3_image_model
from sam3.model.sam3_image_processor import Sam3Processor

# Load model (auto-downloads MLX weights from mlx-community/sam3-image)
model = build_sam3_image_model()
processor = Sam3Processor(model, confidence_threshold=0.5)

# Load and process an image
image = Image.open("your_image.jpg")
state = processor.set_image(image)

# Segment with text prompt
state = processor.set_text_prompt("person", state)

# Access results
masks = state["masks"]       # Binary segmentation masks
boxes = state["boxes"]       # Bounding boxes [x0, y0, x1, y1]
scores = state["scores"]     # Confidence scores

print(f"Found {len(scores)} objects")

# Add a box prompt (normalized coordinates: center_x, center_y, width, height)
# label=True for inclusion, label=False for exclusion
state = processor.add_geometric_prompt(
    box=[0.5, 0.5, 0.3, 0.3],  # Center of image, 30% width/height
    label=True,
    state=state
)

# Clear all prompts while keeping the image
processor.reset_all_prompts(state)

# Try a different prompt
state = processor.set_text_prompt("car", state)

mlx-sam3/
├── sam3/                    # Core MLX SAM3 implementation
│   ├── model/               # Model components
│   │   ├── sam3_image.py    # Main model architecture
│   │   ├── vitdet.py        # Vision Transformer backbone
│   │   ├── text_encoder_ve.py # Text encoder
│   │   └── ...
│   ├── model_builder.py     # Model construction utilities
│   ├── convert.py           # Weight conversion from PyTorch
│   └── utils.py             # Helper utilities
├── app/                     # Web application
│   ├── backend/             # FastAPI server
│   ├── frontend/            # Next.js React app
│   └── run.sh               # One-command launcher
├── assets/                  # Static assets & test images
├── examples/                # Jupyter notebook examples
└── pyproject.toml           # Project configuration

# Upload an image
curl -X POST "http://localhost:8000/upload" \
  -F "file=@your_image.jpg"

# Response: {"session_id": "abc-123", "width": 1920, "height": 1080, ...}

# Segment with text
curl -X POST "http://localhost:8000/segment/text" \
  -H "Content-Type: application/json" \
  -d '{"session_id": "abc-123", "prompt": "car"}'

Jupyter notebooks are available in the examples/ directory:

• sam3_image_predictor_example.ipynb — Basic image segmentation
• sam3_image_interactive.ipynb — Interactive prompting workflows

Run them with:

cd examples
jupyter notebook

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

This project is licensed under the Apache 2.0 License — see the LICENSE file for details.

• Meta AI for the original SAM3 model
• MLX Team at Apple for the incredible ML framework
• The open-source community for continuous inspiration