FactoryMind R2D2 is a research and demo stack for orchestrating general-purpose autonomous robots in a factory or warehouse setting. It combines a real-time digital twin simulation, path planning, and an optional AI leader (NVIDIA Nemotron via NIM) that proposes work assignments—while NemoClaw-style policies define what the leader and workers are allowed to do.

The project makes concrete a problem many operations teams face: how to coordinate many mobile agents safely under one “brain,” with guardrails, without each robot running unconstrained logic. Here, that brain is explicit, inspectable, and policy-bound.

Modern warehouses and plants mix fixed automation (conveyors, AS/RS) with mobile platforms (AMRs, cobot arms on carts, human-assisted fleets). Coordinating them usually requires:

• A single source of truth for floor state, orders, and robot status
• Safe delegation: who may assign motion, pick, place, or change equipment
• Predictable fallback when the cloud, GPU host, or model endpoint is slow or down
• Something you can demo, replay, and explain to stakeholders—not only a slide deck

FactoryMind R2D2 is a compact simulation that stresses exactly those ideas: multi-agent dispatch, constraints, and optional LLM reasoning.

When you press START in the web UI:

1. Factories produce colored packages onto conveyor pads.
2. The leader periodically plans which idle worker should serve which package and which matching-color drop box.
3. Workers pathfind, pick up only at pads, drop only at matching boxes, and report completion.
4. If NIM is unreachable, the deterministic planner keeps dispatch alive; Nemotron is an enhancement, not a single point of failure for the loop.

You can also build the layout (walls, factories, drop boxes, workers) from the browser and experiment with connectivity and traffic.

pip3 install -r requirements.txt
cp .env.example .env
# Edit .env: API keys for cloud mode; GX10_IP / NIM URLs for local NIM.

python3 -m factorymind.web_main

Open http://localhost:8080 (or your host/port if overridden). Use START to run the autonomous loop.

Convenience:

./scripts/run_web.sh

To log launcher context and optionally use a sandbox (OpenShell / firejail) per scripts/run_with_nemoclaw.sh:

./scripts/run_with_nemoclaw.sh

The app still loads scripts/nemoclaw_policy.yaml and writes policy-aligned lines to logs/nemoclaw.log.

python3 -m factorymind.warehouse_main

Useful for verifying stepping logic and metrics without the browser.

Full spec: scripts/nemoclaw_policy.yaml.

• Leader may observe and post worker directives; it may not move workers directly, modify factories/drop boxes, or pick/drop packages.
• Workers may move inside the sim, pick from conveyor pads, drop at matching-color boxes, and report complete; they may not leave the area, cheat pickups/drops, or edit policy.
• Network and filesystem sections document intended sandbox allowances when you use run_with_nemoclaw.sh; on a plain host run, enforcement is primarily application-level plus logging.

factorymind/
  web_main.py       # Default entry: sim loop + HTTP snapshot + drain browser actions
  main.py           # World reset, leader plan, workers, pathfinding, policy hooks
  web_server.py     # Tiny GET/POST server for /state.json and /action
  web/              # index.html, Three.js scene, static JS (layout parser, etc.)
  inference.py      # Nemotron / NIM OpenAI-compatible client
  state.py          # World schema helpers
  agents.py         # In-process Blackboard for messages
  warehouse.py      # Alternate warehouse sim backend
  warehouse_main.py # CLI runner for warehouse backend
  memory.py         # Optional memory / persistence helpers
scripts/
  nemoclaw_policy.yaml
  run_web.sh
  run_with_nemoclaw.sh
  run_nim_*.sh, verify_*.sh   # NIM / tunnel helpers
tests/
  pytest tests for inference, smoke paths, warehouse

python3 -m pytest tests/