GhostPlayer is an AI game-agent prototype that watches your game, reasons about what it sees, and performs actions for repetitive tasks in-game (e.g. mining, farming coins, spawn-killing enemies, etc.).

It is the first flagship product in a larger startup vision: Phantom.one (Phantom1), aimed at building the first general-purpose game agent platform.

Automation. Coaching. Competing. QA testing for game studios.

I'm building the platform that all game agents will eventually run on.

The goal for this hackathon build is simple:

• Optimize for one primary demo game for now
• give it a natural-language objective
• let it play while you keep your hands off the keyboard
• show clear, human-readable logs in a local control panel

• Uses Gemini vision to understand the game screen
• Uses Backboard to persist useful run memory
• Uses pyautogui to control mouse/keyboard
• Provides a local Mission Control web app for start/pause/resume/stop and logs

pip install -r requirements.txt

Create a .env file in the repo root:

GEMINI_API_KEY=your_gemini_key_here
BACKBOARD_API_KEY=your_backboard_key_here
GUI_PORT=5050

python3 main.py

Open:

• http://localhost:5050

That page is Mission Control: set a goal, start the run, and monitor logs. For now, once you click "start", you have 6 seconds to place yourself in your game before the agent takes over keyboard and mouse inputs.

Before first run, grant permissions to the app/terminal running Python:

• Accessibility (for keyboard/mouse control)
• Screen Recording (for frame capture)
• Safety failsafe: moving your cursor to the top-left corner triggers pyautogui failsafe and stops active input automation.
• Emergency Pause & Safe hotkey: Ctrl + Option + Cmd + P

If controls do not work:

• Open System Settings -> Privacy & Security -> Accessibility, enable your terminal app, and restart it.
• Open System Settings -> Privacy & Security -> Screen Recording, enable your terminal app, and restart it.

To keep Gemini focused on the game viewport only, set a capture region.

Run:

python -m ghostplayer.core.region_selector --interactive-validate-point

This helper will:

• ask you to click top-left and bottom-right corners of your game
• print a CAPTURE_REGION_JSON value you can paste into .env
• save capture_region_preview.png so you can verify framing
• ask for one validation click (for example inside Mission Control) and confirm it is excluded

Then add the printed JSON to .env:

CAPTURE_REGION_JSON={"left":200,"top":120,"width":1280,"height":720}

Mouse sensitivity and camera turn behavior must stay consistent between test runs.

python -m ghostplayer.core.calibration_helper --degrees 90 --repeats 1

If the turn amount is off, adjust in .env:

PIXELS_PER_DEGREE=6.67

• Keep mouse sensitivity fixed while calibrating and demoing.
• Keep game resolution and FOV fixed after calibration.
• Keep frame rate stable and avoid heavy background load.
• Re-run calibration if display scaling, sensitivity, or window size changes.

• project_details.md — detailed architecture and behavior
• project_plan.md — step-by-step implementation plan

:)