Run diffusion models directly within Godot.

This project is made with Godot 4.3 and uses the GDExtension plugin py4godot for running Python.

Clone this repository, and then install py4godot 1.0.0-alpha2 into /addons/ in the root folder (see the README in the py4godot repository for more info).

To install Python packages, follow the setup instructions from py4godot here: https://github.com/niklas2902/py4godot/wiki/Installing-packages-for-python

You should have pip installed in addons\py4godot\cpython-3.12.4-windows64\python

In this directory, run the following to set up a virtual environment for Python:

python -m venv .venv

And then activate the environment:

.venv\Scripts\activate

Then install the Diffusers library with the following command:

python -m pip install diffusers["torch"] transformers

And to set up CUDA for PyTorch, run the following:

python -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124

To set up the Palettize tool, install the following packages:

python -m pip install git+https://www.github.com/hbldh/hitherdither
python -m pip install opencv-python

To generate images, run the main scene, and then select a .safetensors model to load and press "Generate".

The Palettize tool by Astropulse has been edited to work with this project's setup. It works with any model, but check out the standalone Retro Diffusion Model to see high quality pixel art results.

The scene Main.tscn is essentially an interface for running diffusion models with Diffusers. It uses a helper node with python_helper.py handling the connections between the UI and Python.

The Palettize tool in scripts/palettize.py has a function run_palettize() that the main scene runs if the tool is enabled by the user.