Skip to content

IconForge: DMI Generation, Code Reorganization, Improved Caching, Cleaner I/O#213

Merged
ZeWaka merged 14 commits intotgstation:masterfrom
itsmeowForks:iconforge-spritesheet-dmi
Jul 4, 2025
Merged

IconForge: DMI Generation, Code Reorganization, Improved Caching, Cleaner I/O#213
ZeWaka merged 14 commits intotgstation:masterfrom
itsmeowForks:iconforge-spritesheet-dmi

Conversation

@itsmeow
Copy link
Copy Markdown
Contributor

@itsmeow itsmeow commented May 1, 2025

What This Does

  • Adds the ability for generate_spritesheet to output DMIs
  • Adds support for multi-dir & animated/multi-frame icons to the new DMI spritesheet generator
  • Improves the caching system by putting the image data into an Arc
  • Removes the old "hash_input" system and I/O conversions which were wasteful
  • Splits the code out into separate files because it was getting messy
  • Renames several functions to be more descriptive
  • Moves a few functions to impls on existing structs to reduce the amount of global functions doing type conversions
  • Slightly more verbose error message if there is an error during the final generation stage of spritesheets - it will now include the associated sprite_name that caused the error, rather than saying "N/A, in final generation stage".

Breaking DM change: add , FALSE, TRUE argument to the end of existing calls to rustg_iconforge_generate() or rustg_iconforge_generate_async()

Breaking DM change: The default arguments of the commonly used uni_icon API need to be adjusted so that dir and frame are null

/datum/universal_icon/New(icon/icon_file, icon_state="", dir=null, frame=null, datum/icon_transformer/transform=null, color=null)

Arbitrary Icon Generation

It's now technically possible to generate just about any icon that the game can make (excluding certain special operations) with just the uni_icon system and rustg_iconforge_generate(), because it now supports multi-dir and animated icons. However, the main benefit of IconForge is still parallelization and caching.

I got this idea because it would make tgstation/tgstation#90940 run significantly faster with very little effort.

I tested this using the linked TG PR with the following method replacement:

/datum/controller/subsystem/processing/greyscale/proc/ExportMapPreviewsForType(filename, atom/atom_typepath, list/type_blacklist)
	var/list/handled_types = list()
	var/list/sprite_entries = list()
	for(var/atom/atom_type as anything in subtypesof(atom_typepath))
		if(type_blacklist && type_blacklist[atom_type])
			continue
		handled_types[atom_type] = TRUE
		var/greyscale_config = atom_type::greyscale_config
		var/greyscale_colors = atom_type::greyscale_colors
		if(!greyscale_config || !greyscale_colors || atom_type::flags_1 & NO_NEW_GAGS_PREVIEW_1)
			continue
		var/datum/universal_icon/map_icon = GetColoredIconByTypeUniversalIcon(greyscale_config, greyscale_colors, atom_type::post_init_icon_state)
		sprite_entries["[atom_type]"] = map_icon.to_list()

	// Skip empty sheets
	if(!length(sprite_entries))
		return handled_types

	var/entries_json = json_encode(sprite_entries)
	var/data_out = rustg_iconforge_generate("icons/map_icons/", filename, entries_json, FALSE, TRUE)
	if (data_out == RUSTG_JOB_ERROR)
		CRASH("GAGS-MapIcon/Spritesheet [filename] JOB PANIC")
	else if(!findtext(data_out, "{", 1, 2))
		rustg_file_write(entries_json, "[GLOB.log_directory]/spritesheet_gags_mapicon_debug_[replacetext(filename, "/", "-")].json")
		CRASH("GAGS-MapIcon/Spritesheet [filename] UNKNOWN ERROR: [data_out]")
	var/data = json_decode(data_out)
	var/list/sizes = data["sizes"]
	var/list/sprites = data["sprites"]
	if(data["error"])
		CRASH("Error during GAGS-MapIcon/spritesheet generation for [filename]: [data["error"]]")
	for(var/size_id in sizes)
		if(size_id == "32x32")
			continue
		fdel("icons/map_icons/[filename]_[size_id].dmi")
		var/list/sprites_with_bad_size = list()
		for(var/sprite_name in sprites)
			var/list/sprite = sprites[sprite_name]
			if(sprite["size_id"] == "32x32")
				continue
			sprites_with_bad_size.Add(sprite_name)
		var/bad_sprites = jointext(sprites_with_bad_size, " \n")
		CRASH("Non 32x32 icons found in GAGS-MapIcon/spritesheet [filename] for typepaths: \n[bad_sprites]")
	var/output_filepath = "icons/map_icons/[filename]_32x32.dmi"
	var/filepath = "icons/map_icons/[filename].dmi"
#ifdef UNIT_TESTS
	var/old_md5 = rustg_hash_file(RUSTG_HASH_MD5, filepath)
#endif
	fcopy(output_filepath, filepath)
#ifdef UNIT_TESTS
	var/new_md5 = rustg_hash_file(RUSTG_HASH_MD5, filepath)
	if(old_md5 != new_md5)
		stack_trace("Generated map icons were different than what is currently saved. If you see this in a CI run it means you need to run the game once through initialization and commit the resulting files in 'icons/map_icons/'")
#endif
	fdel(output_filepath)
	return handled_types

The output was exactly as expected.

image
image
image

I did some tracy profiling, and it's actually faster now, due to changes in how caching and I/O works

image

@itsmeow itsmeow changed the title Add support for DMI output of spritesheets IconForge: Add support for DMI output of spritesheets May 1, 2025
@itsmeow itsmeow changed the title IconForge: Add support for DMI output of spritesheets IconForge: Arbitrary Icon Creation Update, Improved Caching, Cleaner I/O May 9, 2025
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates the IconForge system by adding arbitrary icon generation with support for DMI output, improved caching using Arc and DashMap, and cleaner I/O handling. Key changes include new Rust modules for spritesheet generation, image caching, and blending, along with updated DM macros and documentation to support the new DMI and flatten parameters.

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/iconforge/spritesheet.rs Introduces new structures for spritesheet results with DMI support.
src/iconforge/image_cache.rs Implements enhanced caching for UniversalIcon data and DMI file parsing.
src/iconforge/byond.rs Adds DM bindings for synchronous and asynchronous icon generation functions.
src/iconforge/blending.rs Provides blending operations with a variety of blend modes for RGBA colors.
dmsrc/iconforge.dm Updates macros and documentation to include new generate_dmi and flatten flags.
Comments suppressed due to low confidence (1)

dmsrc/iconforge.dm:6

  • Typo in documentation: 'heplful' should be corrected to 'helpful'.
/// These hashes can be heplful for 'smart' caching (see rustg_iconforge_cache_valid), but require extra computation.

@itsmeow itsmeow changed the title IconForge: Arbitrary Icon Creation Update, Improved Caching, Cleaner I/O IconForge: DMI Generation, Code Reorganization, Improved Caching, Cleaner I/O Jun 21, 2025
@ZeWaka ZeWaka merged commit 787dffe into tgstation:master Jul 4, 2025
4 checks passed
itsmeow added a commit to BeeStation/rust-g that referenced this pull request Feb 26, 2026
itsmeow added a commit to BeeStation/rust-g that referenced this pull request Feb 26, 2026
* fixes the clippy lints on nightly (tgstation#215)

* Adds a new time function, `formatted_timestamp` (tgstation#214)

* dmi_create_png is rgba instead of rgb (tgstation#217)

* add useragent to byond installer

* crate internal updates (tgstation#218)

* v3.9.0 (tgstation#219)

* Enable pathfinder by default (tgstation#220)

* Fix clippy URL lifetime lint (tgstation#223)

* Improve DM test functionality on Windows, fix regressions (tgstation#221)

* Adds `roll_dice`, an advanced xdy dice roller. (tgstation#216)

Co-authored-by: Kapu1178 <75460809+Kapu1178@users.noreply.github.com>

* v3.11.0 (tgstation#224)

* Fix TOML dme test failing (tgstation#227)

* hash: optimize file hashing (tgstation#228)

* IconForge: DMI Generation, Code Reorganization, Improved Caching, Cleaner I/O (tgstation#213)

* Adds a new `uuid` module (tgstation#229)

* IconForge: BYOND Parity + Tests, Optimizations, New Transforms (tgstation#230)

* `clippy` lints for tgstation#230 (tgstation#233)

* Adds `http_request_fire_and_forget` (tgstation#232)

* DMI metadata reading and injection (tgstation#234)

* 4.0.0 (tgstation#235)

* dmi: Add QR code generation (tgstation#226)

* Bump `dmi` to 0.5.0, optimize `dmi_read_metadata` (tgstation#238)

* Update CI to target 516.1666 (tgstation#237)

* 4.1.0 (tgstation#239)

* makes `rustg_noise_poisson_map` around 8x faster (tgstation#240)

* IconForge: Headless Icon Generation (tgstation#236)

* `cargo update` & `cargo upgrade` & `png` fix (tgstation#241)

* assorted optimizations to `cellularnoise`, `dbpnoise`, and `worleynoise` (tgstation#243)

* v4.2.0 (tgstation#242)

* Fix attempt for CI

* hash: Adds ChaCha20 CSPRNG functions, updates TOTP generator, adds Base32 (tgstation#225)

* Handle errors in panic hook explicitly (tgstation#245)

* do not store Git information static (tgstation#247)

* Prevent `decode_base64` and `decode_base32` from panicking if given invalid base64 (tgstation#244)

* 4.3.0 (tgstation#249)

* fix iconforge generate_headless on win for bad paths not panicing (tgstation#248)

* 5.0.0 (tgstation#250)

* redo iconforge errors - 5.0.1 (tgstation#252)

* 6.0.0 - drop windows 7 support (tgstation#251)

* fix spritesheet gen on linux (tgstation#255)

* cargo update and clippy (tgstation#253)

* 6.0.1 - and `cargo update` (tgstation#256)

---------

Co-authored-by: Lucy <lucy@absolucy.moe>
Co-authored-by: TiviPlus <57223640+TiviPlus@users.noreply.github.com>
Co-authored-by: ZeWaka <zewakagamer@gmail.com>
Co-authored-by: Mothblocks <35135081+Mothblocks@users.noreply.github.com>
Co-authored-by: Comrade Niobe <126028983+ComradeNiobe@users.noreply.github.com>
Co-authored-by: Kapu1178 <75460809+Kapu1178@users.noreply.github.com>
Co-authored-by: Y0SH1M4S73R <y0sh1m4s73r@gmail.com>
Co-authored-by: Ivy <distributivgesetz93@gmail.com>
Co-authored-by: Zephyr <12817816+ZephyrTFA@users.noreply.github.com>
Co-authored-by: Kashargul <144968721+Kashargul@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants