Skip to content

hash: optimize file hashing#228

Merged
ZeWaka merged 2 commits intotgstation:masterfrom
Absolucy:file-hash-optimization
Jul 4, 2025
Merged

hash: optimize file hashing#228
ZeWaka merged 2 commits intotgstation:masterfrom
Absolucy:file-hash-optimization

Conversation

@Absolucy
Copy link
Copy Markdown
Member

@Absolucy Absolucy commented Jul 3, 2025

This refactors the hash module slightly, and optimizes the rustg_hash_file function

  • I've added a HashDispatcher enum, replacing the hash_algorithm function, so that we can use update() to update the hash with portions of a file easily
    • base64 is handled by just using a buffer, and encoding the entire buffer at the finish
  • file_hash no longer reads and hashes the entire file at once, instead it reads it in chunks up to 64 KB, updating the hasher with each chunk.
    • the 64 KB buffer is thread-local, instead of being re-allocated with every single file hash.

this should not be a breaking change at all, behavior from the DM side of things / function output should remain completely identical.

@Absolucy Absolucy changed the title Optimize rustg_hash_file hash: optimize file hashing Jul 3, 2025
@Absolucy Absolucy force-pushed the file-hash-optimization branch from 91b1da7 to 4da4df4 Compare July 3, 2025 04:23
@ZeWaka ZeWaka merged commit 8cb606e into tgstation:master Jul 4, 2025
4 checks passed
@Absolucy Absolucy deleted the file-hash-optimization branch July 4, 2025 18:14
Ghommie pushed a commit to tgstation/tgstation that referenced this pull request Jul 6, 2025
## About The Pull Request

This removes the `md5filepath` proc, which was only used in `md5asfile`.

Instead, `md5asfile` will just directly use `rustg_hash_file`, so we
avoid having to read the file that we *just* wrote to disk right back
into BYOND to MD5 it, and we can use rust-g to hash that file instead.

in theory, this should be even more performant if
tgstation/rust-g#228 is ~~merged~~ released

## Why It's Good For The Game

avoids an unneeded round-trip and hopefully improves performance.

## Changelog

no user-facing changes
TealSeer pushed a commit to Bubberstation/Bubberstation that referenced this pull request Jul 11, 2025
## About The Pull Request

This removes the `md5filepath` proc, which was only used in `md5asfile`.

Instead, `md5asfile` will just directly use `rustg_hash_file`, so we
avoid having to read the file that we *just* wrote to disk right back
into BYOND to MD5 it, and we can use rust-g to hash that file instead.

in theory, this should be even more performant if
tgstation/rust-g#228 is ~~merged~~ released

## Why It's Good For The Game

avoids an unneeded round-trip and hopefully improves performance.

## Changelog

no user-facing changes
TealSeer pushed a commit to effigy-se/effigy that referenced this pull request Aug 3, 2025
## About The Pull Request

This removes the `md5filepath` proc, which was only used in `md5asfile`.

Instead, `md5asfile` will just directly use `rustg_hash_file`, so we
avoid having to read the file that we *just* wrote to disk right back
into BYOND to MD5 it, and we can use rust-g to hash that file instead.

in theory, this should be even more performant if
tgstation/rust-g#228 is ~~merged~~ released

## Why It's Good For The Game

avoids an unneeded round-trip and hopefully improves performance.

## Changelog

no user-facing changes
Iajret pushed a commit to Iajret/LizardsPurgatory that referenced this pull request Aug 4, 2025
## About The Pull Request

This removes the `md5filepath` proc, which was only used in `md5asfile`.

Instead, `md5asfile` will just directly use `rustg_hash_file`, so we
avoid having to read the file that we *just* wrote to disk right back
into BYOND to MD5 it, and we can use rust-g to hash that file instead.

in theory, this should be even more performant if
tgstation/rust-g#228 is ~~merged~~ released

## Why It's Good For The Game

avoids an unneeded round-trip and hopefully improves performance.

## Changelog

no user-facing changes
Iajret pushed a commit to Iajret/LizardsPurgatory that referenced this pull request Aug 4, 2025
## About The Pull Request

This removes the `md5filepath` proc, which was only used in `md5asfile`.

Instead, `md5asfile` will just directly use `rustg_hash_file`, so we
avoid having to read the file that we *just* wrote to disk right back
into BYOND to MD5 it, and we can use rust-g to hash that file instead.

in theory, this should be even more performant if
tgstation/rust-g#228 is ~~merged~~ released

## Why It's Good For The Game

avoids an unneeded round-trip and hopefully improves performance.

## Changelog

no user-facing changes
Kocma-san pushed a commit to Fluffy-Frontier/FluffySTG that referenced this pull request Aug 13, 2025
## About The Pull Request

This removes the `md5filepath` proc, which was only used in `md5asfile`.

Instead, `md5asfile` will just directly use `rustg_hash_file`, so we
avoid having to read the file that we *just* wrote to disk right back
into BYOND to MD5 it, and we can use rust-g to hash that file instead.

in theory, this should be even more performant if
tgstation/rust-g#228 is ~~merged~~ released

## Why It's Good For The Game

avoids an unneeded round-trip and hopefully improves performance.

## Changelog

no user-facing changes
Kocma-san pushed a commit to Fluffy-Frontier/FluffySTG that referenced this pull request Aug 17, 2025
## About The Pull Request

This removes the `md5filepath` proc, which was only used in `md5asfile`.

Instead, `md5asfile` will just directly use `rustg_hash_file`, so we
avoid having to read the file that we *just* wrote to disk right back
into BYOND to MD5 it, and we can use rust-g to hash that file instead.

in theory, this should be even more performant if
tgstation/rust-g#228 is ~~merged~~ released

## Why It's Good For The Game

avoids an unneeded round-trip and hopefully improves performance.

## Changelog

no user-facing changes
itsmeow pushed a commit to BeeStation/rust-g that referenced this pull request Feb 26, 2026
itsmeow added a commit to BeeStation/rust-g that referenced this pull request Feb 26, 2026
* fixes the clippy lints on nightly (tgstation#215)

* Adds a new time function, `formatted_timestamp` (tgstation#214)

* dmi_create_png is rgba instead of rgb (tgstation#217)

* add useragent to byond installer

* crate internal updates (tgstation#218)

* v3.9.0 (tgstation#219)

* Enable pathfinder by default (tgstation#220)

* Fix clippy URL lifetime lint (tgstation#223)

* Improve DM test functionality on Windows, fix regressions (tgstation#221)

* Adds `roll_dice`, an advanced xdy dice roller. (tgstation#216)

Co-authored-by: Kapu1178 <75460809+Kapu1178@users.noreply.github.com>

* v3.11.0 (tgstation#224)

* Fix TOML dme test failing (tgstation#227)

* hash: optimize file hashing (tgstation#228)

* IconForge: DMI Generation, Code Reorganization, Improved Caching, Cleaner I/O (tgstation#213)

* Adds a new `uuid` module (tgstation#229)

* IconForge: BYOND Parity + Tests, Optimizations, New Transforms (tgstation#230)

* `clippy` lints for tgstation#230 (tgstation#233)

* Adds `http_request_fire_and_forget` (tgstation#232)

* DMI metadata reading and injection (tgstation#234)

* 4.0.0 (tgstation#235)

* dmi: Add QR code generation (tgstation#226)

* Bump `dmi` to 0.5.0, optimize `dmi_read_metadata` (tgstation#238)

* Update CI to target 516.1666 (tgstation#237)

* 4.1.0 (tgstation#239)

* makes `rustg_noise_poisson_map` around 8x faster (tgstation#240)

* IconForge: Headless Icon Generation (tgstation#236)

* `cargo update` & `cargo upgrade` & `png` fix (tgstation#241)

* assorted optimizations to `cellularnoise`, `dbpnoise`, and `worleynoise` (tgstation#243)

* v4.2.0 (tgstation#242)

* Fix attempt for CI

* hash: Adds ChaCha20 CSPRNG functions, updates TOTP generator, adds Base32 (tgstation#225)

* Handle errors in panic hook explicitly (tgstation#245)

* do not store Git information static (tgstation#247)

* Prevent `decode_base64` and `decode_base32` from panicking if given invalid base64 (tgstation#244)

* 4.3.0 (tgstation#249)

* fix iconforge generate_headless on win for bad paths not panicing (tgstation#248)

* 5.0.0 (tgstation#250)

* redo iconforge errors - 5.0.1 (tgstation#252)

* 6.0.0 - drop windows 7 support (tgstation#251)

* fix spritesheet gen on linux (tgstation#255)

* cargo update and clippy (tgstation#253)

* 6.0.1 - and `cargo update` (tgstation#256)

---------

Co-authored-by: Lucy <lucy@absolucy.moe>
Co-authored-by: TiviPlus <57223640+TiviPlus@users.noreply.github.com>
Co-authored-by: ZeWaka <zewakagamer@gmail.com>
Co-authored-by: Mothblocks <35135081+Mothblocks@users.noreply.github.com>
Co-authored-by: Comrade Niobe <126028983+ComradeNiobe@users.noreply.github.com>
Co-authored-by: Kapu1178 <75460809+Kapu1178@users.noreply.github.com>
Co-authored-by: Y0SH1M4S73R <y0sh1m4s73r@gmail.com>
Co-authored-by: Ivy <distributivgesetz93@gmail.com>
Co-authored-by: Zephyr <12817816+ZephyrTFA@users.noreply.github.com>
Co-authored-by: Kashargul <144968721+Kashargul@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants