Enhance SpinLock by RandomShaper · Pull Request #85167 · godotengine/godot

RandomShaper · 2023-11-21T09:18:31Z

As advised by https://rigtorp.se/spinlock/ and other resources, a good spinlock should do the following at least (and the following points are what this PR adds):

Use the CPU pause instruction while looping, to inform the hardware we are spinning so it does the right thing. Depending on architectures, this may improve power efficiency, for instance (better usage of phone/laptop battery). Not only that, but hyper-threaded cores can also work better since one of the logical hardware threads knows the other isn't doing anything particularly useful while the latter is just spinning around.
If the first attempt to lock fails, instead of blindlessly retrying, loop with relaxed memory ordering until it seems the lock is available again and, only then, retry with acquire semantics. This avoids wasteful sync operations, and may improve performanve.

2024-10-23:
~~As a disclaimer, I haven't benchmarked this for either performance nor power usage. Maybe the Production team can lend a hand.~~
Using a benchmark I've written for this (godotengine/godot-benchmarks#95), these are the results on a Windows 10 PC, before (a) and after (b) this PR (unit is time, lower is better; note how this PR highly improves the contended cases):

2024-11-04: Benchmarked after going back to the CPU pause approach:

{
        "category": "Core > Object Db",
        "name": "12 Threads Full Contention",
        "results": {
                "time": {
                        "a": 1631,
                        "b": 1227,
                        "a_div_b": 1.3292583537082314
                }
        }
}
{
        "category": "Core > Object Db",
        "name": "12 Threads Half Contention",
        "results": {
                "time": {
                        "a": 834.1,
                        "b": 631.4,
                        "a_div_b": 1.3210326259106748
                }
        }
}
{
        "category": "Core > Object Db",
        "name": "12 Threads Little Contention",
        "results": {
                "time": {
                        "a": 186.8,
                        "b": 149.9,
                        "a_div_b": 1.246164109406271
                }
        }
}
{
        "category": "Core > Object Db",
        "name": "12 Threads Slope Contention",
        "results": {
                "time": {
                        "a": 760.3,
                        "b": 614.2,
                        "a_div_b": 1.2378704005210028
                }
        }
}
{
        "category": "Core > Object Db",
        "name": "2 Threads Full Contention",
        "results": {
                "time": {
                        "a": 57.44,
                        "b": 61.74,
                        "a_div_b": 0.9303530936183997
                }
        }
}
{
        "category": "Core > Object Db",
        "name": "2 Threads Half Contention",
        "results": {
                "time": {
                        "a": 45.44,
                        "b": 47.77,
                        "a_div_b": 0.9512246179610633
                }
        }
}
{
        "category": "Core > Object Db",
        "name": "2 Threads Little Contention",
        "results": {
                "time": {
                        "a": 33.54,
                        "b": 34.96,
                        "a_div_b": 0.9593821510297482
                }
        }
}
{
        "category": "Core > Object Db",
        "name": "4 Threads Full Contention",
        "results": {
                "time": {
                        "a": 178.1,
                        "b": 189.9,
                        "a_div_b": 0.9378620326487624
                }
        }
}
{
        "category": "Core > Object Db",
        "name": "4 Threads Half Contention",
        "results": {
                "time": {
                        "a": 103.3,
                        "b": 102.3,
                        "a_div_b": 1.0097751710654936
                }
        }
}
{
        "category": "Core > Object Db",
        "name": "4 Threads Little Contention",
        "results": {
                "time": {
                        "a": 44.67,
                        "b": 46.76,
                        "a_div_b": 0.9553036783575707
                }
        }
}
{
        "category": "Core > Object Db",
        "name": "4 Threads Slope Contention",
        "results": {
                "time": {
                        "a": 132.5,
                        "b": 139.6,
                        "a_div_b": 0.9491404011461319
                }
        }
}
{
        "category": "Core > Object Db",
        "name": "8 Threads Full Contention",
        "results": {
                "time": {
                        "a": 741.4,
                        "b": 636.9,
                        "a_div_b": 1.1640759930915372
                }
        }
}
{
        "category": "Core > Object Db",
        "name": "8 Threads Half Contention",
        "results": {
                "time": {
                        "a": 381.3,
                        "b": 327.2,
                        "a_div_b": 1.1653422982885087
                }
        }
}
{
        "category": "Core > Object Db",
        "name": "8 Threads Little Contention",
        "results": {
                "time": {
                        "a": 99.02,
                        "b": 89.99,
                        "a_div_b": 1.1003444827203022
                }
        }
}
{
        "category": "Core > Object Db",
        "name": "8 Threads Slope Contention",
        "results": {
                "time": {
                        "a": 389.2,
                        "b": 343.2,
                        "a_div_b": 1.134032634032634
                }
        }
}
{
        "category": "Core > Object Db",
        "name": "Single",
        "results": {
                "time": {
                        "a": 30.13,
                        "b": 31.28,
                        "a_div_b": 0.963235294117647
                }
        }
}

Calinou · 2023-11-21T15:10:23Z

Use the CPU pause instruction while looping, to inform the hardware we are spinning so it does the right thing. Depending on architectures, this may improve power efficiency, for instance (better usage of phone/laptop battery). Not only that, but hyper-threaded cores can also work better since one of the logical hardware threads knows the other isn't doing anything particularly useful while the latter is just spinning around.

Is the AMD Zen MWAITX instruction worth using in this case? It's been reported to result in significant power savings which can benefit the Steam Deck. This would need detecting whether the instruction is available though.

Note that if this is implemented, this will need to be tested by someone else as I don't have the hardware to test it.

RandomShaper · 2023-11-21T15:29:54Z

Is the AMD Zen MWAITX instruction worth using in this case?

I didn't know about it, to be honest. However, judging by the spec, it seems that MONITORX-MWAITX make for an extremely efficient hardware-supported spinlock. I asked an AMD engineer about our spinlock, by the way, and, weirdly enough, he wouldn't mention these instructions.

Now I'm wondering if having to branch to the MWAITX-based implementation or the classic one, plus having both inlined would kill the benefits somehow. Hard to tell without measuring.

CedNaru · 2024-04-16T17:13:02Z

I'd like to add my two cents saying that I'm eager to see this class being improved as the current version can really be dangerous for performance.
I suggest this excellent read that goes a bit deeper into Spinlock implementations and shows how the one used by Godot can easily make threads freeze for a few ms:
https://probablydance.com/2019/12/30/measuring-mutexes-spinlocks-and-how-bad-the-linux-scheduler-really-is/

Note that despite the improvements, Mutex is still a better choice on modern hardware and systems in most cases, even for short locks.

RandomShaper · 2024-10-21T11:15:08Z

~~I've removed the arch-dependant pieces and used std::this_thread::yield instead. That should bring the appropriate semantics to all the supported platforms already.~~

RandomShaper · 2024-10-23T13:32:38Z

I'm bumping this because I've added benchmark results to the description.

RandomShaper · 2024-11-04T10:18:12Z

Going back to the CPU pause approach, once realized thread yield is not appropriate for this. I've also applied @bruvzg's suggestions.

In the future we may want to get rid of spinlocks altogether, but for now let's at least have a better one.

Other CPU-specific instructions such as the mwaitxx mentioned by @Calinou or ARM's wfe could make for a subsequent iteration of this.

Calinou

Tested locally (rebased on top of master b00e1cb), it works as expected.

Benchmark

PC specifications

CPU: Intel Core i9-13900K
GPU: NVIDIA GeForce RTX 4090
RAM: 64 GB (2×32 GB DDR5-5800 C30)
SSD: Solidigm P44 Pro 2 TB
OS: Linux (Fedora 41)

Using an optimized editor build for testing (production=yes lto=full).

A is master, B is this PR rebased on master.

{
	"category": "Core > Object Db",
	"name": "12 Threads Full Contention",
	"results": {
		"time": {
			"a": 2297.0,
			"b": 1186.0,
			"a_div_b": 1.936762225969646
		}
	}
}
{
	"category": "Core > Object Db",
	"name": "12 Threads Half Contention",
	"results": {
		"time": {
			"a": 1155.0,
			"b": 605.3,
			"a_div_b": 1.9081447216256404
		}
	}
}
{
	"category": "Core > Object Db",
	"name": "12 Threads Little Contention",
	"results": {
		"time": {
			"a": 233.0,
			"b": 128.8,
			"a_div_b": 1.8090062111801242
		}
	}
}
{
	"category": "Core > Object Db",
	"name": "12 Threads Slope Contention",
	"results": {
		"time": {
			"a": 1064.0,
			"b": 599.5,
			"a_div_b": 1.774812343619683
		}
	}
}
{
	"category": "Core > Object Db",
	"name": "2 Threads Full Contention",
	"results": {
		"time": {
			"a": 83.19,
			"b": 76.33,
			"a_div_b": 1.0898729202148565
		}
	}
}
{
	"category": "Core > Object Db",
	"name": "2 Threads Half Contention",
	"results": {
		"time": {
			"a": 50.19,
			"b": 45.45,
			"a_div_b": 1.104290429042904
		}
	}
}
{
	"category": "Core > Object Db",
	"name": "2 Threads Little Contention",
	"results": {
		"time": {
			"a": 26.51,
			"b": 28.54,
			"a_div_b": 0.9288717589348284
		}
	}
}
{
	"category": "Core > Object Db",
	"name": "4 Threads Full Contention",
	"results": {
		"time": {
			"a": 275.8,
			"b": 214.9,
			"a_div_b": 1.2833876221498373
		}
	}
}
{
	"category": "Core > Object Db",
	"name": "4 Threads Half Contention",
	"results": {
		"time": {
			"a": 148.8,
			"b": 119.2,
			"a_div_b": 1.2483221476510067
		}
	}
}
{
	"category": "Core > Object Db",
	"name": "4 Threads Little Contention",
	"results": {
		"time": {
			"a": 39.28,
			"b": 38.06,
			"a_div_b": 1.0320546505517603
		}
	}
}
{
	"category": "Core > Object Db",
	"name": "4 Threads Slope Contention",
	"results": {
		"time": {
			"a": 199.8,
			"b": 163.7,
			"a_div_b": 1.220525351252291
		}
	}
}
{
	"category": "Core > Object Db",
	"name": "8 Threads Full Contention",
	"results": {
		"time": {
			"a": 1018.0,
			"b": 601.0,
			"a_div_b": 1.6938435940099834
		}
	}
}
{
	"category": "Core > Object Db",
	"name": "8 Threads Half Contention",
	"results": {
		"time": {
			"a": 520.0,
			"b": 319.8,
			"a_div_b": 1.6260162601626016
		}
	}
}
{
	"category": "Core > Object Db",
	"name": "8 Threads Little Contention",
	"results": {
		"time": {
			"a": 118.2,
			"b": 78.52,
			"a_div_b": 1.5053489556800816
		}
	}
}
{
	"category": "Core > Object Db",
	"name": "8 Threads Slope Contention",
	"results": {
		"time": {
			"a": 551.2,
			"b": 356.9,
			"a_div_b": 1.5444101989352763
		}
	}
}

clayjohn

Looks good to me. The comments from bruvzg and mrsaturnsan have both been resolved. So I think we can go ahead with this.

Repiteo · 2024-11-10T18:19:57Z

Thanks!

neptuwunium · 2024-11-11T05:54:42Z

This changeset causes Godot to crash on startup on Linux when using builds compiled with AVX2 ISAs enabled (AMD Excavator, Intel Haswell or newer)

git checkout 01ad56da38fa10949ff7878d4d6a5d6c195eff39
scons target=editor ccflags="-march=x86-64-v3"
./bin/godot.linuxbsd.editor.x86_64

This occurred on an AMD Ryzen 9 7900X.

Edit 2: This seems to be a cacheline issue. Adding alignas(CACHE_LINE_SIZE) to every usage of spinline makes it segfault in a different place

Stacktrace 1 (stock)

* thread #1, name = 'godot.linuxbsd.editor.x86_64', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
  * frame #0: RendererSceneCull::RendererSceneCull(this=<unavailable>) at self_list.h:165:25
    frame #1: RenderingServerDefault::_init(this=<unavailable>) at rendering_server_default.cpp:215:26
    frame #2: Main::setup2(p_show_boot_logo=<unavailable>) at main.cpp:3106:25
    frame #3: Main::setup(execpath=<unavailable>, argc=0, argv=<unavailable>, p_second_phase=true) at main.cpp:2648:20
    frame #4: main(argc=1, argv=<unavailable>) at godot_linuxbsd.cpp:74:25

Stacktrace 2 (inline everywhere)

* thread #1, name = 'godot.linuxbsd.editor.x86_64', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
  * frame #0: WorkerThreadPool::WorkerThreadPool() [inlined] SelfList<WorkerThreadPool::Task>::List::List(this=<unavailable>) at self_list.h:165:25
    frame #1: WorkerThreadPool::WorkerThreadPool(this=<unavailable>) at worker_thread_pool.cpp:835:36
    frame #2: register_core_types() at register_core_types.cpp:301:23
    frame #3: Main::setup(execpath=<unavailable>, argc=0, argv=<unavailable>, p_second_phase=true) at main.cpp:952:21
    frame #4: main(argc=1, argv=<unavailable>) at godot_linuxbsd.cpp:74:25

The segfault occurs when zeroing the atomic lock data which it does by moving a zero from the ymm register.

jams3223 · 2024-11-11T11:32:38Z

@RandomShaper AMD stated in their optimization guide that the compiler can automatically compile std_mutex into the mwaitx hardware instructions.
https://gpuopen.com/gdc-presentations/2023/GDC-2023-AMD-Ryzen-Processor-Software-Optimization.pdf

neptuwunium · 2024-11-11T16:24:27Z

As a note, that's not guaranteed. On POSIX it will be delegated to pthread_mutex_{lock,unlock}, which calls lock cmpxchg instead of mwaitx, however from what I understand depending on architecture microcode it may end up being identical over a longer period of time. (Especially with kernel-level thread schedulers.)

Edit: Looking over the assembly, this seems to do lock cmpxchg anyway due to the atomic set/clear. It might just be simpler and safer to use a mutex.

neptuwunium · 2024-11-11T17:30:22Z

Removing the alignas(Thread::CACHE_LINE_BYTES) line fixes the crashes on AVX2 compile targets.

__attribute__((aligned(Thread::CACHE_LINE_BYTES))) strangely, does as well. MSVC equivalent would be __declspec(align(Thread::CACHE_LINE_BYTES))

Either this attribute is ignored or alignas does not behave as expected.

Edit: Reading up on the alignas keyword, it apparently cannot exceed alignof(max_align_t) which is 16. Cache lines are defined to be 64 bytes on most x86 systems. So any alignas falls back to 16 (which is standard for x86-64-v1) however needs 32 and 64 for AVX2 and AVX512 respectively. __attribute__((aligned())) and __declspec(align()) function differently as they enforce alignment regardless of any other factors.

Edit 2: 😅 now it's segfaulting in AVX-512 CPU targets, resolved by bumping the alignment to 128. The particular sensitivity to cache line size is frustrating.

Edit 3: It still crashes, just at random points now. This is very unstable on x86-64-v3 and v4.

RandomShaper · 2024-11-12T12:28:02Z

@yretenai. thank you for your findings. Can you give me stack traces of the crashes? I'm wondering if there are cases where the SpinLock ends up misaligned for some reason. This constructor may help:

	SpinLock() {
		CRASH_COND(uintptr_t(this) % Thread::CACHE_LINE_BYTES != 0);
	}

neptuwunium · 2024-11-12T19:27:24Z

Thread::CACHE_LINE_BYTES = 64

Built with scons target=editor debug_symbols=on -j24

ERROR: FATAL: Condition "uintptr_t(this) % Thread::CACHE_LINE_BYTES != 0" is true.
   at: SpinLock (./core/os/spin_lock.h:93)

  * frame #0: 0x000055555a3cc8fd CallQueue::CallQueue(PagedAllocator<CallQueue::Page, true, 4096u>*, unsigned int, String const&) [inlined] SpinLock::SpinLock(this=0x000055555d7951b0) at spin_lock.h:93:3
    frame #1: 0x000055555a3cc8d3 CallQueue::CallQueue(PagedAllocator<CallQueue::Page, true, 4096u>*, unsigned int, String const&) [inlined] SpinLock::SpinLock(this=0x000055555d7951b0) at spin_lock.h:92:2
    frame #2: 0x000055555a3cc8d3 CallQueue::CallQueue(PagedAllocator<CallQueue::Page, true, 4096u>*, unsigned int, String const&) [inlined] PagedAllocator<CallQueue::Page, true, 4096u>::PagedAllocator(p_page_size=16, this=0x000055555d795170) at paged_allocator.h:159:59
    frame #3: 0x000055555a3cc8c0 CallQueue::CallQueue(this=0x000055555d794dc0, p_custom_allocator=<unavailable>, p_max_pages=8192, p_error_text=0x00007fffffffd310) at message_queue.cpp:468:15
    frame #4: 0x000055555a3ccad1 MessageQueue::MessageQueue(this=0x000055555d794dc0) at message_queue.cpp:509:113
    frame #5: 0x0000555555caba3e Main::setup(execpath=<unavailable>, argc=0, argv=0x00007fffffffdc80, p_second_phase=true) at main.cpp:2634:18
    frame #6: 0x0000555555bc83ff main(argc=1, argv=0x00007fffffffdc78) at godot_linuxbsd.cpp:74:25

Built with scons target=editor ccflags="-march=x86-64-v3" debug_symbols=on -j24

ERROR: FATAL: Condition "uintptr_t(this) % Thread::CACHE_LINE_BYTES != 0" is true.
   at: SpinLock (./core/os/spin_lock.h:93)

  * frame #0: 0x000055555a48d795 WorkerThreadPool::WorkerThreadPool() [inlined] SpinLock::SpinLock(this=0x000055555d689160) at spin_lock.h:93:3
    frame #1: 0x000055555a48d76b WorkerThreadPool::WorkerThreadPool() [inlined] SpinLock::SpinLock(this=<unavailable>) at spin_lock.h:92:2
    frame #2: 0x000055555a48d76b WorkerThreadPool::WorkerThreadPool() [inlined] PagedAllocator<WorkerThreadPool::Task, false, 1024u>::PagedAllocator(p_page_size=<unavailable>, this=<unavailable>) at paged_allocator.h:159:59
    frame #3: 0x000055555a48d76b WorkerThreadPool::WorkerThreadPool(this=0x000055555d688fa0) at worker_thread_pool.cpp:835:36
    frame #4: 0x0000555559df39ec register_core_types() at register_core_types.cpp:301:23
    frame #5: 0x0000555555c9db0c Main::setup(execpath=<unavailable>, argc=0, argv=0x00007fffffffdc80, p_second_phase=true) at main.cpp:952:21
    frame #6: 0x0000555555bc724f main(argc=1, argv=0x00007fffffffdc78) at godot_linuxbsd.cpp:74:25

Reminder: the code works without alignment. The compiler will attempt to ensure std::atomic is aligned to the required alignment by introducing padding structures (which it seems to fail to do when forced alignment is introduced,) but this may not per-se be aligned to the cache line as demonstrated here:

ERROR: FATAL: Condition "uintptr_t(this) % Thread::CACHE_LINE_BYTES != 0" is true.
   at: SpinLock (./core/os/spin_lock.h:93)

  * frame #0: 0x0000555555bf666d _GLOBAL__sub_I__ZN7Variant5Pools13_bucket_smallE() [inlined] SpinLock::SpinLock(this=0x0000000007cb1844) at spin_lock.h:93:3
    frame #1: 0x0000555555bf6663 _GLOBAL__sub_I__ZN7Variant5Pools13_bucket_smallE() [inlined] SpinLock::SpinLock(this=0x0000000007cb1844) at spin_lock.h:92:2
    frame #2: 0x0000555555bf665c _GLOBAL__sub_I__ZN7Variant5Pools13_bucket_smallE() [inlined] PagedAllocator<Variant::Pools::BucketSmall, true, 4096u>::PagedAllocator(p_page_size=4096, this=0x0000000007cb1820) at paged_allocator.h:159:59
    frame #3: 0x0000555555bf663e _GLOBAL__sub_I__ZN7Variant5Pools13_bucket_smallE() [inlined] __static_initialization_and_destruction_0 at variant.cpp:41:67
    frame #4: 0x0000555555bf663e _GLOBAL__sub_I__ZN7Variant5Pools13_bucket_smallE() at variant.cpp:3698:1

Also: Enforcing alignment with __attribute__((aligned(Thread::CACHE_LINE_BYTES))) does not stop the assertion. Having run it a bunch of times now, due to ASLR sometimes it magically aligns everything and runs fine.

RandomShaper added enhancement topic:core labels Nov 21, 2023

RandomShaper added this to the 4.3 milestone Nov 21, 2023

RandomShaper requested a review from a team as a code owner November 21, 2023 09:18

RandomShaper force-pushed the good_spinlock branch 3 times, most recently from b29354a to 8e1132e Compare November 21, 2023 09:46

RandomShaper marked this pull request as draft November 21, 2023 09:47

RandomShaper force-pushed the good_spinlock branch 2 times, most recently from 221abf0 to 8bc78ab Compare November 21, 2023 10:27

RandomShaper force-pushed the good_spinlock branch from 8bc78ab to 14f66bc Compare November 21, 2023 15:31

RandomShaper marked this pull request as ready for review November 21, 2023 15:32

bruvzg reviewed Nov 22, 2023

View reviewed changes

Comment thread core/os/spin_lock.h Outdated

Comment thread core/os/spin_lock.h Outdated

RandomShaper mentioned this pull request Dec 21, 2023

Make RID_Owner lock-free for fetching. #86333

Closed

RandomShaper marked this pull request as draft February 6, 2024 18:58

akien-mga modified the milestones: 4.3, 4.x Jun 14, 2024

RandomShaper force-pushed the good_spinlock branch from 14f66bc to c5a3b6b Compare October 21, 2024 11:11

RandomShaper added performance cherrypick:4.3 Considered for cherry-picking into a future 4.3.x release labels Oct 21, 2024

RandomShaper force-pushed the good_spinlock branch from c5a3b6b to 2652397 Compare October 21, 2024 11:13

RandomShaper requested a review from bruvzg October 21, 2024 11:13

RandomShaper marked this pull request as ready for review October 21, 2024 11:15

RandomShaper force-pushed the good_spinlock branch from 2652397 to f545d40 Compare October 21, 2024 11:16

RandomShaper modified the milestones: 4.x, 4.4 Oct 21, 2024

RandomShaper force-pushed the good_spinlock branch from 8addc1f to d0fb2fc Compare October 22, 2024 11:54

RandomShaper force-pushed the good_spinlock branch from d0fb2fc to d5a24f9 Compare October 24, 2024 10:08

This was referenced Oct 24, 2024

Fix RID_Owner synchronization #98488

Merged

Improve core and GDScript VM multithreading performance #98469

Draft

RandomShaper force-pushed the good_spinlock branch 2 times, most recently from 8bc703d to 03069b4 Compare November 4, 2024 17:35

Calinou approved these changes Nov 5, 2024

View reviewed changes

clayjohn reviewed Nov 6, 2024

View reviewed changes

Comment thread core/os/thread.h Outdated

RandomShaper mentioned this pull request Nov 7, 2024

Implement adaptive spin lock on ARM Apple devices #98696

Closed

RandomShaper added 2 commits November 8, 2024 08:06

Add a utility cache line size constant

95c6a24

Enhance SpinLock

360ed2b

RandomShaper force-pushed the good_spinlock branch from 03069b4 to 360ed2b Compare November 8, 2024 07:06

clayjohn approved these changes Nov 9, 2024

View reviewed changes

Repiteo merged commit 01ad56d into godotengine:master Nov 10, 2024

RandomShaper deleted the good_spinlock branch November 12, 2024 01:09

RandomShaper mentioned this pull request Nov 13, 2024

SpinLock: Overhaul false sharing prevention #99168

Merged

jss2a98aj mentioned this pull request Nov 17, 2024

[4.4 backport] Assorted fixes blazium-games/blazium#140

Merged

hpvb mentioned this pull request Nov 27, 2024

Use atomic_flag for SpinLock #99738

Open

MBCX mentioned this pull request Dec 10, 2024

32-bit builds of Godot do not work with LLVM MinGW #100163

Closed

Spartan322 mentioned this pull request Jan 17, 2025

[4.3] Cherry-picks for the 4.3 (4.3.1) branch - 1st crashes batch Redot-Engine/redot-engine#928

Merged

akien-mga removed the cherrypick:4.3 Considered for cherry-picking into a future 4.3.x release label Jan 30, 2025

Uh oh!

Conversation

RandomShaper commented Nov 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Calinou commented Nov 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RandomShaper commented Nov 21, 2023

Uh oh!

Uh oh!

Uh oh!

CedNaru commented Apr 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RandomShaper commented Oct 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RandomShaper commented Oct 23, 2024

Uh oh!

RandomShaper commented Nov 4, 2024

Uh oh!

Calinou left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Benchmark

Uh oh!

Uh oh!

clayjohn left a comment

Choose a reason for hiding this comment

Uh oh!

Repiteo commented Nov 10, 2024

Uh oh!

neptuwunium commented Nov 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jams3223 commented Nov 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neptuwunium commented Nov 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neptuwunium commented Nov 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RandomShaper commented Nov 12, 2024

Uh oh!

neptuwunium commented Nov 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

RandomShaper commented Nov 21, 2023 •

edited

Loading

Calinou commented Nov 21, 2023 •

edited

Loading

CedNaru commented Apr 16, 2024 •

edited

Loading

RandomShaper commented Oct 21, 2024 •

edited

Loading

Calinou left a comment •

edited

Loading

neptuwunium commented Nov 11, 2024 •

edited

Loading

jams3223 commented Nov 11, 2024 •

edited

Loading

neptuwunium commented Nov 11, 2024 •

edited

Loading

neptuwunium commented Nov 11, 2024 •

edited

Loading

neptuwunium commented Nov 12, 2024 •

edited

Loading