[Compiled Graph] Enhance Compile Graph with Multi-Device Support by hipudding · Pull Request #51032 · ray-project/ray

hipudding · 2025-03-03T11:47:28Z

Why are these changes needed?

This PR improves multi-device support in Compile Graph, which significantly reduces Tensor transmission latency by utilizing out-of-band communication. Currently, this feature only supports CUDA’s NCCL. Since Ray already supports multiple accelerators, it is necessary to extend Compile Graph to support multi-device as well.

This PR mainly introduces two key changes:
1. Removed dependency on cupy.cuda.ExternalStream – Since this library only supports CUDA devices, we replaced it with a more general stream context manager to accommodate various accelerators. The new implementation uses torch.{device}.StreamContext.
2. Replaced hardcoded torch.cuda.xxx calls with AcceleratorRuntime – This allows automatic detection of the accelerator type and invokes the appropriate device-specific functions.

How to add a new backend for CG? here's an example for Ascend NPU:

import ray
import torch
import torch_npu
from ray.dag import InputNode
# implement customer Communicator class
from ray.experimental.channel.hccl_group import _HcclGroup
from ray.experimental.channel.accelerator_context import register_accelerator_context

@ray.remote
class TorchTensorWorker:
    def __init__(self):
        self.device = torch.device('npu:0')
        torch.npu.set_device(self.device)

    def send(self, shape, dtype, value: int):
        return torch.ones(shape, dtype=dtype, device=self.device) * value

    def recv(self, tensor):
        return (tensor[0].item(), tensor.shape, tensor.dtype)

# global register accelerator context
register_accelerator_context('npu', _HcclGroup)

actor_cls = TorchTensorWorker.options(num_cpus=0, resources={'NPU': 1})

sender = actor_cls.remote()
receiver = actor_cls.remote()

with InputNode() as inp:
    dag = sender.send.bind(inp.shape, inp.dtype, inp[0])
    dag = dag.with_tensor_transport(transport='nccl')
    dag = receiver.recv.bind(dag)

shape = (10,)
dtype = torch.float16

compiled_dag = dag.experimental_compile()
for i in range(3):
    ref = compiled_dag.execute(i, shape=shape, dtype=dtype)
    assert ray.get(ref) == (i, shape, dtype)

print("Success")

Related issue number

This PR is the main part of Task 2 in #51574
It would better to set the function name more general, such as changing requires_nccl to require_communicator. This is implemented in #51061.

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

hipudding · 2025-03-04T06:16:16Z

@ruisearch42 Good Day. Could you please review this PR? Thanks.

noemotiovon · 2025-03-04T06:30:38Z

python/ray/experimental/channel/communicator.py

If the method returns None, is it still necessary to mark it as an abstract method?

Keep this class abstrct.

ruisearch42 · 2025-03-05T04:35:08Z

Hi @hipudding , is this ready for review? btw, aDAG was renamed to Compiled Graph: https://docs.ray.io/en/latest/ray-core/compiled-graph/ray-compiled-graph.html

hipudding · 2025-03-05T04:38:54Z

Hi @hipudding , is this ready for review? btw, aDAG was renamed to Compiled Graph: https://docs.ray.io/en/latest/ray-core/compiled-graph/ray-compiled-graph.html

Yes, I think the main functionality of this PR is ready, I may make some minor adjustments later. Thanks for your review.
I'vd noticed that there's some test cases failed. I will fix them ASAP.

ruisearch42 · 2025-03-05T05:54:48Z

Yes, I think the main functionality of this PR is ready, I may make some minor adjustments later. Thanks for your review. I'vd noticed that there's some test cases failed. I will fix them ASAP.

Thanks. I will review tomorrow!

ruisearch42

Trying to better understand the PR

ruisearch42 · 2025-03-06T01:52:45Z

python/ray/dag/compiled_dag_node.py

-                with self._send_stream:
-                    return self._write()


Why don't we want to use context manager?

This stream context manager is provided by cupy, it only support CUDA devices, So I just remote context manager and specified stream in read and write.
But I think I can implement a stream context manager in Ray, so cupy is not needed. Which is better?

Done. use torch StreamContext instead to avoid too many changes.

ruisearch42 · 2025-03-06T01:53:45Z

python/ray/dag/compiled_dag_node.py

        self._intermediate_future = future

-    def reset_and_wait_intermediate_future(self) -> Any:
+    def reset_and_wait_intermediate_future(self, stream: Any = None) -> Any:


update doc for arg

Done. reverted this part.

ruisearch42 · 2025-03-06T01:54:01Z

python/ray/dag/compiled_dag_node.py

+        self,
+        val: Any,
+        wrap_in_gpu_future: bool,
+        stream: Any = None,


update doc for stream

Done. reverted this part.

ruisearch42 · 2025-03-06T01:57:26Z

python/ray/experimental/channel/driver_communicator_holder.py

+class _DriverGroupHolder(Communicator):
+    """
+    Communicator place holder for Driver, Since driver may has no
+    Accelerators, TorchDeviceManager cannot be used.
+    """


Can you elaborate a bit why this is needed?

I replace all torch.cuda.xxx by TorchDeviceManager, including create a _NcclGroup. But driver will keep a communicator instance itself. Driver will not join the group, but need record some information like world size and actors.
But Driver may has no GPU ( or other accelerators), it will choose the wrong TorchDeviceManager (CPUTorchDeviceManager), and get_communicator will return None.
Since Driver will not join the group, it use Communicator only for store some information. So I create a Hardware-independent class for driver.

Comment updated.

ruisearch42 · 2025-03-06T02:04:35Z

python/ray/experimental/channel/hccl_group.py

+        comm_id: int,
+        rank: Optional[int],
+        actor_handles: List["ray.actor.ActorHandle"],
+        acl_stream: Optional[Any],


What is acl_tream? doc for args

acl_stream is the raw stream in Ascend NPU. Just like cuda_stream to GPU.
What about name this stream torch_stream? since acl_stream and cuda_stream are both torch_stream now.

Done. I changed it to npu_stream for better understanding.

ruisearch42 · 2025-03-06T02:06:12Z

python/ray/experimental/channel/hccl_group.py

+    def get_world_size(self) -> int:
+        return self._world_size
+
+    def send(self, buf: "torch.Tensor", peer_rank: int) -> None:


many methods in this class look like code duplication from _NcclGroup with minor changes. Should they be unified?

Yes, we should indeed reduce duplicate code. I will think about how to design it.

Indeed, these two files share logical similarities and contain a lot of duplicate code. However, there are several differences in the details. For example, torch.cuda.xxx needs to be replaced with torch.npu.xxx, hccl_group requires an additional StreamContext, and the set device operation must be performed before get_unique_id and creating the HCCL group. Therefore, I haven’t found a good way to merge these two files. Do you have any suggestions?

hipudding · 2025-03-07T06:26:43Z

Hi @ruisearch42, Could you please review this PR again？

BTW, I fixed test case fail in doc_test_cgraph_nccl. It blocked this PR.

noemotiovon

LGTM!

noemotiovon · 2025-04-29T06:56:05Z

python/ray/experimental/channel/accelerator_context.py

Not registered yet.

Got it. Big enough.

hipudding · 2025-05-06T01:02:15Z

@ruisearch42 @stephanie-wang This PR is ready for review now. Could you please review this PR again? Thanks.

ruisearch42 · 2025-05-07T00:43:06Z

@ruisearch42 @stephanie-wang This PR is ready for review now. Could you please review this PR again? Thanks.

Thanks for the efforts! will do!

stephanie-wang

Thanks for the contribution! I like the new APIs. Let's check if we can reduce the driver API; eventually it would be good to use this as a single-controller API for Ray GPU objects too.

stephanie-wang · 2025-05-07T00:50:40Z

python/ray/experimental/channel/accelerator_context.py

What is "it" in this comment?

Rewrite the class comment.

stephanie-wang · 2025-05-07T00:51:12Z

python/ray/experimental/channel/accelerator_context.py

Type hints and comments.

stephanie-wang · 2025-05-07T00:53:09Z

python/ray/experimental/channel/accelerator_context.py

Why use _set_context instead of just passing these args to cls() constructor?

There are two steps here:
1. Set class attributes, including _torch_module_name and _communicator_cls. These attributes can be configured via register_accelerator_context; if not set, the system will automatically detect whether to use GPU or CPU.
2. Instantiate the AcceleratorContext object based on these class attributes.

If we were to use the constructor here, we would need to introduce global variables to store _torch_module_name and _communicator_cls.

Which approach do you think is better?

Hmm the current code is brittle because if you call AcceleratorContext._set_context, it will modify a previously created singleton returned by AcceleratorContext.get() as well. It would be better if you can make it so that calling AcceleratorContext.get() returns an instance whose attributes are not tied to the class attributes. This make it a bit nicer for testing or if we eventually want to support multiple contexts in the same process.

I don't have a strong preference on how you should do that, although I agree it would be nice to not need global variables for the torch_module_name and communicator_cls. You could follow the DataContext pattern, which stores the defaults on the class but uses dataclass and a global default context for the current instance.

stephanie-wang · 2025-05-07T00:56:09Z

python/ray/experimental/channel/accelerator_context.py

Suggested change

def get_communicator(self, *args, **kwargs):

def create_communicator(self, *args, **kwargs):

stephanie-wang · 2025-05-07T00:56:21Z

python/ray/experimental/channel/accelerator_context.py

Suggested change

Retrieves the communication group for collective operations.

Creates a communication group for collective operations.

stephanie-wang · 2025-05-07T01:01:16Z

python/ray/experimental/channel/torch_tensor_nccl_channel.py

Rename to do_register_accelerator_context?

stephanie-wang · 2025-05-07T01:02:03Z

python/ray/experimental/channel/torch_tensor_nccl_channel.py

A bit confusing that this is calling _do_check_has_communicator but the variable is called has_accelerators. Can't the actors have accelerators but not a communicator created yet?

Also I think this check is not exactly equivalent to the previous check - is it right that each actor could now have an AcceleratorContext but they could be for different devices? Do we have a unit test for this?

You're right. I will fix this.

This has been changed to check whether all actors use the same accelerator, to ensure consistency across all actors. A test case added.

stephanie-wang · 2025-05-07T01:02:32Z

python/ray/experimental/channel/utils.py

Just checking - why was it that this code could be removed?

This function returns the logical IDs of the CUDA devices visible to the current worker. Currently, in CG, the device at index 0 is used as the default device, and the logical ID of the device at index 0 is also 0, so I believe calling get_cuda_devices is unnecessary. I think this part needs to be double-checked by @ruisearch42.

stephanie-wang · 2025-05-07T01:06:22Z

python/ray/experimental/channel/torch_tensor_nccl_channel.py

Suggested change

# set custom communicator on all actors

# Set custom communicator on all actors.

stephanie-wang · 2025-05-07T01:08:28Z

python/ray/experimental/channel/torch_tensor_nccl_channel.py

Is there a way we can take in the _communicator_cls and _torch_module_name as arguments to this function? That makes it explicit what the dependencies of this function are, instead of passing the dependencies through the accelerator context.

Is there a way we can take in the _communicator_cls and _torch_module_name as arguments to this function? That makes it explicit what the dependencies of this function are, instead of passing the dependencies through the accelerator context.

Are you suggesting that _communicator_cls and _torch_module_name should be passed as parameters to the _init_communicator function?

hipudding · 2025-05-07T06:36:25Z

Thanks for the contribution! I like the new APIs. Let's check if we can reduce the driver API; eventually it would be good to use this as a single-controller API for Ray GPU objects too.

Sorry, I didn’t quite understand what you meant. Could you please explain this sentence again?
“Let’s check if we can reduce the driver API; eventually it would be good to use this as a single-controller API for Ray GPU objects too.”

stephanie-wang · 2025-05-07T17:55:18Z

python/ray/experimental/channel/accelerator_context.py

Suggested change

# Not registrey yet.

# Not registered yet.

This part of code has been rewirte.

stephanie-wang · 2025-05-07T18:00:06Z

Thanks for the contribution! I like the new APIs. Let's check if we can reduce the driver API; eventually it would be good to use this as a single-controller API for Ray GPU objects too.

Sorry, I didn’t quite understand what you meant. Could you please explain this sentence again? “Let’s check if we can reduce the driver API; eventually it would be good to use this as a single-controller API for Ray GPU objects too.”

Ah this was just the comment about removing NotImplemented methods for the CommunicatorHandle class. Eventually we would like to use this API to program collective operations in general Ray programs, not just in compiled graphs (see #51173)

Bye-legumes · 2025-05-12T18:31:37Z

I think currently looks great, while currently, the assumption is that all the device that support this should support torch stream, right? Is that possible that we list the APIs that to support all these features at least?

hipudding · 2025-05-13T08:03:25Z

I think currently looks great, while currently, the assumption is that all the device that support this should support torch stream, right? Is that possible that we list the APIs that to support all these features at least?

Yes, because currently CG only supports out-of-band transmission of torch.tensor, the accelerator must support the PyTorch interface—not just the torch stream, but also the interfaces related to device and event. I can add the backend capability requirements in the comments of accelerator_context.py.

stephanie-wang

Thanks, this looks good! Please address the last few comments and then we can merge it.

stephanie-wang · 2025-05-13T18:05:20Z

python/ray/experimental/channel/accelerator_context.py

+class AcceleratorContext:
+    """
+    Provides a unified interface for managing different accelerator backends
+    This eincludes stram management, event creation, device context control,


Suggested change

This eincludes stram management, event creation, device context control,

This includes stream management, event creation, device context control,

stephanie-wang · 2025-05-13T18:05:34Z

python/ray/experimental/channel/accelerator_context.py

+    and communicator support for distributed communication.
+    """
+
+    def __init__(self, torch_module_name: str, commumcator_cls: Type[Communicator]):


Suggested change

def __init__(self, torch_module_name: str, commumcator_cls: Type[Communicator]):

def __init__(self, torch_module_name: str, communicator_cls: Type[Communicator]):

stephanie-wang · 2025-05-13T18:05:43Z

python/ray/experimental/channel/accelerator_context.py

+
+        Args:
+            torch_module_name: Name of the torch device module (e.g., "cuda", "cpu").
+            commumcator_cls: Class used to handle communication.


Suggested change

commumcator_cls: Class used to handle communication.

communicator_cls: Class used to handle communication.

stephanie-wang · 2025-05-13T18:07:38Z

python/ray/experimental/channel/accelerator_context.py

+    @staticmethod
+    def set(accelerator_context: "AcceleratorContext") -> None:
+        """
+        Sets the accelerator context to the default context.


Suggested change

Sets the accelerator context to the default context.

Overwrites the default accelerator context.

stephanie-wang · 2025-05-13T18:08:53Z

python/ray/experimental/channel/accelerator_context.py

+        """
+        return self._communicator_cls(*args, **kwargs)
+
+    def get_module_name(self) -> str:


Could you change these getter methods to use @property?

stephanie-wang · 2025-05-13T18:18:45Z

python/ray/experimental/channel/accelerator_context.py

+        torch_module_name: The name of the device module under torch.
+        communicator: The communicator class associated with the device.
+    """
+    accleerator_context = AcceleratorContext(torch_module_name, communicator_cls)


Suggested change

accleerator_context = AcceleratorContext(torch_module_name, communicator_cls)

accelerator_context = AcceleratorContext(torch_module_name, communicator_cls)

stephanie-wang · 2025-05-13T18:21:58Z

python/ray/experimental/channel/accelerator_context.py

+        if self._is_registered:
+            return self._torch_module_name
+        return None


I found this a bit confusing: I think it is odd that you can call register multiple times and end up with multiple AcceleratorContexts that have _is_registered=True.

Instead, how about removing the _is_registered flag and just having get_module_name. You can create a _global_custom_context along with the _default_accelerator_context, and AcceleratorContext.get() will return the custom context if set else it returns the default context. This makes it more explicit which context is registered or not and whether the context is custom or not.

I fully agree with your suggestion. I have implemented _global_custom_context following your design and provided a method is_accelerator_context_registered to check whether a context is registered. The code now looks much clearer. Thank you again for your valuable feedback!

stephanie-wang · 2025-05-13T18:22:33Z

python/ray/experimental/channel/communicator_handle.py

+
+class CommunicatorHandle:
+    """
+    A lightweight communicator handle used to store actors in the communicator.


Suggested change

A lightweight communicator handle used to store actors in the communicator.

A lightweight communicator handle used by the driver to store handles to the actors in the communicator.

hipudding · 2025-05-15T01:11:47Z

Thanks, this looks good! Please address the last few comments and then we can merge it.

Sorry for all the typos—I should have enabled local spell checking. Thank you for going through so many rounds of review and providing suggestions for improvement. I’ll fix these issues as soon as possible.

stephanie-wang · 2025-05-21T00:16:52Z

Looks like there are just some lint errors to fix.

This commit introduces multi-device support in Compile Graph, extending beyond CUDA's NCCL to accommodate various accelerators. The key changes include: 1. Removed dependency on `cupy.cuda.ExternalStream`, Replaced with `torch.{device}.StreamContext` to support diverse hardware. 2. Replaced hardcoded `torch.cuda.xxx` calls with `AcceleratorRuntime` Enables automatic detection and invocation of device-specific functions. Co-authored-by: hipudding <huafengchun@gmail.com> Signed-off-by: noemotiovon <757486878@qq.com>

hipudding · 2025-05-22T09:32:18Z

@stephanie-wang @ruisearch42 Hi, Could you please merge this PR? Thanks.

daiping8 · 2025-10-10T09:15:10Z

@hipudding Hello, this is a great piece of work that I can learn from. However, I have a few questions—would you mind helping me clarify them?

I couldn't find hccl_group.py. Was it removed from the commit because hccl_group and nccl_group are logically similar and contain a lot of duplicated code?
Is the current method for adding a new backend to CG still as described in your PR—implementing an _XcclGroup class similar to _NcclGroup, and registering it via register_accelerator_context?

Thank you for your time.

hipudding · 2025-10-20T03:53:03Z

I couldn't find hccl_group.py. Was it removed from the commit because hccl_group and nccl_group are logically similar and contain a lot of duplicated code?

Yes, this part has removed and leave a register method to add your own backend. Currently, hccl backend can't add to Ray project because ray community has no Ascend servers and can't test it. I keep a backup of hccl and hccl_group, you can see 0482edc for an example.

Is the current method for adding a new backend to CG still as described in your PR—implementing an _XcclGroup class similar to _NcclGroup, and registering it via register_accelerator_context?

Yes, you're right, Use this register and enable your backend.

hipudding marked this pull request as draft March 3, 2025 11:47

hipudding force-pushed the cupystream branch 2 times, most recently from f63223c to 27215f3 Compare March 4, 2025 02:55

hipudding marked this pull request as ready for review March 4, 2025 06:16

hipudding changed the title ~~[WIP] Replacement for cupy.cuda.ExternalStream for hardware diversity~~ Replacement for cupy.cuda.ExternalStream for hardware diversity Mar 4, 2025

hipudding changed the title ~~Replacement for cupy.cuda.ExternalStream for hardware diversity~~ [aDAG] Replacement for cupy.cuda.ExternalStream for hardware diversity Mar 4, 2025

noemotiovon reviewed Mar 4, 2025

View reviewed changes

hipudding force-pushed the cupystream branch 2 times, most recently from a4d06c9 to 705a10d Compare March 4, 2025 11:18

jcotant1 added the core Issues that should be addressed in Ray Core label Mar 4, 2025

hipudding force-pushed the cupystream branch from 8ae23d2 to 6e72a0b Compare March 5, 2025 03:18

hipudding changed the title ~~[aDAG] Replacement for cupy.cuda.ExternalStream for hardware diversity~~ [aDAG] Change for acclerator diversity and add HCCL communication lib Mar 5, 2025

hipudding changed the title ~~[aDAG] Change for acclerator diversity and add HCCL communication lib~~ [Compiled Graph] Change for acclerator diversity and add HCCL communication lib Mar 5, 2025

hipudding force-pushed the cupystream branch from a261a94 to a89c1e6 Compare March 6, 2025 00:51

ruisearch42 reviewed Mar 6, 2025

View reviewed changes

ruisearch42 mentioned this pull request Mar 6, 2025

[ADAG]Enable NPU (hccl) communication for CG #47658

Closed

8 tasks

hipudding force-pushed the cupystream branch 2 times, most recently from 1e80d9f to d682702 Compare March 6, 2025 08:03

hipudding changed the title ~~[Compiled Graph] Change for acclerator diversity and add HCCL communication lib~~ [Compiled Graph] Enhance Compile Graph with Multi-Device Support and HCCL Integration Mar 7, 2025

hipudding force-pushed the cupystream branch from 01bd0c4 to 18cd7e3 Compare March 7, 2025 01:59

hipudding requested review from noemotiovon and ruisearch42 March 7, 2025 02:32

hipudding force-pushed the cupystream branch from 18cd7e3 to 8bcfd7a Compare March 7, 2025 03:11

hipudding requested a review from a team as a code owner March 7, 2025 06:26

hipudding force-pushed the cupystream branch from 20b28c8 to 1a5988e Compare March 7, 2025 06:34

hipudding force-pushed the cupystream branch 6 times, most recently from 61d9600 to d6d4b76 Compare April 29, 2025 02:24

hipudding requested a review from noemotiovon April 29, 2025 02:27

noemotiovon approved these changes Apr 29, 2025

View reviewed changes

hipudding requested review from ruisearch42 and stephanie-wang April 29, 2025 11:42

stephanie-wang reviewed May 7, 2025

View reviewed changes

stephanie-wang approved these changes May 13, 2025

View reviewed changes

Yikun mentioned this pull request May 15, 2025

[Usage]: 4*910B2 部署deepseek r1/v3 报错 vllm-project/vllm-ascend#485

Closed

wangxiyuan mentioned this pull request May 17, 2025

[Bug]: Qwen3-235B cannot be run successfully with vllm v1 engine on version 0.8.5rc1 vllm-project/vllm-ascend#781

Open

stephanie-wang approved these changes May 20, 2025

View reviewed changes

eicherseiji mentioned this pull request May 23, 2025

Release test llm_serve_correctness failed #53267

Closed

hipudding mentioned this pull request May 29, 2025

[Compiled Graph] Enhance Compile Graph with Multi-Device Support #53395

Merged

8 tasks

	def get_communicator(self, args, *kwargs):
	def create_communicator(self, args, *kwargs):

	Retrieves the communication group for collective operations.
	Creates a communication group for collective operations.

	# set custom communicator on all actors
	# Set custom communicator on all actors.

	This eincludes stram management, event creation, device context control,
	This includes stream management, event creation, device context control,

	def __init__(self, torch_module_name: str, commumcator_cls: Type[Communicator]):
	def __init__(self, torch_module_name: str, communicator_cls: Type[Communicator]):

	commumcator_cls: Class used to handle communication.
	communicator_cls: Class used to handle communication.

	Sets the accelerator context to the default context.
	Overwrites the default accelerator context.

	accleerator_context = AcceleratorContext(torch_module_name, communicator_cls)
	accelerator_context = AcceleratorContext(torch_module_name, communicator_cls)

	A lightweight communicator handle used to store actors in the communicator.
	A lightweight communicator handle used by the driver to store handles to the actors in the communicator.

Conversation

hipudding commented Mar 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

How to add a new backend for CG? here's an example for Ascend NPU:

Related issue number

Checks

Uh oh!

hipudding commented Mar 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hipudding Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ruisearch42 commented Mar 5, 2025

Uh oh!

hipudding commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ruisearch42 commented Mar 5, 2025

Uh oh!

ruisearch42 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hipudding Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hipudding Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hipudding Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hipudding Mar 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hipudding commented Mar 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

noemotiovon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Not registered yet.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hipudding commented May 6, 2025

Uh oh!

ruisearch42 commented May 7, 2025

hipudding commented Mar 3, 2025 •

edited

Loading

hipudding Mar 4, 2025 •

edited

Loading

hipudding commented Mar 5, 2025 •

edited

Loading

hipudding Mar 6, 2025 •

edited

Loading

hipudding Mar 6, 2025 •

edited

Loading

hipudding Mar 6, 2025 •

edited

Loading

hipudding Mar 6, 2025 •

edited

Loading

hipudding commented Mar 7, 2025 •

edited

Loading