…1516)

UCXX is becoming the default UCX communicator via a proxy backend with rapidsai/rapids-dask-dependency#116. This proxy backend now points `protocol="ucx"` to UCXX provided distributed-ucxx is installed, otherwise fallback to UCX-Py and emit a warning, and while UCX-Py is not archived provide a `protocol="ucx-old"` as fallback. This change ensures the proper (UCXX or UCX-Py) is initialized based on the specified protocol and whether distributed-ucxx is installed or not.

Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)

URL: #1516

Switch the default distributed-ucxx progress mode to `"blocking"`, which is equivalent to UCX-Py's default progress mode, allowing a more transparent move of Dask to UCXX. This change may later be reverted once we have more confidence in a successful transition to UCXX.

Additionally allow both `"ucx://"` and `"ucxx://"` address prefixes, since with the change from rapidsai/rapids-dask-dependency#116 both will be valid to enable UCXX when distributed-ucxx is installed.

Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)
  - Tom Augspurger (https://github.com/TomAugspurger)

URL: #453

With changes from rapidsai/rapids-dask-dependency#116, `protocol="ucx"` now uses UCXX when `distributed-ucxx` is installed as is the case in CI. Some tests broke because of the conditionals to validate UCX-Py vs UCXX, this change fixes those that are currently failing in CI, the remaining tests will be addressed in a follow-up PR.

Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Lawrence Mitchell (https://github.com/wence-)

URL: #1518

rapidsai/rapids-dask-dependency#116 changes the
meaning of "ucx". To restore the old behavior, we use "ucx-old".

ucx-py, which the protocol `ucx-old` uses, is being deprecated in favor
of `ucxx` (rapidsai/ucx-py#1151) so we could
also consider removing this job / tests.

rapidsai/rapids-dask-dependency#116 changes the meaning of `protocol="ucx"`. To restore the old behavior, we use `"ucx-old"`.

UCX-Py, which the protocol `"ucx-old"` uses, is being deprecated in favor of UCXX (rapidsai/ucx-py#1151) and is scheduled for removal in 25.10, allowing us to then remove UCX-Py tests entirely.

Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Corey J. Nolet (https://github.com/cjnolet)
  - Bradley Dice (https://github.com/bdice)

URL: #2743

rapidsai/rapids-dask-dependency#116 changes the meaning of "ucx". To restore the old behavior, we use "ucx-old".

ucx-py, which the protocol `ucx-old` uses, is being deprecated in favor of `ucxx` (rapidsai/ucx-py#1151) so we could also consider removing this job / tests.

Authors:
  - Tom Augspurger (https://github.com/TomAugspurger)

Approvers:
  - Simon Adorf (https://github.com/csadorf)
  - Bradley Dice (https://github.com/bdice)

URL: #7008

Rewrite the UCXX/UCX-Py proxy backends added in #116 to deprecate UCX-Py. We have already removed the UCX-Py backend from Distributed in dask/distributed#9105, but while there's no release we cannot rely on that, so the changes in the PR work in the same way we expect when the migration is completed and Distributed has a new release. If distributed-ucxx is not installed a warning and exception are raised telling the user to install it.

The state after this PR is as follows:

1. If distributed-ucxx is installed:
    1. `protocol="ucx"` / `protocol="ucxx"` should continue working as before, both were already using distributed-ucxx
    2. `protocol="ucx-old"` (only exists in rapids-dask-dependency added in #116) will raise the same `FutureWarning` as `protocol="ucx"` without RAPIDS
2. If distributed-ucxx is NOT installed:
    1. `protocol="ucx"` / `protocol="ucxx"` / `protocol="ucx-old"` all raise a `FutureWarning` and don’t let users create processes.

The final state is expected to be achieved when a new Distributed release is cut and the `__rdd_patch_ucx.py` file is removed from rapids-dask-dependency, which will be:

1. If distributed-ucxx is installed:
    1. `protocol="ucx"` (now transparently uses distributed-ucxx) / `protocol="ucxx"` (always used distributed-ucxx) should continue working as before
2. If distributed-ucxx is NOT installed:
    1. `protocol="ucx"` raises a `FutureWarning` and doesn’t let users create processes
    2. `protocol="ucxx"` never existed before distributed-ucxx so it will just fail with a wrong protocol name

Specifically I've tested the following to confirm what's behavior matches what is described above as the state of this PR (using all `ucx`/`ucxx`/`ucx-old` combinations):

1. `dask scheduler --protocol=ucx`
2. `dask worker ucx://127.0.0.1:8786`
3. `LocalCluster(protocol="ucx", silence_logs=False)`
4. `LocalCUDACluster(protocol="ucx", silence_logs=False)`

Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - Tom Augspurger (https://github.com/TomAugspurger)

URL: #120