DEBUG log messages imply UCX config options are invalid #4845
Description
When debugging is enabled, distributed outputs the following messages:
distributed.comm.ucx - DEBUG - Key: cuda_copy with value: False not a valid UCX configuration option.
distributed.comm.ucx - DEBUG - Key: tcp with value: False not a valid UCX configuration option.
distributed.comm.ucx - DEBUG - Key: nvlink with value: False not a valid UCX configuration option.
distributed.comm.ucx - DEBUG - Key: infiniband with value: False not a valid UCX configuration option.
distributed.comm.ucx - DEBUG - Key: rdmacm with value: False not a valid UCX configuration option.
distributed.comm.ucx - DEBUG - Key: net-devices with value: None not a valid UCX configuration option.
distributed.comm.ucx - DEBUG - Key: reuse-endpoints with value: None not a valid UCX configuration option.
When the corresponding env vars as set, such as export DASK_UCX__CUDA_COPY=True, the code in ucx.py here seems to behave correctly, but the check here still fails and prints the debug messages.
This can be seen simply by setting:
export DASK_LOGGING__DISTRIBUTED="DEBUG"
and running the scheduler like so:
python -m distributed.cli.dask_scheduler
Since the default values even cause the debug messages to be printed, there's no need to set the UCX config options to see the debug messages above.