When using rllib and turning on evaluation, rllib tries to convert pytorch cuda tensor to numpy and fails with exception "TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first."

Simple reproduced script:

from ray import tune
from ray.rllib.agents.dqn import ApexTrainer


ray.init()


tune.run(
    ApexTrainer,
    stop={"episode_reward_mean": 20000},
    config={
        "env": "SpaceInvaders-v0",
        "num_gpus": 1,
        "num_workers": 16,
        "use_pytorch": True,
        "evaluation_interval": 1,
    },
)

Is that really a bug or am I doing something wrong? Any workarounds?