This avoids importing pyarrow unconditionally in dataframe.backends, which is reachable in some dask.array workloads (notably, those using xarray, which use pandas.Index classes, tokenization of which requires importing this module; however that class of users doesn't depend on dask[dataframe] and so may not have the pyarrow dependency.

To accomplish this, we need to reorganize the root of the dask.dataframe package a bit. I've moved some uses of pandas in dask.dataframe._compat to dask._pandas_compat, which is outside of the dask.dataframe subpackage.

This lets users of just pandas, but not dask.dataframe, (like tokenization registration for pandas objects) import from dask._pandas_compat, while avoiding all of dask.dataframe.

Closes #12072

At the moment, I've only tested this manually using the reproducer from #12072. I don't have a unit test yet. I started with

@pytest.mark.filterwarnings("ignore:Passing an object to dask.array")
def test_tokenize_without_pyarrow(monkeypatch: pytest.MonkeyPatch) -> None:
    # https://github.com/dask/dask/issues/12072
    packages = list(sys.modules)
    for package in packages:
        if package.startswith(("pyarrow", "dask")):
            monkeypatch.delitem(sys.modules, package)

    monkeypatch.setitem(sys.modules, "pyarrow", None)

    y = xr.DataArray(
        data=da.zeros((1)),
        dims=('y',),
        coords={'y': np.arange(1)},
        name='foo'
    ).to_dataset()
    y['foo'] = y.foo.dims, y.foo.data + 1
    dask.optimize(y)

which reproduces the issues. However, the trick of messing with sys.modules to force re-importing modules / import errors doesn't play nicely with dask's import-time tokenization registration, and so we end up in a weird state.