This PR is a follow up of #5631

Last successful run of the new docs GHA workflow could be seen at https://github.com/martin-g/arrow-datafusion/actions/runs/4479933269/jobs/7874551780
Content of asf-site branch in my fork: https://github.com/martin-g/arrow-datafusion/tree/asf-site

TODOs for the Datafusion committers:

1. create asf-site branch in arrow-datafusion repo
2. delete the datafusion folder in arrow-site repo

CC @kou @alamb

1. create asf-site branch in arrow-datafusion repo

Done.

I tested this PR out in my fork and it works great! Thank you @martin-g

Here https://github.com/alamb/arrow-datafusion/tree/asf-site

What are the next steps after this PR to actually get this site served?

On my fork I configured github pages:

And it published https://github.com/alamb/arrow-datafusion/actions/runs/4483635485 which is cool

But the site looks pretty bad https://alamb.github.io/arrow-datafusion

2. delete the datafusion folder in arrow-site repo

Here is a PR to do this apache/arrow-site#336

However, when I check out the site locally and open it, it works great 👍

So I think that means the issue is with how github pages is publishing it

We need to add .nojekyll to the asf-site like https://github.com/apache/arrow-site/blob/main/.github/workflows/deploy.yml#L112 to use GitHub Pages for preview.
If we don't have .nojekyll, our asf-site contents are processed by Jekyll. Jekyll ignores _*. So _static/styles/* aren't published on GitHub Pages.

It would be nice to sort out how this content will be hosted (though maybe I just don't understand how the process works)

ASF provides https://arrow.apache.org/ (not GitHub Pages) that serves the contents in https://github.com/apache/arrow-site/tree/asf-site .
With the .asf.yaml configuration in this PR, https://arrow.apache.org/datafusion/ uses https://github.com/apache/arrow-datafusion/tree/asf-site instead of https://github.com/apache/arrow-site/tree/asf-site/datafusion .

I tested the latest and greatest code on my fork and it seems to have worked great! https://alamb.github.io/arrow-datafusion/

Thank you so much @martin-g and @kou

I am going to merge this PR and see how it goes!

I am happy to report that the first run of https://github.com/apache/arrow-datafusion/actions/runs/4488868585/jobs/7893990469 has worked

And the content appears to be on https://github.com/apache/arrow-datafusion/tree/asf-site 👍

I am going to test making changes to the datafusion site and verify they appear on https://arrow.apache.org/datafusion/

My test shows that the process is working : #5500 (comment)

Thanks again @kou and @martin-g -- this is really great.

PR to update release process: #5687