The problem is the repo(bufbuild/buf), doesn't publish Linux-arm64 package.

When going though their build script, I stumbled upon this changes.
https://github.com/bufbuild/buf/pull/323/files

Looks like for Linux-arm64 package, it instead builds Linux-aarch64 package.

And the setup action also will need to make similar change in the code base if this is to work.