All CLA requirements met.

Otherwise, LGTM. Thanks!

cc @janvorli

@jessicah, is this still valid? I was testing it locally today and found this:

$ wget -qO- \
    --post-data='{"jsonrpc":"2.0","method":"getPkg","params":[{"name":"'gmp'","repositorySourceCode":"haikuports_x86_64","versionType":"LATEST","naturalLanguageCode":"en"}],"id":0}' \
    --header='Content-Type:application/json' https://depot.haiku-os.org/__api/v1/pkg

{"jsonrpc":"2.0","id":0,"error":{"code":-32601,"message":"method not found"}}

According to the linked docs in first commit https://depot.haiku-os.org/__docs/api.html#d0e1599, The JSON-RPC API is known as "version 1" and the RPC-over-HTTP is known as "version 2". The latter will eventually completely replace the former. and they have removed getPkg. I checked on google cache that it was there a few months ago, now section 13.1 is updated.

BTW, if these kind of bold/merciless breaking changes are inevitable on Haiku servers, we can simplify this workflow by uploading the distilled version of these few packages on some GitHub repo's releases (e.g. https://github.com/illumos/sysroot/releases/tag/20181213-de6af22ae73b-v1 is where we get illumos archive) and pull it from there in build-rootfs script. This is an internal script and its real usage is to create docker images for dotnet CI, those docker images always have the distilled versions of installed software (the docker layers do not rebuild unless someone touches the corresponding Docker file).

@am11 The code mentioned above is mine.
I've tested it on my machine a month ago, but I'm not on my laptop now. Will try to fix this a few hours later when I come back.

on some GitHub repo's releases

Do they have to be "official" repos of Haiku?

Do they have to be "official" repos of Haiku?

Ideally, official repo would be a good place, but I think it is not necessary. I will defer to @jkotas and @janvorli for a conclusive answer. For example, if you are going to work on the .NET port for a foreseeable future, then you can upload the required artifacts in your dotnet/arcade repo fork's release page.

For Haiku servers, breaking change disregarding their own API version is really a bad idea. What's worse is not only v2 became v1 but getPkg method was completely dropped off. This infra rely on permalinks and trustable sources and expects upstreams to cooperate and play nice. We had issues before in https://github.com/dotnet/dotnet-buildtools-prereqs-docker/ (where this script is used), which blocked build for other platforms as well.

If official repos are not required, then @jessicah can you update the krb5 and libunwind hpkgs to the packages folder on your GitHub repo?

And... If possible, can you say something about the server to the devs since you're a member of Haiku...

Replacing with this:

wget -qO-     \
--post-data='{"name":"'gmp'","repositorySourceCode":"haikuports_x86_64","versionType":"LATEST","naturalLanguageCode":"en"}' \
--header='Content-Type:application/json' https://depot.haiku-os.org/__api/v2/pkg/get-pkg

works.

@jessicah If v2 turns out to be stable, then your branch for this pull request should be reset to this.

Ah yeah, there was a recent update to the Haiku Depot server 3 days ago. I'll get some feedback from the dev about it.

@am11 would it be okay to say require jq? Using sed is personally just a little brittle, I'd rather do something like wget ... | jq -r '.result.versions[].hpkgDownloadURL'. Thoughts?

I've also had confirmation that this new API shouldn't change in future, so should be able to rely on it.

Yup, jq is fine (and recommended tool for parsing JSON on shell). We previously added jq with bash3 fallback in 0fa903a#diff-bdcb2ee87da0a73b7095bff4e13cac0563da00f8b93fec8a10d6de71b88fd3b1R78. In the next step; docker build, we will just need to make sure the image has jq available (either in base layer or the Haiku specific layer).

Yup, jq is fine (and recommended tool for parsing JSON on shell). We previously added jq with bash3 fallback in 0fa903a#diff-bdcb2ee87da0a73b7095bff4e13cac0563da00f8b93fec8a10d6de71b88fd3b1R78. In the next step; docker build, we will just need to make sure the image has jq available (either in base layer or the Haiku specific layer).

Seems like the function in the commit above tries to detect jq and then falls back to manually parsing JSON.
Should we do the same (try to find jq and if it fails, parse the JSON using sed)?

Using sed is personally just a little brittle, I'd rather do something like wget ... | jq -r '.result.versions[].hpkgDownloadURL'.

I also heard of jq when searching a solution for this but I didn't want to use it since it isn't a standard POSIX tool and is unavailable on many Linux distros.

I personally have not came across any Linux or Unix-y OS where jq is not available via package manager. :)

The good thing about build-rootfs.sh is that it is an internal script to create docker cross build images, and it is not supposed to be used directly beyond testing for those docker images. We do not care about wide portability. It is enough if it works on Ubuntu 18.04 / 20.04 which is the base layer OS of the docker file (see the example usage: dotnet/dotnet-buildtools-prereqs-docker#638)

Current revision looks good. Please resolve the merge conflicts. Thanks!

@akoeplinger, could you please take look?

I personally have not came across any Linux or Unix-y OS where jq is not available via package manager. :)

But it's not available by default. I can confirm for Ubuntu 22.04.

Also, browsing through the sources here for Ubuntu 22.04, none of the files seem to install jq.

Should something be done about that?

We can install stuff as we need. jq is something we can justify installing in base layer as it is recommended tool for JSON parsing in shell scripts. Otherwise we can add a RUN apt install -y jq line in Haiku layer, once we get there.

dotnet/dotnet-buildtools-prereqs-docker#643

Conflicts resolved :)

Looks like failed job needs retrying?