This issue is an "epic" that describes what we are aiming for through the 21.1 cycle and beyond.

Summary: in v21.1 we are going to:

• sunset the /_admin/v1, and /_status URL prefixes for consumption by end-users. In particular /_status will be eventually removed e.g. in v21.2.
• instead, introduce a new /api/v2 URL prefix under which all public services will be available. From then on, all API additions for end-users will go under /api and follow our general API versioning rules. It will be possible for multiple versions to co-exist side-by-side to offer forward and backward compatibility to clients.
• The Admin UI will continue to get all the information it needs under /_admin which will not be versioned and will evolve in lock-step with the UI code, for simplicity by CockroachDB developers. There will be no guarantees offered to external users about the behavior of /_admin. We will also migrate anything under /_status and /debug used by the admin UI to live under /_admin.
• if a service is useful to both the admin UI and end-users, then it will have two URLs: one under /api and one under /_admin. Only the former will be public and documented.
• /debug will continue to be a special "anything goes" so that CockroachDB developers can quickly add or change troubleshooting endpoints there. However, any debug endpoint useful for the admin UI will also have an entry point under /_admin and any debug endpoint with value to end-users will need to be "promoted" under the /api tree.
• /health will also continue to be special and will not change URLs nor behavior in this cycle. However, we're going to attempt to pull it out of mandatory TLS, so that non-TLS HTTP clients can check health too. (This is a separate project)

Internal changes

We acknowledge that all our HTTP APIs today are also not used internally by RPC clients inside CockroachDB; and conversely all RPC services are otherwise not used by HTTP clients. Therefore:

• we are going to bring all the current server methods for HTTP endpoints under a HTTP server directly.
• we are going to disconnect the RPC services from the gRPC-HTTP gateway.

This will greatly reduce the complexity of the server code, and also ensure that our RPC and HTTP authentication mechanisms are clearly separate.

Then at a later phase (see separate issue TBD) connect the SQL layer to the (new) HTTP methods via the new HTTP-over-SQL gateway (new project/component).

Possible strategy (needs to be discussed / refined further)

1. add a "native" Go http method dispatcher to the server code.
2. exercise step (1) by seeding the /api/v1 tree with just a few of the APIs we already know are popular and need to be provided:
   • list of nodes in cluster
   • list of hot ranges
3. concurrently with steps 1 and 2, adapt the admin UI client code to only query endpoints under /_admin (i.e. bring all uses of /_status under /_admin, and add URL aliases accordingly
4. concurrently with steps 1-3, initiate a documentation project with docs to explain this transition
5. fix issue server: introduce versioning and pagination in the HTTP APIs #54022 for the few new APIs in step (2) by creating example pagination and some Go code skeletons that all future API additions can follow / reuse for pagination.
6. exercise (5) by incrementally adding more complex APIs to check that the pagination is robust. Starting with:
   • list of jobs
   • list of log entries
7. concurrently with (3)-(6) work on the new HTTP-over-SQL gateway so that SQL code can also query the public APIs.

Related wiki pages:

• API Design guidelines (needs to be made public eventually)
• HTTP-over-SQL gateway (needs to be made public eventually)

Related github issues:

• server, ui: Merge Admin and Status endpoints server, ui: Merge Admin and Status endpoints #24989
• server: introduce versioning and pagination in the HTTP APIs server: introduce versioning and pagination in the HTTP APIs #54022
• server: the HotRangesResponse contains raw RangeDescriptors with sensitive data server: the HotRangesResponse contains raw RangeDescriptors with sensitive data #53212

Jira issue: CRDB-3609