Deprecate REST Api and Replace with Python share script #23259
Description
The REST API introduces functionality that is redundant with the JSON RPCs, but it's designed to be an unauthenticated connection (but still trusted). This is a vauge security model and introduces additional code into bitcoin core.
Instead, it might make sense to deprecate the REST Interface and replace it with a share python script that gives a similar interface. This server connection can receive a whitelistrpc'd credential that only enables the rest API queries. The python script can detect if e.g. twisted is available and run with that or use python3 built in http.server.
One justification for maintaining it's existence in present form is that I have heard is that performance is better than the JSON RPCs.
In terms of deprecation, one path would be (over a few releases):
Phase 1: Marking Deprecated + Adding Python Script Server
Phase 2: Making compilation of rest optional, default disabled
Phase 3: Total Removal
irc log discussion from #bitcoin-core-dev more generally about moving http outside of core
[2021-09-22T15:55:55.479Z] <jeremyrubin> I think what we'd want is to remove the http support from Bitcoin core and then build a separate binary with a simple http proxy to it
[2021-09-22T15:56:31.719Z] <jeremyrubin> That doesn't even have to live in core.
[2021-09-22T15:56:45.905Z] <sipa> that's also a possibility, but probably not a short-term one
[2021-09-22T15:56:55.548Z] <sipa> (also, unix domain sockets don't exist on windows...)
[2021-09-22T15:56:59.655Z] <jeremyrubin> It would be a breaking change so would be a maj release
[2021-09-22T15:57:15.533Z] <jeremyrubin> I think they have something that can be used the same way these days
[2021-09-22T15:57:24.404Z] <sipa> right, but we'd need that too :)
[2021-09-22T15:57:58.032Z] <jeremyrubin> I think it's basically the exact same
[2021-09-22T15:58:41.309Z] <darosior> Windows does support Unix Domain Sockets since W10
[2021-09-22T15:58:45.306Z] <jeremyrubin> Just different headers and negative FDs
[2021-09-22T16:06:49.684Z] <jeremyrubin> I think over the span of a few releases could reasonably do deprecated, not built by default, removed.
[2021-09-22T16:30:01.771Z] <laanwj> yes, modern windows has them too
[2021-09-22T16:36:51.958Z] <laanwj> i might give 'simple line protocol over UNIX socket' a try, if there is really enthousiasm for adding such a thing; at least i don't have to fight libevent's http client then, that's what i lost interest in
[2021-09-22T16:43:05.850Z] <laanwj> i guess the only context information from HTTP that is used inside RPC calls is the wallet name from the path, in case of wallet RPC calls
[2021-09-22T16:44:50.909Z] <laanwj> but would be straightforward to wrap in an object that provides the context otherwise, e.g. { 'wallet': 'name', 'params': {...} }
[2021-09-22T16:57:06.615Z] <jeremyrubin> i guess the only other question is the rest API thing?
[2021-09-22T16:58:39.695Z] <jeremyrubin> that also seems like stuff that should just be a proxy over core tho
[2021-09-22T16:58:58.789Z] <jeremyrubin> (or just removed and see if anyone complains?)
[2021-09-22T17:01:57.578Z] <laanwj> rest is very closely bound to http, how i imagine this would purely be for RPC calls (all information on rest is also available through RPC it's 'extra')
[2021-09-22T17:02:59.063Z] <laanwj> i think that could very well be an optional thing that people can enable if they want it
[2021-09-22T17:03:56.020Z] <laanwj> sharing the http server between rest and the json-rpc interface was always a bit questionable, they are separate things
[2021-09-22T17:05:22.061Z] <jeremyrubin> laanwj does anyone use the rest interface in practice?
[2021-09-22T17:05:56.144Z] <laanwj> i have no idea
[2021-09-22T17:06:39.211Z] <jeremyrubin> i've never heard of anyone using it either
[2021-09-22T17:06:45.038Z] <laanwj> it's much faster than JSON-RPC for some things as it saves on formatting and the info can (basically) directly be dumped to the socket
[2021-09-22T17:10:11.984Z] <laanwj> though it's not as great as it sounds, i wanted to add a rest call for streaming utxo data once (without buffering it in memory first), but libevent's http server wouldn't let me due to not really being thread safe
[2021-09-22T17:12:58.334Z] <laanwj> mostly it's not even issues with http itself we run against, but issues with libevent's http, but adding another dependency for http would be not nice
[2021-09-22T17:17:50.893Z] <jeremyrubin> laanwj i think given that there are few users for rest (most likely) we could ship a /share script with a python http proxy for rest, altho performance might suffer a bit (maybe we add a raw response mode for the json rpcs if it's a concern?)
[2021-09-22T17:20:19.972Z] <laanwj> yes...
[2021-09-22T17:22:52.016Z] <laanwj> in the case where we directly 'own' the socket, and the protocol is up to us to define, in principle it could reply in a different way than a line with JSON on it if so requested, this does make clients that want to support that more complex, so everything should at least support JSON line in JSON line out
[2021-09-22T17:24:11.086Z] <laanwj> honestly that kind of thing seems far enough away, i'm not sure it's worth spendng time on, it works as it does now, not sure about the added benefit of doing things slightly different
[2021-09-22T17:25:23.066Z] <laanwj> i think my reply on twitter was more hypothetical, in a perfect clean-slate world where you don't have to balance so many different concerns and backwards compatibility, it would have been nice