Skip to content

Add support for processing language server protocol messages#10025

Closed
yegappan wants to merge 1 commit intovim:masterfrom
yegappan:cmdexpand
Closed

Add support for processing language server protocol messages#10025
yegappan wants to merge 1 commit intovim:masterfrom
yegappan:cmdexpand

Conversation

@yegappan
Copy link
Member

@yegappan yegappan commented Mar 26, 2022

Add support for processing language server protocol encoded messages. The protocol is
described in: https://microsoft.github.io/language-server-protocol/specification.

The existing support in Vim for processing JSON encoded messages doesn't work
for language server protocol. The LSP messages start with a HTTP header followed
by a JSON encoded dictionary. This format is currently not supported by the Vim
channel "json" mode.

This PR adds a new "lsp" channel mode to process the LSP header. A LSP client
can use the ch_sendexpr() and ch_evalexpr() functions to send a Vim expression to
the LSP server and receive the decoded responses.

I have tested this with the Vim9 LSP client and it works.

This PR adds the features requested in #9390.

@codecov
Copy link

codecov bot commented Mar 26, 2022

Codecov Report

Merging #10025 (9b910d6) into master (2bdad61) will increase coverage by 0.01%.
The diff coverage is 81.60%.

❗ Current head 9b910d6 differs from pull request most recent head 9b521ee. Consider uploading reports for the commit 9b521ee to get more accurate results

@@            Coverage Diff             @@
##           master   #10025      +/-   ##
==========================================
+ Coverage   81.91%   81.93%   +0.01%     
==========================================
  Files         167      167              
  Lines      187325   187385      +60     
  Branches    42236    42254      +18     
==========================================
+ Hits       153448   153528      +80     
+ Misses      21531    21507      -24     
- Partials    12346    12350       +4     
Flag Coverage Δ
huge-clang-none 82.34% <78.22%> (+<0.01%) ⬆️
huge-gcc-none 82.68% <78.22%> (+<0.01%) ⬆️
huge-gcc-testgui 81.14% <78.22%> (+<0.01%) ⬆️
huge-gcc-unittests 2.01% <0.00%> (-0.01%) ⬇️
linux 83.92% <78.40%> (+<0.01%) ⬆️
mingw-x64-HUGE 0.00% <0.00%> (ø)
mingw-x64-HUGE-gui 77.18% <81.41%> (+<0.01%) ⬆️
windows 75.96% <80.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/channel.c 83.65% <79.46%> (-0.12%) ⬇️
src/job.c 90.47% <100.00%> (+0.01%) ⬆️
src/json.c 84.48% <100.00%> (+0.27%) ⬆️
src/profiler.c 84.07% <0.00%> (-0.75%) ⬇️
src/vim9script.c 87.60% <0.00%> (-0.11%) ⬇️
src/getchar.c 84.48% <0.00%> (-0.07%) ⬇️
src/window.c 87.98% <0.00%> (-0.07%) ⬇️
src/gui.c 73.94% <0.00%> (-0.05%) ⬇️
src/vim9type.c 88.05% <0.00%> (-0.04%) ⬇️
src/drawscreen.c 80.02% <0.00%> (-0.02%) ⬇️
... and 16 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2bdad61...9b521ee. Read the comment docs.

@yegappan yegappan force-pushed the cmdexpand branch 2 times, most recently from 9b910d6 to 7b23ee7 Compare March 28, 2022 03:32
@yegappan yegappan changed the title WIP: Add support for processing language server protocol messages Add support for processing language server protocol messages Mar 28, 2022
src/channel.c Outdated
// We find the end once, to avoid calling strlen() many times.
reader->js_end = reader->js_buf + STRLEN(reader->js_buf);

// skip the HTTP header
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic here looks a bit fragile as it assumes a fixed order of header fields (that's irrelevant according to RFC7320 3.2.2) and a fixed number of them (the LSP specification only states there's at least one entry "and that at least one header is mandatory").
A better solution would be to iterate line by line (terminated by \r\n) until an empty one is found and parsing/validating each header entry that's found.
Some validation of the parsed data is also needed as:

  • The Content-Length must match the effective payload length according to the HTTP spec and it can be easily used to check if the given request has been fully received, delaying the parsing otherwise.
When a Content-Length is given in a message where a message-body is allowed, its field value MUST exactly match the number of OCTETs in the message-body. HTTP/1.1 user agents MUST notify the user when an invalid length is received and detected.
  • Content-Type must contain utf8 or utf-8 to make the code more robust wrt encoding changes.

@yegappan yegappan force-pushed the cmdexpand branch 3 times, most recently from 41b4acb to 47ae5e5 Compare March 29, 2022 01:39
@vim-ml
Copy link

vim-ml commented Mar 29, 2022 via email

src/channel.c Outdated
// process the content length field (if present)
if ((p - line_start > 16)
&& STRNICMP(line_start, "Content-Length: ", 16) == 0)
payload_len = atoi((char *)line_start + 16);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is quite dangerous, a negative content length can and will wreak havock.
The use of atoi is also problematic, in case of overflow you may be invoking undefined behaviour, plus the error checking is quite lacking. This call can be replaced with the use of strtol (and proper error checking, don't forget the check for ERANGE) and a check to make sure the value is not negative.

Using unsigned variables for everything that represents a length is also a good idea.

src/channel.c Outdated
&& STRNICMP(line_start, "Content-Length: ", 16) == 0)
payload_len = atoi((char *)line_start + 16);

if ((line_start + 2) == p && line_start[0] == '\r' &&
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor nit, make this consistent with the check above.

Suggested change
if ((line_start + 2) == p && line_start[0] == '\r' &&
if ((p - line_start) == 2 && line_start[0] == '\r' &&

hdr_len = p - reader->js_buf;

// if the entire payload is not received, wait for more data to arrive
if (jsbuf_len < hdr_len + payload_len)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parser should only consume payload_len bytes from the request, the remaining data shall be interpreted as the beginning of a new one.

src/json.c Outdated
garray_T lspga;

ga_init2(&ga, 1, 4000);
json_encode_gap(&ga, val, 0);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If json_encode_gap fails an empty payload is sent, the error should be relayed to the caller.

@yegappan yegappan force-pushed the cmdexpand branch 2 times, most recently from e01c494 to ea03d85 Compare March 29, 2022 14:35
@vim-ml
Copy link

vim-ml commented Mar 29, 2022 via email

@vim-ml
Copy link

vim-ml commented Mar 29, 2022 via email

@yegappan yegappan force-pushed the cmdexpand branch 3 times, most recently from 05e0b0e to f0ba3ab Compare March 30, 2022 01:58
@prabirshrestha
Copy link

Was about to play around and noticed that no release has been created for the past 7 days at https://github.com/vim/vim-win32-installer/releases.

https://ci.appveyor.com/project/chrisbra/vim-win32-installer/builds/43143548/job/okn2ow3fvro3kxvj
Failures: 
	From test_channel.vim:
	Found errors in Test_channel_lsp_mode():
	Run 1:
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[459]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 104: Expected [{'jsonrpc': '2.0', 'result': {'method': 'echo', 'jsonrpc': '2.0', 'id': 110, 'params': {'s': 'msg-with-id'}}}] but got []
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[459]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 110: Expected {'id': 14, 'jsonrpc': '2.0', 'result': 'extra-hdr-fields'} but got ''
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[459]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 119: Expected {'id': 16, 'jsonrpc': '2.0', 'result': 'alive'} but got ''
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[459]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 127: Expected {'id': 18, 'jsonrpc': '2.0', 'result': 'alive'} but got ''
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[459]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 135: Expected {'id': 20, 'jsonrpc': '2.0', 'result': 'alive'} but got ''
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[459]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 143: Expected {'id': 22, 'jsonrpc': '2.0', 'result': 'alive'} but got ''
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[459]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 151: Expected {'id': 24, 'jsonrpc': '2.0', 'result': 'alive'} but got ''
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[459]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 171: Expected [{'jsonrpc': '2.0', 'result': {'method': 'echo', 'jsonrpc': '2.0', 'params': {'s': 'no-callback'}}}] but got []
	Run 2:
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[492]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 104: Expected [{'jsonrpc': '2.0', 'result': {'method': 'echo', 'jsonrpc': '2.0', 'id': 110, 'params': {'s': 'msg-with-id'}}}] but got []
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[492]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 110: Expected {'id': 14, 'jsonrpc': '2.0', 'result': 'extra-hdr-fields'} but got ''
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[492]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 119: Expected {'id': 16, 'jsonrpc': '2.0', 'result': 'alive'} but got ''
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[492]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 127: Expected {'id': 18, 'jsonrpc': '2.0', 'result': 'alive'} but got ''
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[492]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 135: Expected {'id': 20, 'jsonrpc': '2.0', 'result': 'alive'} but got ''
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[492]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 143: Expected {'id': 22, 'jsonrpc': '2.0', 'result': 'alive'} but got ''
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[492]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 151: Expected {'id': 24, 'jsonrpc': '2.0', 'result': 'alive'} but got ''
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[492]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 171: Expected [{'jsonrpc': '2.0', 'result': {'method': 'echo', 'jsonrpc': '2.0', 'params': {'s': 'no-callback'}}}] but got []
	Run 3:
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[492]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 104: Expected [{'jsonrpc': '2.0', 'result': {'method': 'echo', 'jsonrpc': '2.0', 'id': 110, 'params': {'s': 'msg-with-id'}}}] but got []
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[492]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 110: Expected {'id': 14, 'jsonrpc': '2.0', 'result': 'extra-hdr-fields'} but got ''
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[492]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 119: Expected {'id': 16, 'jsonrpc': '2.0', 'result': 'alive'} but got ''
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[492]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 127: Expected {'id': 18, 'jsonrpc': '2.0', 'result': 'alive'} but got ''
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[492]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 135: Expected {'id': 20, 'jsonrpc': '2.0', 'result': 'alive'} but got ''
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[492]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 143: Expected {'id': 22, 'jsonrpc': '2.0', 'result': 'alive'} but got ''
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[492]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 151: Expected {'id': 24, 'jsonrpc': '2.0', 'result': 'alive'} but got ''
	command line..script C:/projects/vim-win32-installer/vim/src/testdir/runtest.vim[492]..function RunTheTest[44]..Test_channel_lsp_mode[1]..RunServer[21]..LspTests line 171: Expected [{'jsonrpc': '2.0', 'result': {'method': 'echo', 'jsonrpc': '2.0', 'params': {'s': 'no-callback'}}}] but got []
	Flaky test failed too often, giving up
TEST FAILURE 
NMAKE : fatal error U1077: 'if' : return code '0x1'
Stop.
Command exited with code 1

@vim-ml
Copy link

vim-ml commented Apr 8, 2022 via email

@prabirshrestha
Copy link

I'm able to download the latest version of vim with lsp support on Windows. Thanks for the quick fix. ArchLinux vim also seems to be updated to include this patch. Looking forward to seeing plugin authors use this feature.

I have started adding support for native lsp client in vim-lsp and I'm very happy with the simplicity and ease of use of the api.

I have just got basic request/response working so this feedback is a bit limited while playing with the api for several mins.

  1. LSP support $/cancelRequest which requires sending the request id. This is heavily used by vim-lsp in order to cancel previous requests such as completion results that could had changed, highlighting current keyword that could had changed due to the cursor movements. Doc explicitly says Note that in the request message the 'id' field should not be specified. Is it possible to have an api to either set the id or get the id of the request?
  2. Is it possible to document what resp is in the example doc. let resp = ch_evalexpr(ch, req, #{timeout: 100}). I also see this being used in the test and it checks for empty object. I thought this was the request id but always seem to be empty during my limited test. Probably this only makes sense for synchronous request? It would still be good to document that it will always be empty for async as it created a bit confusion to me.
  3. Minor: Should the doc include the full raw request? {'jsonrpc': '2.0', 'id': 1, 'result': { ...}}. I see this as useful doc when writing remote LSP/Json RPC plugins.
  4. It took be a bit to figure out the callback args for ch_evalexpr but seems like this is already documented at the end where it says to refer to |channel-callback|. One thing that wasn't clear is if there is a JSON RPC Error Message would it still call the callback function? Clarifying that caller needs to handle the message whether it is success or error rpc messages would be better here. In the future I can see having on_success and on_error callbacks. I don't see me using this in vim-lsp but do see it using in other plugins. https://www.jsonrpc.org/specification#response_object
  5. Can the sync and async method be used on the same channel? Because vim doesn't support async BufWritePre we recommend users to use the following code to auto format on save. While 99% of the code will be using async function there will be sync methods for writing.
let g:lsp_format_sync_timeout = 1000
autocmd! BufWritePre *.rs,*.go call execute('LspDocumentFormatSync')
  1. Is it possible to document the behavior of how to handle timeouts? When using let resp = ch_evalexpr(ch, req, #{timeout: 100}) what error is thrown and what error should I handle?

@prabirshrestha
Copy link

I figured out how to get the id, but seems like the request and response id are not the same. req id seems to always be +1 of response id.

let req = { 'method': 'initialize', 'params': {...} }
call ch_sendexpr(ch, req, { 'callback': function('s:cb', [req] })
call s:log(req) " req => { "method": "initialize", "jsonrpc": "2.0", "id": 2, "params": {} }

function! s:cb(req, ch, res) abort
   " req => { "method": "initialize", "jsonrpc": "2.0", "id": 2, "params": {} }
   " res => { "id": 1, "result": {}, "jsonrpc": "2.0" }
   call s:log(req, res) 
endfunction

@prabirshrestha
Copy link

Seems like bug in my code. Ids are generated correctly but would be good to document that id in request will be added.

@vim-ml
Copy link

vim-ml commented Apr 15, 2022 via email

@vim-ml
Copy link

vim-ml commented Apr 15, 2022 via email

@vim-ml
Copy link

vim-ml commented Apr 16, 2022 via email

@prabirshrestha
Copy link

Thanks for the quick updates on docs.

I'm working on a native-lsp-client branch in vim-lsp but not able to get completion items working. There could be a potential parsing bug.

Here is the link to the full channel log: https://gist.github.com/prabirshrestha/74d69fe9df0a85126727e99d09ddecbb

I see a bunch of decoding failed in the messages.

 25.979927 ERR on 0: Decoding failed - discarding input
 26.147193 : looking for messages on channels
 26.147276 on 0: Incomplete message (126 bytes) - wait 100 msec for more
 26.147304 : SafeState: back to waiting, triggering SafeStateAgain
 26.147323 : looking for messages on channels
 26.147340 on 0: still waiting on incomplete message
 26.147359 : SafeState: back to waiting, triggering SafeStateAgain
 26.147378 on 0: still waiting on incomplete message
 26.147396 on 0: still waiting on incomplete message
 27.438976 : raw key input: "�[O"
 27.439098 : looking for messages on channels
 27.439125 on 0: timed out
 27.439140 ERR on 0: Decoding failed - discarding input

The language server is returning the completion items, but since the parsing fails I can never get the message.

The repository I'm trying is a simple hello world rust program created via cargo new helloworld and try to type and see the completion items.

I also see timed out messages, Do we need to configure some timeout options here?

In one of the logs I see partial messages. https://gist.github.com/prabirshrestha/74d69fe9df0a85126727e99d09ddecbb#file-channel-log-L252

@vim-ml
Copy link

vim-ml commented Apr 17, 2022 via email

@prabirshrestha
Copy link

Glad that you have a repro. Do watch out for perf issues though this can be fixed along the way in the near future while the parsing bug is fixed. In vim-lsp we have a pretty bad perf due to parsing where we concat strings and this causes lot of memory allocation.

This threads might be relevant:

You might also be interested in how vscode lang server implements it. They use a MessageBuffer and MessageReader and directly reading as bytes with allocNative.

@vim-ml
Copy link

vim-ml commented Apr 17, 2022 via email

@prabirshrestha
Copy link

I have tried the PR and it correctly parses the LSP message. Thanks for the quick fix!

Found another bug. Doc says that channel callback will return decoded json rpc message as vim dict. https://github.com/yegappan/vim/blob/aea71ca1f496c89dbfd127376368bb8c7c693782/runtime/doc/channel.txt#L1415-L1422

But from my experience seems like it may return a string if the server writes to stderr. I did a quickfix in vim-lsp to always check the type to be dict. prabirshrestha/vim-lsp@71efb2d. I think the better option would be to never call the callback for stderr and ask the caller to attach out_err instead.

Here is when a callback response is string.

Sun Apr 17 14:58:54 2022:["s:native_notification_callback","2022/04/17 14:58:54 efm-langserver: reading on stdin, writing on stdout"]

Here is when callback response is a dict

Sun Apr 17 14:59:25 2022:["s:native_notification_callback",{"id":0,"jsonrpc":"2.0","method":"window/workDoneProgress/create","params":{"token":"rustAnalyzer/Fetching"}}]
let l:jobopt = { 'in_mode': 'lsp', 'out_mode': 'lsp', 'noblock': 1, 'callback': function('s:native_notification_callback') }
let l:job = job_start(a:opts['cmd'], l:jobopt)

function! s:native_notification_callback(channel, response) abort
    call s:log('s:s:native_notification_callback', a:response)
    if type(a:response) == type({})
       " a:response is json decoded lsp message 
    else
        " a:respone is string from stderr
    endif
endfunction

@vim-ml
Copy link

vim-ml commented Apr 17, 2022 via email

@prabirshrestha
Copy link

I'm unblocked now that I can use out_cb and err_cb. Makes sense to clarify this in the doc.

@vim-ml
Copy link

vim-ml commented Apr 21, 2022 via email

@prabirshrestha
Copy link

I have been using native-lsp-client in vim-lsp for several days and haven't seen any issues with it. I should be all good. I need to rector code on my side and add support for tcp to be on par with the master branch before I can merge it.

My next would then be to profile and see the native client's perf improvements but this probably few weeks later. I usually use the typescript emiter.ts file for perf benchmarks on lsp as it is ~6k of real code in case you want to ahead and try it out.

@brammool
Copy link
Contributor

brammool commented Oct 11, 2022 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants