I would really like to see consistency in both vim and neovim so would be good if both agrees on this.

Certainly if Vim adds this, Neovim could just dump the whole thing into its tree like we did for xdiff.

But we probably won't. Without NPM/v8, this lacks the chief advantages of Javascript. As an alternative to Vimscript, Lua is the strongest option: small, fast, elegant (3 core concepts used as pervasive building blocks: closures + tables + coroutines).

With this change, will Vim be stuck with duktape, or could another JS engine be swapped-in?

This proposal allows anyone to swap duktape with any JS engines. In order to have compatibility with other engines we need to support few functions as possible. In this case I proposed to have __vimcall, __vimeval, __vimcmd and require. require is currently not implemented yet but I do have plans to implement it. This will most likely follow the node resolution algorithm.

Given the JS is dynamic language one can do whatever you want. For example some one could write this which exposes nodejs compatible apis but using pure vim functions. Neovim could go and expose the full node runtime.

(function (oldRequire) {
    function newRequire(name) {
        if (name == 'fs') {
            return {
                readFileSync: function (path) {
                    return vim.call('readfile', [path]);
                }
            };
        } else {
            return oldRequire(name);
        }
    }
    require = newRequire; // overwrite global require with new require
})(require);

Then as a plugin author users can use the following. The best part about this is that the below code can also run in nodejs.

var fs = require('fs');
var contents = fs.readFileSync('hello.txt');

It is very common in JS world to monkey patch features that doesn't exist - Polyfill.

Another api I'm thinking is to only expose require function as minimum and expose a vim module as the minimum module which acts like std lib. Then if other hosts wants to add more modules they are free to use. Neovim could even expose own neovim module.

var vim = require('vim');
var contents = vim.call('readfile', ['hello.txt']);

Basically as a plugin author all I care is about having the stdlib in this case require('vim') and require() function and marshaling between vimscript world and javascript world to work consistently. This require('vim') as stdclib but for vim. Rest is up to them if they want to support more features such as WebWorkers, node compatible apis.

My biggest worry about adding full nodejs api is that it will increase the size and we are back to making ecmascript support optin which means most vim distributions out there won't enable it by default. I would really like this feature to also be available in tiny builds.

The same goes for using the npm packages and the entire nodejs ecosystem. If we make the spec for require function to follow the nodejs module resolutions as described here we don't need to ship the npm package manager. Users can just run npm install -g myvimplugin or yarn install -g myvimplugin. Let the users choose their favorite package manager. If Neovim does want to ship with package manager and have :EcmaScriptInstall myvimplugin that install to ~/.config/neovim/node_modules they can easily modify NODE_PATH enviornment variable as NODE_PATH=~/.config/neovim/node_modules;$NODE_PATH and based on the spec the require would look into this path.

If Neovim does have full support for node I would also be happy with it since then I can use certain features such as watch file.

var vim = require('vim');
var fs;
try {
   fs = require('fs');
} catch () {}

if (fs) {
    fs.watchFile('filetowatch', (cur, prev) => {
       // file changed.
    });
}

As an alternative to Vimscript, Lua is the strongest option: small, fast, elegant (3 core concepts used as pervasive building blocks: closures + tables + coroutines).

I would be curious to hear @brammool thoughts on always enabling Lua and requiring explicit optout to not include lua. Also might be converting the vim/runtime folder to slowly starting using lua/ecmascript code would force this.

JS itself is also very small and easy to undertand. It does have generators and async/await similar to lua though all js engines might not support it. It also does have closures (The first example of replacing require js shows an example of closure). The biggest advantage of JS instead of lua is the ability to tap into the entire nodejs and npm ecosystem by getting the require() module system right. The only issue I currently have with duktape is that it doesn't seem to support some of the biggest and latest JS spec. quickjs seems to be another one that supports most of the features and not sure how portable it is given that it is fairly new.

The goal of this PR is to start conversations like these so we can get the thoughts and vision from the vim and neovim devs as well as plugin authors.

The other option is to completely ignore ECMAScript and Lua as default scripting and go full on with WebAssembly. The only problem with this is that we currently don't have any good runtimes written in pure c now. Biggest advantage of WASM over Ecmascript and JS it that it allows to get the entire ecosystem of almost all languages out there and can also be high performant. Here is the list of wasm runtimes. https://github.com/appcypher/awesome-wasm-runtimes. wac seems to be written in c but I couldn't get to compile it on mac and windows yet but should be simple. It is MPL license. If anyone is interested to work on creating a minimal wasm runtime in c let me know and I would be happy to work on this with others. And of course similar to lua and luajit support we could have a basic unoptimized version of wasm c runtime and interpreter in vim and if people do want a fast one they could use other complicated wasm runtime and interpreters that support JIT.

As for Javascript vs WebAssembly I create a twitter poll that will last for 1 day. Will be curious to see the results.

https://twitter.com/PrabirShrestha/status/1193363217763799045

I would really like this feature to also be available in tiny builds.

But not even Vim's own FEAT_EVAL is in TINY builds. That's why we've been stuck with "Ex-command APIs" like :syntax and errorformat (instead of errorfunc) for decades.

This would anyways defeat the purpose of the TINY build, which is a vanity exercise for people who insist Vim HEAD is needed on ultra-constrained-resource systems.

converting the vim/runtime folder to slowly starting using lua/ecmascript code would force this.

Good idea.

JS itself is also very small and easy to undertand
...
The only issue I currently have with duktape is that it doesn't seem to support some of the biggest and latest JS spec

Doesn't that conflict with "small and easy to understand"? JS keeps adding language features, workarounds for an incomplete design.

If Neovim does have full support for node I would also be happy with it since then I can use certain features such as watch file.

Can do this already in Neovim, with Lua vim.loop. See :help file-change-detect in Nvim 0.5 (HEAD).

it is also very difficult to express some of the constructs in vimscript.

What are these?

it is also very difficult to express some of the constructs in vimscript.
What are these?

Multi line lambdas in vim script with closures. Very important to express for async code. Both Lua and JS shines on this one compared to vimscript.

For vim-lsp I use a helper until function to express async workflow. https://github.com/prabirshrestha/vim-lsp/blob/master/autoload/lsp/utils/step.vim

Wasn’t aware of FEAT_EVAL.

Doesn't that conflict with "small and easy to understand"? JS keeps adding language features, workarounds for an incomplete design.

This is one of the reason I’m leaning to WASM. Create a simple good wasm vim in c and let the respective language author do the design.

I don't understand why you would wasm at all. You would need a compiler, so you might as well just compile to an .so file and load that - free performance!

As for JavaScript - we've already got Python. You say that your goal here is to provide a less bloated feature, so that more distributions ship it by default. In my opinion, the reasonable thing to do here is to reduce the overhead of Python, instead of shipping yet another language, which will increase the size of a full build, and hence increase the pressure to leave some features disabled.

Moreover, the design of JavaScript is absolutely terrible. An undefined member variable should clearly be an unrecoverable error, but JavaScript insists on doing something. This is just one example, but this philosophy is applied everywhere.

I don't understand why you would wasm at all. You would need a compiler, so you might as well just compile to an .so file and load that - free performance!

Wasm binary is portable so users don't need to compile sources in their environment. If wasm is integrated to Vim, plugin authors can build a plugin with a language which can be compiled into Wasm and can compile it to .wasm in their local. Then they can distribute the .wasm files to users.

API to use system interfaces are being defined as WebAssembly System Interface portably. Plugin authors (hopefully) don't need to take care about differences of platforms.

Another advantage is security. WebAssembly is run in sandboxed separated environment.

BTW I'm not sure what you mean with '.so file', but interfaces between Vim script and .so (or .dll on Windows) files are only libcall() and libcallnr(). They are very poor and difficult to use.

Wasm binary is portable so users don't need to compile sources in their environment. If wasm is integrated to Vim, plugin authors can build a plugin with a language which can be compiled into Wasm and can compile it to .wasm in their local. Then they can distribute the .wasm files to users.

I feel like this is an easy way to get proprietary vim plugins. I'm not sure if that's desirable.

Another advantage is security.

This would require a creating a permission model. Malicious plugins won't become harmless just because they're being executed by a wasm runtime.

BTW I'm not sure what you mean with '.so file', but interfaces between Vim script and .so (or .dll on Windows) files are only libcall() and libcallnr(). They are very poor and difficult to use.

Improving this would certainly be easier and introduce less bloat.

A few comments:

• The tiny build doesn't even have Vim script, it will not have any other scripting support either.
• Javascript is going to fade away, Typescript is replacing it. If we aim at Javascript now, soon there will ALSO be a Typescript version, and we end up with two more languages.
• The language is only one part, the interface to Vim is the critical part. Other interfaces, also for a popular language like Python, lack popularity mainly because the interface to Vim is the limiting factor. Not because users don't know or like that language.
• If we make Vim script faster and perhaps add a few more things, wouldn't it be the better solution?

So I have been playing more with duktype and it not supporting lambdas is very disappointing.

Thought typescript is gaining popularity the target language of JS will always be more popular. Browsers have no intention to make TS a first class language. They only have plans to support webassembly.

For me the main problem with other languages is that it doesn't come out of the box. So writing a plugin in vimscript is the only way to get 100% coverage. Some OS comes with python but older version and trying to manage is another mess. Though once done it is very easy.

If there are ways to make vimscript faster it would be good but I'm not sure it is worth it. I would rather have the investments in making lua the default language or webassembly vm that is always part of vim. Then rather then executing vimscript you compile vimscript to lua or webassembly and let their engine execute. All neovim builds already ship with lua. It would be good if vim does the same. It will almost be impossible to bit the speed of LuaJIIT. Converting all the vimscript in vim to lua would be one way to force people to enable lua in vim for their distributions.

At some universities and work we can't install random binaries so having lua/webassembly working with vim by default would be good.

The language is only one part, the interface to Vim is the critical part.

Agreed, and that hints at a question I've silently had: if a JS dev isn't willing to learn Lua, how in the world will they have the patience to write a plugin for any editor--Vim or otherwise--given that learning the API is the most annoying part.

So "JS instead of Lua" isn't compelling in terms of mere syntax.

Other interfaces, also for a popular language like Python, lack popularity mainly because the interface to Vim is the limiting factor.

No. Interface is very important, but "mainly" is an overstatement: at least half of the reason is that Python is optional in Vim. Defaults matter (as Vim admits implicitly, via the defaults.vim half-solution). Not to mention, Vim Python support requires system libs to be upgraded in lockstep.

If we make Vim script faster and perhaps add a few more things, wouldn't it be the better solution?

That's a strong solution if Vimscript is backwards-compatible. But it's not, and I expect it will introduce more breaking changes to achieve such speedups.

If "Vimscript version 17" makes backwards-incompatible changes then it's a different language than "Vimscript version 1". And this means there was no reason to keep it instead of using Lua: entire purpose of supporting Vimscript is to support existing plugins. If those plugins don't get the performance improvements because they use an older :scriptversion, then nothing was gained by improving the speed of "Vimscript version 17", except to make a new language that will never be as good as Lua or other existing languages.

Python is the most popular scripting language. Unfortunately, embedding it is problematic. And even then a runtime environment is needed, which is too big to ship together with Vim, thus installing Python (in a version that matches what Vim was build with) is then a requirement, which apparently is too much of a burden for most users.

The same is true for all script languages. Lua might be smaller, but I don't think it is feasible to include it with the distribution. And Lua is not popular at all, it dangles near the end of popular language rankings, if it appears at all.

Javascript is popular, but that's for use in a browser. It doesn't even have file access in the language itself, requires a runtime like NodeJS, which is big (or perhaps huge, depending on what you expect).

Perl appears to have dropped out of the "popular languages" rankings. The same might happen to any less known or less used language.

Lua might be smaller, but I don't think it is feasible to include it with the distribution.

Lua 5.1 is 10k LOC.

xdiff was pasted-into Vim's source tree without hesitation, adding 5k LOC:

e828b76#diff-60e734234eae4792117ac2f0f869eb98

libvterm was also pasted-into Vim's tree.

Perl appears to have dropped out of the "popular languages" rankings. The same might happen to any less known or less used language.

How is that relevant? Vimscript is even less popular than perl, Lua, Visual Basic, Haskell, and many others. What matters is that Lua is a mature, well-designed language with multiple high-quality implementations.

"Vimscript version 1" will never compete in terms of performance nor popularity. "Vimscript version 87" may someday match the performance of Ruby 1.x, yet it will be even less popular than "Vimscript version 1".

Lua 5.1 is 10k LOC.

xdiff was pasted-into Vim's source tree without hesitation, adding 5k LOC:

e828b76#diff-60e734234eae4792117ac2f0f869eb98

Besides the fact that I personally don't like lua, I don't see how this is relevant. So lua is still twice as big as xdiff (which made a smaller by adding many ifdefs) and it does not fullfill a need. We already have python, perl and tcl that basically serve the same purpose. So that argument doesn't hold at all.

If you want to convince me personally, then please come with some arguments why yet another language should be imported and supported. I don't see this here.

Vim is used more often than Lua: https://github.com/oprogramador/github-languages#all
Just a matter of finding the right ranking :-).

My 2cents,

Vim, and more so Neovim, already has integration with many 3rd party languages, most commonly Python. For example, Coc is mostly implemented in TypeScript.

The core issue is not that 3rd party languages are excluded, it's that 3rd party integrations are fragile and can sometimes be a pain in the behind when updates occur (system, Homebrew, etc).

It is way too easy to break Python plugins as noted here.

I avoid Python plugins like the plague for this reason.

Fundamentally it should be about what runtimes are available and their capabilities. As much as JavaScript and TypeScript are the new hotness, that can fade, I can not see their runtimes ever being built into Vim. Too big and too much of a moving target. Note that does not preclude JavaScript-based plugins from existing, they already do as noted by Coc.

Vim provides Vimscript which has Frankensteined its way into be decent.
Neovim provides Vimscript and now Lua.

@justinmk in his recent VimConf provided this VimScript vs Lua micro-performance result:

foo.vim:
 let s:sum = 0
 for i in range(1, 9999999)
 let s:sum = s:sum + i
 endfor
 call append('$', s:sum)
Time: 31.611 seconds

foo.lua:
 sum = 0
 for i = 1, 9999999 do
 sum = sum + i
 end
 vim.api.nvim_call_function('append',
 {'$', tostring(sum)})
Time: 0.015 seconds
speedup: 31.611 / 0.015 = 2107

That's a 2000 times speedup.
One of the reasons we have as many Python plugins as we do is due to performance as noted by YouCompleteMe here

Why isn't YCM just written in plain VimScript, FFS?
Because of the identifier completion engine and subsequence-based filtering. Let's say you have many dozens of files open in a single Vim instance (I often do); the identifier-based engine then needs to store thousands (if not tens of thousands) of identifiers in its internal data-structures. When the user types, YCM needs to perform subsequence-based filtering on all of those identifiers (every single one!) in less than 10 milliseconds.

I'm sorry, but that level of performance is just plain impossible to achieve with VimScript. I've tried, and the language is just too slow. No, you can't get acceptable performance even if you limit yourself to just the identifiers in the current file and simple prefix-based filtering.

Plugins implemented in Vimscript will hit problems when dealing with large amounts of items, sorting, filtering, etc. Many, maybe even most, plugins will be fine. However, some do require something faster than Vimscript.

If the maintainers of Vim can speed Vimscript up, that would be awesome, it would benefit every Vim user. I am not sure how easy that will be, runtimes and JITs sounds hard.

The Neovim crew have incorporated Lua and LuaJIT. Of all the choices available this seems ideal. Very small size, yet comes with a high performance JIT. Lua is a niche language, for sure, but so is Vimscript. Most Vim users aren't writing plugins, hence learning Lua will be limited to the subset of Neovim users who code plugins. In that context it's an ideal choice. And I am very confident that Lua has a bright future since one of the biggest Internet infrastructure companies, Cloudflare, makes use of Lua and LuaJIT.

It will be very interesting to see what happens with respect to pure Lua Neovim plugins. Will we see Deoplete-like (completion engine) or CtrlP-like (fuzzy finders) plugins implemented in pure Lua?

Maybe we will have a best of both worlds:

• Vimscript becomes slightly faster

• We see lots of awesome Lua-based Neovim plugins hence resulting in Vim also incorporating their own Lua-runtime into core Vim.

Enough ramblings.

QuickJS is tailor-made for this use case:

QuickJS is a small and embeddable JavaScript engine. It supports the ES2020 specification including modules, asynchronous generators and proxies.

Javascript is going to fade away, Typescript is replacing it.

As much as JavaScript and TypeScript are the new hotness

😕

There is no such language as TypeScript as far as embedding/eval is concerned, any more than there are CoffeeScript, MoonScript, or Riml runtimes/interpreters. Deno (and TypeScript (tsc) itself) transpiles the syntax to JavaScript.

Talking about the scripting language, I think there are two choices in front of us:

1. Adopting a mature language.

IMO, lua is the best option which is very fast, very slim and embedable. I am willing to learn it if it can improve the performance, and of course it will cause many people have proved it. And neovim also adopt lua already, if vim adopts too, we'll benefit from neovim's plugin ecosystem.

2. Grow vim-script into a mature one.

@brammool and many other developers are doing for this. (Thanks for your working!) We can see patches are uploaded almost everyday. But when can vim-script be ready as a mature language? I don't know. Besides, as vim-script growing, scriptversion ## number grows as well, managing this is painful for not only language developers but also plugin developers!

As an average developer. (Yes we use Vim, and yes we are part of the equation too)

If I get to choose between a faster VimScript Implementation vs Lua. I'll choose Lua.

Lua is worth learning, is a good investment of my time, it can be used elsewhere to build things other than plugins for a text editor.

This is an interesting discussion, but not a feasible pull request. Therefore closing.

@prabirshrestha @brammool there are too many awesome programming language, can we add all of them? No, of cause, so the best way is add remote API support, just like what neovim did.

The all plugins developers can choose their own language!