Add optional WASM feature to the native library, allowing it to run wasm-compiled parsers via wasmtime by maxbrunsfeld · Pull Request #1864 · tree-sitter/tree-sitter

maxbrunsfeld · 2022-09-08T22:05:23Z

Background

Currently, Tree-sitter parsers can be compiled to WebAssembly (aka 'WASM') and run within a web browser, via the web-tree-sitter JavaScript library, which contains a WASM build of the Tree-sitter library.

In some applications that use the native Tree-sitter library, it would also be very useful to be able to load these same WASM builds of parsers, in order to support adding parsers as 🔌 plugins 🔌 , without requiring users to download platform-specific native binaries or to compile C code on their own machines.

Change

This PR adds a new optional WASM feature to the core library, which can be enabled in Rust via the wasm cargo feature, and in C via the TREE_SITTER_FEATURE_WASM macro.

This feature allows you to build a native TSLanguage object from a WASM buffer. You can then use this language object just like any other (native-compiled) language object: parsing text that lives in native memory, constructing syntax trees on the native heap, sending it between threads, etc. Tree-sitter languages are mostly just plain immutable data, so they're easy to unmarshal from a compiled wasm module.

The only difference is that when using a wasm-based language with a TSParser, you must first provide the parser with a TSWasmStore. This wasm store object wraps a wasmtime::Store object, which is used by the Tree-sitter library to invoke the language's lexing functions which are code, not data, so they require a WASM runtime to execute.

Notes

Wasmtime Dependency - I originally thought that it'd be cool to code against WebAssembly's standard wasm.h C interface, which is supposedly implemented by multiple runtimes (mainly wasmtime and V8). This way, applications using Tree-sitter would have multiple choices of which WASM runtime to link Tree-sitter against.

As I got into the details, I ended finding that some of the C APIs that I needed were currently unimplemented in wasmtime, but that Wasmtime provides its own custom C interface which is fully implemented, and which is supposedly better-designed from a performance perspective. So I ended up using Wasmtime's own C API. This means you can't use V8 as the wasm runtime when using this Tree-sitter feature, at least for now.

Tasks

build
- Open wasmtime PR for tweaks to the wasmtime-c-api Cargo.toml
library
- Support parsers without external scanners
- Support parsers whose external scanners don't use libc or libc++
- Support parsers that use libc and libc++ by bundling a small WASM module exporting all of the same stdlib symbols that as web-tree-sitter
- Generalize logic for copying parse tables out of wasm memory, removing hard-coded stuff
- Make sure wasm stores work correctly when loading multiple languages
- Make error handling robust
  - return an informative error from ts_wasm_store_load_language
cli
- Add a --wasm flag to the tree-sitter parse CLI command, which causes any parsers to be compiled to WASM instead of native shared libraries, and loaded via the new logic.
- Add a --wasm flag to the tree-sitter test command that works the same way
- Exercise this new behavior in the test suite. Maybe just run the entire test suite in both WASM and non-WASM modes?

…mtime

Replace non-mutating `ts_parser_wasm_store` function with `ts_parser_take_wasm_store`, which removes and returns the wasm store, in order to facilitate single ownership.

…anged

RubixDev · 2023-07-15T06:10:04Z

Open wasmtime PR for tweaks to the wasmtime-c-api Cargo.toml

I don't exactly understand why the Rust dependency on wasmtime-c-api is necessary. The only place it is used is the test() function in lib/binding_rust/wasm_language.rs which appears to be unused, and even that just seems to call wasmtime::Engine::default() internally. So is it really necessary to create a PR for allowing use of that crate from Rust?

RubixDev · 2023-07-25T03:57:37Z

Okay it seems I completely missed the point. Depending on the C API from Rust allows the tree-sitter C lib to be compiled and linked easily without having to manually vendor wasmtime. I created a PR on wasmtime here: bytecodealliance/wasmtime#6765.

maxbrunsfeld · 2023-11-28T20:10:58Z

Ok, this experimental feature is complete enough for people to try using in downstream applications. There will probably be bugs, but the feature basically works.

panekj · 2023-11-28T20:12:33Z

Thanks a lot for the work! We will be happy to test it in Lapce (hopefully soon)

clason · 2023-11-28T21:54:26Z

Huge step towards a truly platform-independent parser ecosystem. Out of interest (and I apologize if it's obvious), how does this relate to #949?

maxbrunsfeld · 2023-11-28T22:08:29Z

Out of interest (and I apologize if it's obvious), how does this relate to #949?

Yeah, that's a good question. That same limitation applies when using wasm-compiled parsers in this mode: there is a fixed (small) subset of the C and C++ standard libraries available that is compiled into the library, and any external scanners that rely on symbols that aren't in this subset will not work.

There may be some way to change how parsers are compiled to wasm, such that they can include code that they need from the standard library, but I don't know how to do that with Emscripten (or Clang) thus far. I feel like this design, though somewhat limiting, mostly works in practice.

But as part of stabilizing this feature, it would probably be good to add some tooling around detecting when external scanners use functions that are unavailable in a wasm context, and emitting warnings.

clason · 2023-11-28T22:12:11Z

Ah, pity. That is a dealbreaker for Neovim (as the Markdown parser is now required infrastructure, and many parsers in nvim-treesitter have a scanner.c).

But still, great feature!

maxbrunsfeld · 2023-11-28T22:12:54Z

Also, some big limitations to call out:

Right now, there's no way to delete a Language, because they previously consisted only of static data. Now, with wasm-based languages, they are allocated dynamically at runtime. I plan to add an API for deleting a Wasm-based language, so that you can dynamically reload a given language after changing its wasm (enabling a hot-reloading workflow for parsers). That'll happen in a future PR.
There's not yet any docs for how to use this feature. I'll add those in a subsequent PR.
I haven't used this feature in an actual editor yet, so I repeat, there are probably bugs.

clason · 2023-11-28T22:13:48Z

There's not yet any docs for how to use this feature. I'll add those in a subsequent PR.

Yeah, I noticed ;) Lots of stabbing in the dark for me...

maxbrunsfeld · 2023-11-28T22:14:13Z

Ah, pity. That is a dealbreaker for Neovim (as the Markdown parser is now required infrastructure, and many parsers in nvim-treesitter have a scanner.c).

Just to be clear, parsers with a scanner.c (or scanner.cc) will still work fine, as long as the scanner only uses certain functions from the C and C++ standard libraries. If there is some critical function that the Markdown scanner uses that I haven't thought to include, it's pretty easy to add it to the core. We need to add it here and here.

savetheclocktower · 2023-11-29T00:58:31Z

But as part of stabilizing this feature, it would probably be good to add some tooling around detecting when external scanners use functions that are unavailable in a wasm context, and emitting warnings.

This would be fantastic. We've tried to do so in Pulsar by adding a runtime check, but that requires a user to hit the specific code path first, and serves only to present a more understandable error message. If there were (or if there is) a way to analyze a wasm file and know which externals it wants, that'd be a major improvement.

clason · 2023-11-30T11:40:47Z

Another question, just to be sure: If this mode is enabled, will you still be able to use native parsers (i.e., support both .so and .wasm in an editor)?

maxbrunsfeld · 2023-11-30T16:12:34Z

Yes, you can still use the native parsers.

This PR adds undocumented functionality for loading custom language plugins at runtime. I don't intend to expose the functionality to end users yet, but this will allow the team to test the capability internally. ### Implementation There isn't much new code in Zed. Most of the work here is within Tree-sitter, in PRs tree-sitter/tree-sitter#1864 and tree-sitter/tree-sitter#2840, which allow Tree-sitter to load languages from WASM blobs. I've tested the functionality in Tree-sitter's test suite and via its CLI, but having it wired into Zed allows us to test the functionality more fully. ### Details Now, on startup, Zed will look for subdirectories inside of `~/Application Support/plugins`. These subdirectories are expected to look similar to the per-language subdirectories in [`crates/zed2/src/languages`](https://github.com/zed-industries/zed/tree/main/crates/zed2/src/languages), except that they also contain a `.wasm` file for the parser itself. I'll add more details here as I go.

joaomoreno · 2024-01-24T13:45:59Z

Any plan on shipping this?

icp1994 · 2024-01-26T07:11:37Z

Can (should?) the latest tree-sitter release be built against a published wasmtime version instead of git rev?

maxbrunsfeld · 2024-01-30T18:06:29Z

The challenge right now is that wasmtime-c-api is not available on crates.io. I'm trying to get that resolved (bytecodealliance/wasmtime#7837), but if there are blockers, I may need to work around it by temporarily removing the wasm feature from the version that we publish to crates.io, or vendoring the wasmtime-c-api sources somehow.

maxbrunsfeld mentioned this pull request Sep 8, 2022

Tree-sitter 1.0 Checklist #930

Open

31 tasks

maxbrunsfeld force-pushed the wasm-language branch 2 times, most recently from 951b5ce to 4bb82ce Compare October 24, 2022 22:36

maxbrunsfeld added 11 commits November 15, 2022 17:14

Start work on ability to load wasm languages from native lib, via was…

3f1a7f9

…mtime

Restucture wasm module, return a normal TSLanguage from load_language

042e6f9

Integrate WASM compilation into the CLI's Loader

d47713e

Use unchecked host functions for lexer callbacks

3869878

Use an unchecked function call for running the lex functions

ef1e3fb

Get wasm store working with parser with stateless external scanners

7dc8130

Provide minimal C/C++ std library to wasm external scanners

98ccfcf

Use my fork of wasmtime temporarily

7eb3e50

Get wasm.c compiling when wasm feature is disabled

9cbeaf2

Disable most wasmtime features

08ccf5b

Start work on a --wasm flag for the test subcommand

51720be

maxbrunsfeld force-pushed the wasm-language branch from 60b4f81 to 51720be Compare November 16, 2022 01:21

web: Remove conditional logic for older emscripten versions

35ce2e4

maxbrunsfeld mentioned this pull request Nov 17, 2022

Allow adding basic language language support for new languages via custom extensions zed-industries/zed#1899

Closed

maxbrunsfeld added 3 commits January 23, 2023 11:46

Allow testing highlight and tag queries when testing wasm build

555277a

Replace non-mutating `ts_parser_wasm_store` function with `ts_parser_take_wasm_store`, which removes and returns the wasm store, in order to facilitate single ownership.

Refactor storage of wasm stdlib function indices

6d34e5c

Fix crash when parser's wasm store is taken before its language is ch…

f7c9989

…anged

maxbrunsfeld force-pushed the master branch from 3707f48 to 125503f Compare February 14, 2023 07:40

clason mentioned this pull request Feb 18, 2023

treesitter distribution strategy (tree-sitter) neovim/neovim#22313

Open

ahlinc added enhancement c-library wasm labels Apr 4, 2023

pokey mentioned this pull request May 20, 2023

Merge vscode-parse-tree extension into Cursorless monorepo cursorless-dev/cursorless#1488

Open

RubixDev mentioned this pull request Jul 25, 2023

Make C-API usable from Rust bytecodealliance/wasmtime#6765

Merged

maxbrunsfeld force-pushed the wasm-language branch from 95369a4 to 6fd7a1e Compare November 26, 2023 20:15

Return an informative error on failing to construct a WasmStore

13dd76e

maxbrunsfeld force-pushed the wasm-language branch from 241de09 to 13dd76e Compare November 28, 2023 01:48

maxbrunsfeld marked this pull request as ready for review November 28, 2023 16:42

maxbrunsfeld merged commit 034f0d0 into master Nov 28, 2023

maxbrunsfeld deleted the wasm-language branch November 28, 2023 20:08

tree-sitter deleted a comment from PyaePhyo8612 Nov 28, 2023

clason mentioned this pull request Nov 29, 2023

Document requirement for static linking with Wasm tree-sitter-grammars/tree-sitter-markdown#126

Closed

This was referenced Dec 27, 2023

Introduce APIs for managing the lifetimes of languages, allow WASM languages to be deleted #2840

Merged

Lay some groundwork for language plugins zed-industries/zed#3837

Merged

maxbrunsfeld mentioned this pull request Jan 30, 2024

Tracking Issue: Language Extensions zed-industries/zed#7096

Closed

20 tasks

maxbrunsfeld mentioned this pull request Feb 2, 2024

Provide a consistent set of standard library function to wasm parsers across environments #2897

Merged

dundargoc added this to the 1.0 milestone Feb 6, 2024

deepak1556 mentioned this pull request Apr 5, 2024

web-tree-sitter fails in Node.js 19 and higher: Error: bad export type for tree_sitter_tsx_external_scanner_create: undefined #2338

Closed

ObserverOfTime mentioned this pull request Aug 19, 2025

Consider supporting wasm-c-api #4715

Closed

Uh oh!

Conversation

maxbrunsfeld commented Sep 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

Change

Notes

Tasks

Uh oh!

RubixDev commented Jul 15, 2023

Uh oh!

RubixDev commented Jul 25, 2023

Uh oh!

maxbrunsfeld commented Nov 28, 2023

Uh oh!

panekj commented Nov 28, 2023

Uh oh!

clason commented Nov 28, 2023

Uh oh!

maxbrunsfeld commented Nov 28, 2023

Uh oh!

clason commented Nov 28, 2023

Uh oh!

maxbrunsfeld commented Nov 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clason commented Nov 28, 2023

Uh oh!

maxbrunsfeld commented Nov 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

savetheclocktower commented Nov 29, 2023

Uh oh!

clason commented Nov 30, 2023

Uh oh!

maxbrunsfeld commented Nov 30, 2023

Uh oh!

joaomoreno commented Jan 24, 2024

Uh oh!

icp1994 commented Jan 26, 2024

Uh oh!

maxbrunsfeld commented Jan 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

maxbrunsfeld commented Sep 8, 2022 •

edited

Loading

maxbrunsfeld commented Nov 28, 2023 •

edited

Loading

maxbrunsfeld commented Nov 28, 2023 •

edited

Loading