Switching away from wazero #361
Pinned
ncruces
announced in
Announcements
Replies: 9 comments 26 replies
-
|
I'll definitely be taking a look at this new project of yours 👀 |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
semi related to this change (as i'm wanting to get a little more involved), do you have an IRC or matrix to discuss go-sqlite3 related changes and development @ncruces? |
Beta Was this translation helpful? Give feedback.
-
|
No, not really. If you're looking for a public discussion forum, maybe you can open a general discussion here? Otherwise, maybe email? |
Beta Was this translation helpful? Give feedback.
-
|
Very cool! It will be interesting to see how the performance changes. I find modern and you wasm version too each have performance advantages in different areas that I can necessarily explain. I switch between them for apps where I want to be independent of CGo and pick the version which handles the critical paths the best. I wish you all the best in this effort. I'm happy to assist as a tester if you need me. lbe Edit: Will this allow you to support mmap_size? |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
Not likely, no. With this change the major difference between this and Also each sandbox has a hard limit to how much memory it can allocate (4GB) and since it can't share, if you need 8 conns each with 4GB… you see where this is going. Yes, I can already So if you put these restrictions together, you see the way SQLite implements Also a big differentiator here is safety. I want to ensure things don't crash. Recoverable panics, maybe; no crashes. Memory mapping files also turns IO errors into hard crashes. |
Beta Was this translation helpful? Give feedback.
-
|
Got it. Thank you for the clearer explanation! |
Beta Was this translation helpful? Give feedback.
-
|
Compiling wasm2go sources seems quite heavy, on my machine it took ~40 seconds and 4GiB of RAM to compile a source generated from an sqlite3 WASM build. Once it's in the build cache it's quick, but it's quite a shift from the light weight wazero + embedded WASM binary. I could see it being an issue if someone is trying to compile on a low-end system or something like a single-board computer or low spec CI runner. |
Beta Was this translation helpful? Give feedback.
6 replies
-
|
How about asking the Go team to add it as a compiler benchmark? They did that for typescript's recently announced Go port, which is also a massive project that ends up doubling as a compiler stress test. |
Beta Was this translation helpful? Give feedback.
-
|
I don't think my project deserves that status 😄 |
Beta Was this translation helpful? Give feedback.
-
|
I wouldn't say that. If anything, I think they'd appreciate you giving them a heads up about this right now, instead of finding out about it on launch day through a flurry of bug reports, and having to scramble to fix things. |
Beta Was this translation helpful? Give feedback.
-
|
modernc/sqlite has lots more dependents, and suffers the same issue. I personally wouldn't optimize the compiler based on generated code. |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
This is ready to try at #362. I'll likely merge it to main, and create a branch for the wazero version. PS: this has been released. |
Beta Was this translation helpful? Give feedback.
6 replies
-
|
I decided to always build I still want to restrict usages of libc to just these tests, and disallow access to the actual file system except through the VFS. |
Beta Was this translation helpful? Give feedback.
-
|
Can you pin or at least record the wasm2go version in the go-sqlite3-wasm repo? |
Beta Was this translation helpful? Give feedback.
-
|
Sure. I merged the code in this repo to I am not ready to make a release of this just yet. The wazero version stays in a And before a release I will definitely:
The Go converted sources aren't binaries, but neither are they "readable." I created the other repository for one reason, that I'd like to see discussed. I want people to still be able to compile their own versions of SQLite. Making it a module, allows anyone who wants to, to use a One custom build I had was the one supporting |
Beta Was this translation helpful? Give feedback.
-
That's kind of a heavy-handed solution since Another option would be to have the module New function come from a global variable which can be overridden by other packages. This does have a slight downside of precluding compiler optimizations though, but I'd imagine the impact is relatively small. |
Beta Was this translation helpful? Give feedback.
-
|
That's true, but build tags (like Your suggestion would work if I know |
Beta Was this translation helpful? Give feedback.
-
|
Slightly off-topic, but I just switched one of my wazero-based bindings (github.com/pgaskin/go-marisa) to wasm2go, and it seems pretty good overall. Benchmarkspecs
wasm2go075b64a04783d6c188ac9f66ea52688bc71edb9e wazero37cde21a937175ff0928e5e6035b00a08be91daa benchstatIt passes all tests, and performs better in almost every way except for one SIMD-heavy part (build). |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
This is perfect and timely - i was hitting performance issues and instability with Wasm in Wasm . Many thanks for all of this |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
This may or may not be helpful there, as the binaries produced are quite large. But I'd be happy to know how it goes. |
Beta Was this translation helpful? Give feedback.
-
|
It looks like go-sqlite-bench was updated for go-sqlite3 v0.33.0 with wasm2go. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Is |
Beta Was this translation helpful? Give feedback.
6 replies
-
|
Yes, it's WAL mode on Linux. Previously, I decided to use modernc but I refactored my code since then (so it doesn't run additional queries between |
Beta Was this translation helpful? Give feedback.
-
|
Each connection still runs in a Wasm sandbox with |
Beta Was this translation helpful? Give feedback.
-
|
Oh, now I understand. The sandbox is still there, with memory management on top of Go slices and an |
Beta Was this translation helpful? Give feedback.
-
|
Did a quick and dirty wasm64 translation in https://github.com/dimkr/go-sqlite3-wasm/tree/64 and https://github.com/dimkr/go-sqlite3/tree/64 to see how hard it would be to eliminate 4 GB limits - tests seem to pass, and I even ran the test suite of a project of mine that currently uses 0.33.2 against this with |
Beta Was this translation helpful? Give feedback.
-
|
Yes, tests should pass with Memory64, it's just unlikely there's any advantage to allocating more than 4GB of memory to a single SQLite connection. |
Beta Was this translation helpful? Give feedback.
-
|
What's the stance going forward with extensions? Contribute to https://github.com/ncruces/go-sqlite3-wasm by compiling the extension and creating the shim in https://github.com/ncruces/go-sqlite3/tree/main/ext? sqlite-vec go bindings were overriding the Wasm binary https://github.com/asg017/sqlite-vec-go-bindings/blob/b64d0e563e615c99c19df431909a5970963d8fc0/ncruces/init.go#L21, bindings built on top of that need to be migrated |
Beta Was this translation helpful? Give feedback.
3 replies
-
|
If it's an extension you want to support, you can and should do all that in your own repo. I offered Alex to help with this for The intent is to help people build their own things with as standard tools as possible, not for me to become the maintainer of all existing and future SQLite extensions. If you look at how I'm compiling the two extensions in As you try to build extensions, it is perfectly natural (as I explained in the bug above) that you'll find missing sqlite or libc API. Ping me if you need any to be added. I will probably blindly accept anything missing from SQLite (unless it's something not built by default); libc I'll evaluate on a case by case basis. I will not add I/O stuff, but can add headers (you'll have to implement I/O yourself). |
Beta Was this translation helpful? Give feedback.
-
|
So, @luist18, I noticed your If you upgrade If this works for you, would you mind pinging Alex's repo? Might be helpful for others. |
Beta Was this translation helpful? Give feedback.
-
|
BTW, I've learned that |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
The current version, or more realistically the next version, of this SQLite bindings/driver will likely be the last ones to depend on wazero.
wazero has served me really well here, and is the reason this even exists. I'm also a wazero maintainer, and as time permits, I plan to continue that role.
However, I've spent the past few weeks developing wasm2go which I feel will serve this project better.
wasm2go does not obsolete wazero, which will always be relevant if you want to load Wasm modules at runtime. Or if you need pre-baked WASI/emscripten host modules (which I won't provide).
But for SQLite, it resolves long standing issues, such as the slow performance on interpreter targets, and slow startup on compiler targets. And seems, frankly, easier to maintain going forward than wazero (even considering the bus factor of 1).
So, as wasm2go stabilizes, I will very likely be moving to using it.
This will require some breaking changes, but mostly related to import paths and wazero configuration. I don't expect any changes to the SQLite API, at all. The transition should be really seamless.
Beta Was this translation helpful? Give feedback.
All reactions