Skip to content

Implement shared memory WAL.#71

Merged
ncruces merged 30 commits into
mainfrom
wal
Apr 10, 2024
Merged

Implement shared memory WAL.#71
ncruces merged 30 commits into
mainfrom
wal

Conversation

@ncruces

@ncruces ncruces commented Apr 2, 2024

Copy link
Copy Markdown
Owner

This intends to implement shared memory WAL for both 64-bit Linux and macOS (amd64/arm64).

This depends on wazero/wazero#2157 (or some potentially better solution) to ensure Wasm memory is mmaped.

Current solution already passes mptest but still needs stability, portability and documentation work.

@ncruces

ncruces commented Apr 3, 2024

Copy link
Copy Markdown
Owner Author

Portability is done: only Linux and macOS (on amd64 and arm64) are supported.

illumos (and other 64-bit Linux archs) could potentially be supported, but they'd be hard to test; BSDs don't work due to locking; Windows is out due to memory remapping; 32-bit doesn't have enough address space.

@ncruces

ncruces commented Apr 4, 2024

Copy link
Copy Markdown
Owner Author

Mostly done. Still missing read-only mode, wazero/wazero#2157 improvements, and documentation.

@ncruces

ncruces commented Apr 4, 2024

Copy link
Copy Markdown
Owner Author

Will need an improved design over wazero/wazero#2157.

@ncruces ncruces force-pushed the wal branch 2 times, most recently from 2e924bb to faa9f6f Compare April 4, 2024 14:04
dependabot Bot and others added 17 commits April 4, 2024 23:50
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.21.0 to 0.22.0.
- [Commits](golang/crypto@v0.21.0...v0.22.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [golang.org/x/sync](https://github.com/golang/sync) from 0.6.0 to 0.7.0.
- [Commits](golang/sync@v0.6.0...v0.7.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sync
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@ncruces

ncruces commented Apr 6, 2024

Copy link
Copy Markdown
Owner Author

This is pretty much ready for testing, @infogulch.

This should also make these bindings fully compatible with Litestream, @Koeng101. Suggestions welcome for a Litestream test setup.

It also opens the way for SQLite memory mapped IO, and WAL-Mode Blocking Locks.

@Koeng101

Koeng101 commented Apr 6, 2024

Copy link
Copy Markdown

I'll test out later! Very exciting!

@ncruces

ncruces commented Apr 10, 2024

Copy link
Copy Markdown
Owner Author

Windows may be possible after all. For reference:
https://devblogs.microsoft.com/oldnewthing/20240201-00/?p=109346

Shared memory primitives should work fine on any Unix OS, as implemented. In practice, they're useless without OFD locks, so are unavailable on BSD. illumos could work though, but is disabled for being impossible to test.

@ncruces

ncruces commented Apr 10, 2024

Copy link
Copy Markdown
Owner Author

wazero/wazero#2177 was merged, and will be in the next release.

@ncruces ncruces marked this pull request as ready for review April 10, 2024 11:49
@ncruces ncruces merged commit 11c03a1 into main Apr 10, 2024
@ncruces ncruces deleted the wal branch April 10, 2024 12:20
@ncruces

ncruces commented Apr 11, 2024

Copy link
Copy Markdown
Owner Author

It also opens the way for SQLite memory mapped IO, and WAL-Mode Blocking Locks.

I briefly looked into memory mapped IO, and I won't be pursuing it at this time. It has numerous caveats, especially if not implemented exactly like SQLite does it (mapping the entire file, hoping the size doesn't change often). Trying to implement the API straightforwardly (map/unmap this page), leads to a bunch of syscalls, memory fragmentation, wasted space. Also not a fan of "crashes on IO errors."

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants