Basic IndexedDB Functionality by rasviitanen · Pull Request #25214 · servo/servo

rasviitanen · 2019-12-09T14:40:10Z

This PR consists of basic functionality for IndexedDB.
The database engine used is Rkv, and might not be the most optimal engine to use, so I created a pluggable engine. This makes it easy to change for eventual benchmarks etc.
Recently, Rkv has raised warnings for using it in production, and the plan seems to be to change the backend to sqlite instead. If we expect that IndexedDB is finished after the issues in Rkv has been fixed, we should be able to use it after all. The ergonomics of Rkv is really good, and suits IndexedDB quite well.

What works:

Basic Open
Basic Put
Basic Add
Basic Get
Basic Remove

Some things that don't work:

Key generators
Key ranges
Aborting/Reverting transactions, including upgrades
Closing/Removing a store/db
Injection of keys
Proper scheduling of transactions (they are started immediately atm)
IDBIndex
IDBCursor
Manually committing transactions
Some Error handling is missing
... lots of other stuff

Implementation details worth noting:

Transactions run in a custom thread pool. read and write transactions use the same pool, which in the future probably should be separate.
IndexedDB is disabled by default, as it has a lot of troubles

I am making this PR pre-prematurely, as it is part of a course I take in school (and it's soon due date), so I would be very happy if reviews can be done rather quickly.

./mach build -d does not report any errors
./mach test-tidy does not report any errors
These changes fix (PARTLY) Implement IndexedDB. #6963

These changes do not require tests because:
There are many WebIDL tests that we can run to test the code. I was not able to run wpt tests with logging on windows, so test metadata is currently missing. I have, however, run some tests manually, and we can expect some of the tests to PASS.
Some examples of passing tests:
idbobjectstore_put.htm
idbobjectstore_put2.htm
idbobjectstore_put3.htm
idbobjectstore_put5.htm
idbobjectstore_put7.htm
idbobjectstore_put9.htm,
idbobjectstore_put10.htm
idbobjectstore_put11.htm
...
idbfactory_open.htm
idbfactory_open2.htm
idbfactory_open3.htm
idbfactory_open4.htm
...

highfive · 2019-12-09T14:40:18Z

Thanks for the pull request, and welcome! The Servo team is excited to review your changes, and you should hear from @asajeffrey (or someone else) soon.

highfive · 2019-12-09T14:40:19Z

Heads up! This PR modifies the following files:

@asajeffrey: components/script/dom/webidls/IDBTransaction.webidl, components/script/dom/idbrequest.rs, components/script/dom/mod.rs, components/script/dom/window.rs, components/script/dom/idbobjectstore.rs and 14 more
@KiChjang: components/script/dom/webidls/IDBTransaction.webidl, components/net/lib.rs, components/script/dom/idbrequest.rs, components/script/dom/mod.rs, components/net/indexeddb/mod.rs and 24 more

highfive · 2019-12-09T14:40:22Z

Warning

These commits modify unsafe code. Please review it carefully!
These commits modify net and script code, but no tests are modified. Please consider adding a test!

jdm · 2019-12-09T14:44:20Z

Please add a __dir__.ini to the indexeddb metadata directory to enable the new preference. This will make it possible for us to run the tests on CI and get the updated metadata you're looking for!

jdm · 2019-12-09T19:23:11Z

@bors-servo try=wpt

bors-servo · 2019-12-09T19:23:16Z

⌛ Trying commit f062693 with merge e720b98...

Basic IndexedDB Functionality  This PR consists of basic functionality for `IndexedDB`. The database engine used is `Rkv`, and might not be the most optimal engine to use, so I created a *pluggable* engine. This makes it easy to change for eventual benchmarks etc. Recently, `Rkv` has raised [warnings](https://github.com/mozilla/rkv) for using it in production, and the plan seems to be to change the backend to sqlite instead. If we expect that IndexedDB is finished after the issues in `Rkv` has been fixed, we should be able to use it after all. The ergonomics of Rkv is really good, and suits IndexedDB quite well. What works: * Basic Open * Basic Put * Basic Add * Basic Get * Basic Remove Some things that don't work: * Key generators * Key ranges * Aborting/Reverting transactions, including upgrades * Closing/Removing a store/db * Injection of keys * Proper scheduling of transactions (they are started immediately atm) * IDBIndex * IDBCursor * Manually committing transactions * Some Error handling is missing * ... lots of other stuff Implementation details worth noting: * Transactions run in a custom thread pool. `read` and `write` transactions use the same pool, which in the future probably should be separate. * IndexedDB is disabled by default, as it has a lot of troubles I am making this PR pre-prematurely, as it is part of a course I take in school (and it's soon due date), so I would be very happy if *reviews can be done rather quickly*. ---  - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] These changes fix (PARTLY) #6963  - [X] These changes do not require tests because: There are many WebIDL tests that we can run to test the code. I was not able to run wpt tests with logging on windows, so test metadata is currently missing. I have, however, run some tests manually, and we can expect some of the tests to `PASS`. Some examples of passing tests: `idbobjectstore_put.htm` `idbobjectstore_put2.htm` `idbobjectstore_put3.htm` `idbobjectstore_put5.htm` `idbobjectstore_put7.htm` `idbobjectstore_put9.htm`, `idbobjectstore_put10.htm` `idbobjectstore_put11.htm` ... `idbfactory_open.htm` `idbfactory_open2.htm` `idbfactory_open3.htm` `idbfactory_open4.htm` ...

bors-servo · 2019-12-09T20:01:40Z

💔 Test failed - status-taskcluster

jdm · 2019-12-09T21:48:42Z

@bors-servo try=wpt

bors-servo · 2019-12-09T21:48:47Z

⌛ Trying commit afbe66e with merge be4d861...

Basic IndexedDB Functionality  This PR consists of basic functionality for `IndexedDB`. The database engine used is `Rkv`, and might not be the most optimal engine to use, so I created a *pluggable* engine. This makes it easy to change for eventual benchmarks etc. Recently, `Rkv` has raised [warnings](https://github.com/mozilla/rkv) for using it in production, and the plan seems to be to change the backend to sqlite instead. If we expect that IndexedDB is finished after the issues in `Rkv` has been fixed, we should be able to use it after all. The ergonomics of Rkv is really good, and suits IndexedDB quite well. What works: * Basic Open * Basic Put * Basic Add * Basic Get * Basic Remove Some things that don't work: * Key generators * Key ranges * Aborting/Reverting transactions, including upgrades * Closing/Removing a store/db * Injection of keys * Proper scheduling of transactions (they are started immediately atm) * IDBIndex * IDBCursor * Manually committing transactions * Some Error handling is missing * ... lots of other stuff Implementation details worth noting: * Transactions run in a custom thread pool. `read` and `write` transactions use the same pool, which in the future probably should be separate. * IndexedDB is disabled by default, as it has a lot of troubles I am making this PR pre-prematurely, as it is part of a course I take in school (and it's soon due date), so I would be very happy if *reviews can be done rather quickly*. ---  - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] These changes fix (PARTLY) #6963  - [X] These changes do not require tests because: There are many WebIDL tests that we can run to test the code. I was not able to run wpt tests with logging on windows, so test metadata is currently missing. I have, however, run some tests manually, and we can expect some of the tests to `PASS`. Some examples of passing tests: `idbobjectstore_put.htm` `idbobjectstore_put2.htm` `idbobjectstore_put3.htm` `idbobjectstore_put5.htm` `idbobjectstore_put7.htm` `idbobjectstore_put9.htm`, `idbobjectstore_put10.htm` `idbobjectstore_put11.htm` ... `idbfactory_open.htm` `idbfactory_open2.htm` `idbfactory_open3.htm` `idbfactory_open4.htm` ...

bors-servo · 2019-12-09T22:18:37Z

💔 Test failed - status-taskcluster

jdm · 2019-12-09T23:59:47Z

Run wget https://community-tc.services.mozilla.com/api/queue/v1/task/P8Qk0aigQUKtiQRajoTYzg/runs/0/artifacts/public/test-wpt.log and then ./mach update-wpt test-wpt.log to update the metadata automatically. Don't forget to git add tests/wpt/metadata/IndexedDB to pick up any new ini files.

rasviitanen · 2019-12-10T07:30:38Z

Huh, those are not the test results I was expecting. Looking at the test harness and databases that are created for each test, it looks like we must be able to create multiple databases with different names.

dbname = (dbname ? dbname : "testdb-" + new Date().getTime() + Math.random() );

It looks like this doesn't work at the moment, and everything essentially operates on the same database. I assume this is why e.g. idbobjectstores_add.htm fails. This probably explains why I have been able to get a lot more passes when running one test at a time. I think I just forgot to add a call to Open and it shouldn't be too difficult to fix.

Hopefully, we'll get more sensical test results once it's fixed.

components/script/dom/idbrequest.rs

components/script/dom/idbfactory.rs

components/net/indexeddb/thread_pool.rs

components/net/indexeddb/idb_thread.rs

components/script/dom/idbfactory.rs

bors-servo · 2019-12-17T22:18:04Z

⌛ Testing commit fa8a94e with merge 9bda2c9...

Basic IndexedDB Functionality  This PR consists of basic functionality for `IndexedDB`. The database engine used is `Rkv`, and might not be the most optimal engine to use, so I created a *pluggable* engine. This makes it easy to change for eventual benchmarks etc. Recently, `Rkv` has raised [warnings](https://github.com/mozilla/rkv) for using it in production, and the plan seems to be to change the backend to sqlite instead. If we expect that IndexedDB is finished after the issues in `Rkv` has been fixed, we should be able to use it after all. The ergonomics of Rkv is really good, and suits IndexedDB quite well. What works: * Basic Open * Basic Put * Basic Add * Basic Get * Basic Remove Some things that don't work: * Key generators * Key ranges * Aborting/Reverting transactions, including upgrades * Closing/Removing a store/db * Injection of keys * Proper scheduling of transactions (they are started immediately atm) * IDBIndex * IDBCursor * Manually committing transactions * Some Error handling is missing * ... lots of other stuff Implementation details worth noting: * Transactions run in a custom thread pool. `read` and `write` transactions use the same pool, which in the future probably should be separate. * IndexedDB is disabled by default, as it has a lot of troubles I am making this PR pre-prematurely, as it is part of a course I take in school (and it's soon due date), so I would be very happy if *reviews can be done rather quickly*. ---  - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] These changes fix (PARTLY) #6963  - [X] These changes do not require tests because: There are many WebIDL tests that we can run to test the code. I was not able to run wpt tests with logging on windows, so test metadata is currently missing. I have, however, run some tests manually, and we can expect some of the tests to `PASS`. Some examples of passing tests: `idbobjectstore_put.htm` `idbobjectstore_put2.htm` `idbobjectstore_put3.htm` `idbobjectstore_put5.htm` `idbobjectstore_put7.htm` `idbobjectstore_put9.htm`, `idbobjectstore_put10.htm` `idbobjectstore_put11.htm` ... `idbfactory_open.htm` `idbfactory_open2.htm` `idbfactory_open3.htm` `idbfactory_open4.htm` ...

bors-servo · 2019-12-17T22:26:09Z

💔 Test failed - status-taskcluster

jdm · 2019-12-18T13:17:11Z

@bors-servo r+

bors-servo · 2019-12-18T13:17:12Z

📌 Commit f27c84f has been approved by jdm

bors-servo · 2019-12-18T14:54:11Z

⌛ Testing commit f27c84f with merge 9621c2a...

Basic IndexedDB Functionality  This PR consists of basic functionality for `IndexedDB`. The database engine used is `Rkv`, and might not be the most optimal engine to use, so I created a *pluggable* engine. This makes it easy to change for eventual benchmarks etc. Recently, `Rkv` has raised [warnings](https://github.com/mozilla/rkv) for using it in production, and the plan seems to be to change the backend to sqlite instead. If we expect that IndexedDB is finished after the issues in `Rkv` has been fixed, we should be able to use it after all. The ergonomics of Rkv is really good, and suits IndexedDB quite well. What works: * Basic Open * Basic Put * Basic Add * Basic Get * Basic Remove Some things that don't work: * Key generators * Key ranges * Aborting/Reverting transactions, including upgrades * Closing/Removing a store/db * Injection of keys * Proper scheduling of transactions (they are started immediately atm) * IDBIndex * IDBCursor * Manually committing transactions * Some Error handling is missing * ... lots of other stuff Implementation details worth noting: * Transactions run in a custom thread pool. `read` and `write` transactions use the same pool, which in the future probably should be separate. * IndexedDB is disabled by default, as it has a lot of troubles I am making this PR pre-prematurely, as it is part of a course I take in school (and it's soon due date), so I would be very happy if *reviews can be done rather quickly*. ---  - [X] `./mach build -d` does not report any errors - [X] `./mach test-tidy` does not report any errors - [X] These changes fix (PARTLY) #6963  - [X] These changes do not require tests because: There are many WebIDL tests that we can run to test the code. I was not able to run wpt tests with logging on windows, so test metadata is currently missing. I have, however, run some tests manually, and we can expect some of the tests to `PASS`. Some examples of passing tests: `idbobjectstore_put.htm` `idbobjectstore_put2.htm` `idbobjectstore_put3.htm` `idbobjectstore_put5.htm` `idbobjectstore_put7.htm` `idbobjectstore_put9.htm`, `idbobjectstore_put10.htm` `idbobjectstore_put11.htm` ... `idbfactory_open.htm` `idbfactory_open2.htm` `idbfactory_open3.htm` `idbfactory_open4.htm` ...

bors-servo · 2019-12-18T15:02:01Z

💔 Test failed - status-taskcluster

jdm · 2019-12-18T15:09:33Z

Darn - lmdb-rkv-sys contains native Windows code that isn't compatible with UWP builds. The shortest path to success here is to have a dummy IDB backend that's used when cfg(target_vendor = "uwp") is true that means that we don't compile the rkv crates.

bors-servo · 2020-01-06T15:37:04Z

☔ The latest upstream changes (presumably #23545) made this pull request unmergeable. Please resolve the merge conflicts.

bors-servo · 2020-11-11T20:47:52Z

🔒 Merge conflict

bors-servo · 2022-02-11T04:25:43Z

🔒 Merge conflict

jdm · 2024-10-19T01:29:40Z

Closing in favour of #33044.

Adds indexeddb support to servo. At the moment heed is being used as the backend, although this can be swapped out by implementing `KvsEngine`. This PR adds a thread + a thread pool for Indexeddb related operations. Also `database_access_task_source` is added for Indexeddb related operations. This is a partial rewrite of #25214. (Reopened due to branching issue) Fixes #6963 --------- Signed-off-by: Ashwin Naren <arihant2math@gmail.com> Signed-off-by: Josh Matthews <josh@joshmatthews.net> Co-authored-by: Rasmus Viitanen <rasviitanen@gmail.com> Co-authored-by: Josh Matthews <josh@joshmatthews.net>

highfive assigned asajeffrey Dec 9, 2019

highfive added the S-awaiting-review There is new code that needs to be reviewed. label Dec 9, 2019

rasviitanen force-pushed the indexed_db branch 3 times, most recently from 9ce7b52 to f062693 Compare December 9, 2019 18:04

highfive added the S-tests-failed The changes caused existing tests to fail. label Dec 9, 2019

rasviitanen force-pushed the indexed_db branch from f062693 to afbe66e Compare December 9, 2019 21:45

highfive removed the S-tests-failed The changes caused existing tests to fail. label Dec 9, 2019

highfive added the S-tests-failed The changes caused existing tests to fail. label Dec 9, 2019