Perf enhancement: load partial data for UI needs #10933

tiensonqin · 2024-01-29T13:58:14Z

This PR aims to reduce both the app start time and memory usage.

A demo with 30k pages:
https://www.loom.com/share/69a2b9d81be5416eb6f56bacb15daff6

TODO:

Instead of all the datoms

Use search indice to search first and then filter the result by the db worker. This avoids using `d/datoms` which loads all the blocks to the memory.

Also, add a script to build a test graph

Also avoid calls to worker if a block has been fully loaded

tiensonqin · 2024-02-06T20:19:40Z

Of the 3 graphs I measured via the :restore-graph-from-sqlite!-prepare, the only one that had a slower time was my db graph. Maybe it's because it has 5.8k properties? Here's the raw numbers:
- feat/db
	- work/docs: 2447 ms  Datoms in total:  76492
	- logseq-notes: 2730 ms  Datoms in total:  134855
	- durm: 632 ms  Datoms in total:  38572
- new branch
	- work/docs: 1153 ms
	- logseq-notes: 2280ms
	- durm: 1646 ms
I've done some QA and hope to do a little more tomorrow. A weird bug I'm seeing is the lazy loading of page names in page refs. They initially show up as all lower cased but when I navigate back, they show up as correctly cased. This is also occurring with page property values and with tag names that are listed to the right of a tagged block. This bug was noticed in publishing first but this also affects desktop

Your durm graph should be loaded faster with 140287e, the lower-cased bug should be fixed too.

tiensonqin · 2024-02-06T20:23:14Z

Deleting a db graph while in the graph consistently fails

I'm unable to reproduce this bug, is it on Web or Electron?

tiensonqin · 2024-02-06T20:30:05Z

On a blank journal page, hit tab and then enter on the first block. No edits are able to save after this. This also happens if the block has content. Basically testing if someone accidentally indents in an unexpected place

I can reproduce this one, looking into it now.

logseq-cldwalker · 2024-02-06T20:45:18Z

Your durm graph should be loaded faster with 140287e, the lower-cased bug should be fixed too

Confirmed the lower-case page bug is fixed. durm and other db graphs are loading faster. Thanks! Seeing a major improvement now at around 150ms

re-init commands after pages has been loaded

logseq-cldwalker · 2024-02-06T20:58:08Z

Deleting a db graph while in the graph consistently fails

I'm unable to reproduce this bug, is it on Web or Electron?

It was on electron. I'm no longer able to reproduce so that's good

tiensonqin · 2024-02-06T21:25:43Z

No edits are able to save

@logseq-cldwalker This should be fixed now. I have to clear the worker requests if they haven't updated the db.

logseq-cldwalker

Thanks for fixing the indent bug and the frontend tests. As discussed elsewhere I was able to see subsecond load times on a graph with 50k pages. Amazing work! 👍 🚀

logseq-cldwalker · 2024-02-07T17:16:00Z

@tiensonqin A bug I encountered on a browser db graph: Go to an existing journal page. Press g p to go to a previous day that does not exist. No journal page shows up

also remove unused binding not caught by linter b/c it's used

Removed in 0d6c65e

publishing apps can take awhile to load so we want the loading to be obvious, not muted

tiensonqin · 2024-02-09T01:29:40Z

Press g p to go to a previous day that does not exist. No journal page shows up

@logseq-cldwalker I'll fix this in another commit.

tiensonqin · 2024-02-09T01:49:35Z

@logseq-cldwalker The bug has been addressed by cba42f1.

logseq-cldwalker · 2024-02-09T14:00:10Z

Confirmed. Thanks!

This ns is more appropriate in db and was only in graph-parser because the date-time-util lib wasn't accessible to db until #10933 landed. graph-parser dep is specific to file graphs and as much as possible anything unrelated to this should not be in this dep

tiensonqin added 14 commits January 29, 2024 13:17

perf: load partial data when loading a graph

3addb59

Instead of all the datoms

Async get block and children

752f88d

Lazy load block container

eba0fe9

Lazy load page/block references and embeds

21c1f4f

fix: lazy load contents

0a5ca16

fix: lazy load whiteboards

b5b85dd

Remove block loading placeholder

059e8a8

wip: async custom query

26d4648

fix: can't open block on cmdk

3fd3073

fix: page/block refs and refs count

1e6daf6

fix: page not full loaded

694e9d0

fix: page unlinked refs

7b754c6

Use search indice to search first and then filter the result by the db worker. This avoids using `d/datoms` which loads all the blocks to the memory.

chore: code cleanup

b8c9206

chore: code cleanup

96dcdd9

tiensonqin changed the title ~~Perf enhancement: lazy load data for UI~~ Perf enhancement: load partial data for UI needs Jan 30, 2024

tiensonqin added 15 commits January 30, 2024 20:03

fix: more async queries and export to roam

99b6d54

fix: wrong kv? check

46f2106

fix: async <get-file and <get-files

420e361

partition load initial pages data

7f6cb9b

Also, add a script to build a test graph

Raise it to 30k pages

12263ed

wip: plugin api calls

44dca4f

Also avoid calls to worker if a block has been fully loaded

fix: plugin api calls

85f6b3d

fix: async query for scheduled and deadlines

ff77698

fix: lint warnings

9355520

fix: unit tests

a4431d8

fix: can't save file

1b99388

refactor: reindex graph equals delete and re-add the same graph

fd306bf

Merge branch 'feat/db' into perf/lazy-load-data

c1a15c0

fix: ref not shows when starting the app

c9f52e8

fix: don't create page if exists

d110dc4

tiensonqin and others added 2 commits February 7, 2024 04:07

Merge branch 'feat/db' into perf/lazy-load-data

7e8ab72

fix: unit test, lint + script

b1bcc1b

tiensonqin force-pushed the perf/lazy-load-data branch from 1718231 to b1bcc1b Compare February 6, 2024 20:09

tiensonqin added 2 commits February 7, 2024 04:47

Load classes for initial data since task commands require them

6c6cc8e

perf: skip loading classes for the initial data

bd9ebaf

re-init commands after pages has been loaded

fix: no edits are able to save after tab and then enter

7464984

tiensonqin added 2 commits February 7, 2024 05:36

fix: lint warnings

ad67e90

fix: editor unit tests

0a028ba

tiensonqin requested a review from logseq-cldwalker February 6, 2024 22:41

fix: load page tags

bee1cbf

logseq-cldwalker approved these changes Feb 7, 2024

View reviewed changes

logseq-cldwalker and others added 6 commits February 7, 2024 12:23

fix: test and lints

58b2888

also remove unused binding not caught by linter b/c it's used

chore: simplify create script with date helpers now available in dep

121bc21

fix: enter on db import submits

5487b90

fix: remove unused electron handler

f9063e7

Removed in 0d6c65e

enhance: make loading indicator noticeable for publishing

5945646

publishing apps can take awhile to load so we want the loading to be obvious, not muted

Merge branch 'feat/db' into perf/lazy-load-data

62b90b7

tiensonqin merged commit df2925f into feat/db Feb 9, 2024

tiensonqin deleted the perf/lazy-load-data branch February 9, 2024 01:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Perf enhancement: load partial data for UI needs #10933

Perf enhancement: load partial data for UI needs #10933

Uh oh!

tiensonqin commented Jan 29, 2024 •

edited

Loading

Uh oh!

tiensonqin commented Feb 6, 2024

Uh oh!

tiensonqin commented Feb 6, 2024

Uh oh!

tiensonqin commented Feb 6, 2024

Uh oh!

logseq-cldwalker commented Feb 6, 2024 •

edited

Loading

Uh oh!

logseq-cldwalker commented Feb 6, 2024

Uh oh!

tiensonqin commented Feb 6, 2024

Uh oh!

logseq-cldwalker left a comment

Uh oh!

logseq-cldwalker commented Feb 7, 2024

Uh oh!

tiensonqin commented Feb 9, 2024

Uh oh!

tiensonqin commented Feb 9, 2024

Uh oh!

logseq-cldwalker commented Feb 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Perf enhancement: load partial data for UI needs #10933

Perf enhancement: load partial data for UI needs #10933

Uh oh!

Conversation

tiensonqin commented Jan 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tiensonqin commented Feb 6, 2024

Uh oh!

tiensonqin commented Feb 6, 2024

Uh oh!

tiensonqin commented Feb 6, 2024

Uh oh!

logseq-cldwalker commented Feb 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

logseq-cldwalker commented Feb 6, 2024

Uh oh!

tiensonqin commented Feb 6, 2024

Uh oh!

logseq-cldwalker left a comment

Choose a reason for hiding this comment

Uh oh!

logseq-cldwalker commented Feb 7, 2024

Uh oh!

tiensonqin commented Feb 9, 2024

Uh oh!

tiensonqin commented Feb 9, 2024

Uh oh!

logseq-cldwalker commented Feb 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tiensonqin commented Jan 29, 2024 •

edited

Loading

logseq-cldwalker commented Feb 6, 2024 •

edited

Loading