perf: `flattenedFields` sanitized collection/global property, remove deep copying in `validateQueryPaths` by r1tsuu · Pull Request #9299 · payloadcms/payload

r1tsuu · 2024-11-18T17:24:08Z

What?

Improves querying performance of the Local API, reduces copying overhead

How?

Adds flattenedFields property for sanitized collection/global config which contains fields in database schema structure.
For example, It removes rows / collapsible / unnamed tabs and merges them to the parent fields, ignores UI fields, named tabs are added as type: 'tab'.
This simplifies code in places like Drizzle transform/traverseFields. Also, now we can avoid calling flattenTopLevelFields which adds some overhead and use collection.flattenedFields / field.flattenedFields.

By refactoring configToJSONSchema.ts with flattenedFields it also

Fixes interfaceName does not work for named tabs. #9467
Fixes types for UI fields were generated for select

Removes this deep copying for each where query constraint

payload/packages/payload/src/database/queryValidation/validateQueryPaths.ts

Lines 69 to 73 in 58ac784

    
           validateSearchParam({ 
        
             collectionConfig: deepCopyObject(collectionConfig), 
        
             errors, 
        
             fields: fields as Field[], 
        
             globalConfig: deepCopyObject(globalConfig),

which potentially can add overhead if you have a large collection/global config

UPD:
The overhead is even much more than in the benchmark below if you have Lexical with blocks.

Benchmark in relationships/int.spec.ts:

const now = Date.now()
for (let i = 0; i < 10; i++) {
  const now = Date.now()
  for (let i = 0; i < 300; i++) {
    const query = await payload.find({
      collection: 'chained',
      where: {
        'relation.relation.name': {
          equals: 'third',
        },
        and: [
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
          {
            'relation.relation.name': {
              equals: 'third',
            },
          },
        ],
      },
    })
  }

  payload.logger.info(`#${i + 1} ${Date.now() - now}`)
}
payload.logger.info(`Total ${Date.now() - now}`)

Before:

[02:11:48] INFO: #1 3682
[02:11:50] INFO: #2 2199
[02:11:54] INFO: #3 3483
[02:11:56] INFO: #4 2516
[02:11:59] INFO: #5 2467
[02:12:01] INFO: #6 1987
[02:12:03] INFO: #7 1986
[02:12:05] INFO: #8 2375
[02:12:07] INFO: #9 2040
[02:12:09] INFO: #10 1920
    [PASS] Relationships > Querying > Nested Querying > should allow querying two levels deep (24667ms)
[02:12:09] INFO: Total 24657

After:

[02:12:36] INFO: #1 2113
[02:12:38] INFO: #2 1854
[02:12:40] INFO: #3 1700
[02:12:42] INFO: #4 1797
[02:12:44] INFO: #5 2121
[02:12:46] INFO: #6 1774
[02:12:47] INFO: #7 1670
[02:12:49] INFO: #8 1610
[02:12:50] INFO: #9 1596
[02:12:52] INFO: #10 1576
    [PASS] Relationships > Querying > Nested Querying > should allow querying two levels deep (17828ms)
[02:12:52] INFO: Total 17818

…ep copying for each constraint in `validateQueryPaths`

DanRibbens

This looks good! Kind of a pain to change all the names, but I think it would be good to make the name change.
What do you think?

DanRibbens · 2024-11-21T16:17:14Z

+  flattenFields: FlattenField[]
+} & MarkRequired<TabAsField, 'name'>
+
+export type FlattenField =


The name for this type is slightly off for me which hurts readability a bit. I would name it FlattenedField and change that on these other types too. FlattenedArrayField is the result of flattenFields() the function.

DanRibbens · 2024-11-21T16:19:53Z

+   * Fields in the database schema structure
+   * Rows / collapsible / tabs w/o name `fields` merged to top, UIs are excluded
+   */
+  flattenFields: FlattenField[]


I think this should be named flattenedFields to be more clear. flattenFields sounds like a function.

…s-improvements

…ixed

…s-improvements

DanRibbens · 2024-11-25T15:24:54Z

+      }
+
+      default: {
+        if (field.type !== 'ui') {


I would have made added case 'ui': break; for easier readability. This is fine since it is one if statement but if you want to make it better you could.

I also would prefer to make use of the utilities/traverseFields as much as possible. Again not a deal breaker though.

…ng in `validateQueryPaths` (#9299) ### What? Improves querying performance of the Local API, reduces copying overhead ### How? Adds `flattenedFields` property for sanitized collection/global config which contains fields in database schema structure. For example, It removes rows / collapsible / unnamed tabs and merges them to the parent `fields`, ignores UI fields, named tabs are added as `type: 'tab'`. This simplifies code in places like Drizzle `transform/traverseFields`. Also, now we can avoid calling `flattenTopLevelFields` which adds some overhead and use `collection.flattenedFields` / `field.flattenedFields`. By refactoring `configToJSONSchema.ts` with `flattenedFields` it also 1. Fixes #9467 2. Fixes types for UI fields were generated for `select` Removes this deep copying for each `where` query constraint https://github.com/payloadcms/payload/blob/58ac784425b411fc28c91dbb3e7df06cc26b6320/packages/payload/src/database/queryValidation/validateQueryPaths.ts#L69-L73 which potentially can add overhead if you have a large collection/global config UPD: The overhead is even much more than in the benchmark below if you have Lexical with blocks. Benchmark in `relationships/int.spec.ts`: ```ts const now = Date.now() for (let i = 0; i < 10; i++) { const now = Date.now() for (let i = 0; i < 300; i++) { const query = await payload.find({ collection: 'chained', where: { 'relation.relation.name': { equals: 'third', }, and: [ { 'relation.relation.name': { equals: 'third', }, }, { 'relation.relation.name': { equals: 'third', }, }, { 'relation.relation.name': { equals: 'third', }, }, { 'relation.relation.name': { equals: 'third', }, }, { 'relation.relation.name': { equals: 'third', }, }, { 'relation.relation.name': { equals: 'third', }, }, { 'relation.relation.name': { equals: 'third', }, }, { 'relation.relation.name': { equals: 'third', }, }, { 'relation.relation.name': { equals: 'third', }, }, ], }, }) } payload.logger.info(`#${i + 1} ${Date.now() - now}`) } payload.logger.info(`Total ${Date.now() - now}`) ``` Before: ``` [02:11:48] INFO: #1 3682 [02:11:50] INFO: #2 2199 [02:11:54] INFO: #3 3483 [02:11:56] INFO: #4 2516 [02:11:59] INFO: #5 2467 [02:12:01] INFO: #6 1987 [02:12:03] INFO: #7 1986 [02:12:05] INFO: #8 2375 [02:12:07] INFO: #9 2040 [02:12:09] INFO: #10 1920 [PASS] Relationships > Querying > Nested Querying > should allow querying two levels deep (24667ms) [02:12:09] INFO: Total 24657 ``` After: ``` [02:12:36] INFO: #1 2113 [02:12:38] INFO: #2 1854 [02:12:40] INFO: #3 1700 [02:12:42] INFO: #4 1797 [02:12:44] INFO: #5 2121 [02:12:46] INFO: #6 1774 [02:12:47] INFO: #7 1670 [02:12:49] INFO: #8 1610 [02:12:50] INFO: #9 1596 [02:12:52] INFO: #10 1576 [PASS] Relationships > Querying > Nested Querying > should allow querying two levels deep (17828ms) [02:12:52] INFO: Total 17818 ```

github-actions · 2024-11-25T21:49:42Z

🚀 This is included in version v3.1.1

…ng in `validateQueryPaths` (#9299) ### What? Improves querying performance of the Local API, reduces copying overhead ### How? Adds `flattenedFields` property for sanitized collection/global config which contains fields in database schema structure. For example, It removes rows / collapsible / unnamed tabs and merges them to the parent `fields`, ignores UI fields, named tabs are added as `type: 'tab'`. This simplifies code in places like Drizzle `transform/traverseFields`. Also, now we can avoid calling `flattenTopLevelFields` which adds some overhead and use `collection.flattenedFields` / `field.flattenedFields`. By refactoring `configToJSONSchema.ts` with `flattenedFields` it also 1. Fixes #9467 2. Fixes types for UI fields were generated for `select` Removes this deep copying for each `where` query constraint https://github.com/payloadcms/payload/blob/58ac784425b411fc28c91dbb3e7df06cc26b6320/packages/payload/src/database/queryValidation/validateQueryPaths.ts#L69-L73 which potentially can add overhead if you have a large collection/global config UPD: The overhead is even much more than in the benchmark below if you have Lexical with blocks. Benchmark in `relationships/int.spec.ts`: ```ts const now = Date.now() for (let i = 0; i < 10; i++) { const now = Date.now() for (let i = 0; i < 300; i++) { const query = await payload.find({ collection: 'chained', where: { 'relation.relation.name': { equals: 'third', }, and: [ { 'relation.relation.name': { equals: 'third', }, }, { 'relation.relation.name': { equals: 'third', }, }, { 'relation.relation.name': { equals: 'third', }, }, { 'relation.relation.name': { equals: 'third', }, }, { 'relation.relation.name': { equals: 'third', }, }, { 'relation.relation.name': { equals: 'third', }, }, { 'relation.relation.name': { equals: 'third', }, }, { 'relation.relation.name': { equals: 'third', }, }, { 'relation.relation.name': { equals: 'third', }, }, ], }, }) } payload.logger.info(`#${i + 1} ${Date.now() - now}`) } payload.logger.info(`Total ${Date.now() - now}`) ``` Before: ``` [02:11:48] INFO: #1 3682 [02:11:50] INFO: #2 2199 [02:11:54] INFO: #3 3483 [02:11:56] INFO: #4 2516 [02:11:59] INFO: #5 2467 [02:12:01] INFO: #6 1987 [02:12:03] INFO: #7 1986 [02:12:05] INFO: #8 2375 [02:12:07] INFO: #9 2040 [02:12:09] INFO: #10 1920 [PASS] Relationships > Querying > Nested Querying > should allow querying two levels deep (24667ms) [02:12:09] INFO: Total 24657 ``` After: ``` [02:12:36] INFO: #1 2113 [02:12:38] INFO: #2 1854 [02:12:40] INFO: #3 1700 [02:12:42] INFO: #4 1797 [02:12:44] INFO: #5 2121 [02:12:46] INFO: #6 1774 [02:12:47] INFO: #7 1670 [02:12:49] INFO: #8 1610 [02:12:50] INFO: #9 1596 [02:12:52] INFO: #10 1576 [PASS] Relationships > Querying > Nested Querying > should allow querying two levels deep (17828ms) [02:12:52] INFO: Total 17818 ```

perf: flattenFields sanitized collection/global property, remove de…

daed87f

…ep copying for each constraint in `validateQueryPaths`

github-actions Bot added created-by: Payload team v3 labels Nov 18, 2024

r1tsuu added 4 commits November 18, 2024 19:32

chore: fix types

c534cef

chore: upd buildjoinaggreation

b5de31f

chore: update predefined migration

56c6f71

chore: update getlocalizedosrtproperty unit test

6d3503f

r1tsuu force-pushed the perf/various-improvements branch from 2aea43e to 6d3503f Compare November 18, 2024 17:58

r1tsuu marked this pull request as ready for review November 18, 2024 18:25

r1tsuu added 2 commits November 18, 2024 20:27

chore: validatesearchparams remove deep copy from globalConfig as well

bdc5562

chore: remove unused import

8ad1bc7

r1tsuu requested a review from DanRibbens November 19, 2024 00:23

denolfe removed the v3 label Nov 19, 2024

DanRibbens requested changes Nov 21, 2024

View reviewed changes

r1tsuu added 3 commits November 21, 2024 19:28

rename FlattenedField and flattenedFields

1625a8e

rename flattenedFields, modify configToJSONSchema as well

70a2895

Merge branch 'main' of github.com:payloadcms/payload into perf/variou…

50772b8

…s-improvements

r1tsuu changed the title ~~perf: flattenFields sanitized collection/global property, remove deep copying in validateQueryPaths~~ perf: flattenedFields sanitized collection/global property, remove deep copying in validateQueryPaths Nov 21, 2024

r1tsuu added 4 commits November 21, 2024 20:02

fix: remaining flattenedFields

6bcd611

fix: remaining FlattenedField

a3c00b6

fix: types custom id in row/tab was string | null flattenedFields f…

fefa2f3

…ixed

confirm that this PR fixes interfaceName for tabs

a5bedf0

r1tsuu mentioned this pull request Nov 23, 2024

interfaceName does not work for named tabs. #9467

Closed

Merge branch 'main' of github.com:payloadcms/payload into perf/variou…

4de2f5b

…s-improvements

DanRibbens reviewed Nov 25, 2024

View reviewed changes

DanRibbens approved these changes Nov 25, 2024

View reviewed changes

DanRibbens merged commit cae300e into main Nov 25, 2024

DanRibbens deleted the perf/various-improvements branch November 25, 2024 15:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: `flattenedFields` sanitized collection/global property, remove deep copying in `validateQueryPaths`#9299

perf: `flattenedFields` sanitized collection/global property, remove deep copying in `validateQueryPaths`#9299
DanRibbens merged 15 commits into
mainfrom
perf/various-improvements

r1tsuu commented Nov 18, 2024 •

edited

Loading

Uh oh!

DanRibbens left a comment

Uh oh!

DanRibbens Nov 21, 2024

Uh oh!

DanRibbens Nov 21, 2024

Uh oh!

DanRibbens Nov 25, 2024

Uh oh!

github-actions Bot commented Nov 25, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	validateSearchParam({
	collectionConfig: deepCopyObject(collectionConfig),
	errors,
	fields: fields as Field[],
	globalConfig: deepCopyObject(globalConfig),

Conversation

r1tsuu commented Nov 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

How?

Uh oh!

DanRibbens left a comment

Choose a reason for hiding this comment

Uh oh!

DanRibbens Nov 21, 2024

Choose a reason for hiding this comment

Uh oh!

DanRibbens Nov 21, 2024

Choose a reason for hiding this comment

Uh oh!

DanRibbens Nov 25, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Nov 25, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

r1tsuu commented Nov 18, 2024 •

edited

Loading