New script to break markdown in to Algolia search records and publish. by NWylynko · Pull Request #2961 · clerk/clerk-docs

NWylynko · 2026-01-13T18:42:46Z

🔎 Previews:

https://clerk-git-nick-algolia-search.clerkstage.dev/docs

What does this solve?

The current setup with Algolia has a crawler run after changes are deployed live, it has various issues
- The editor on the dashboard seamingly recently fails to validate / run the crawler
- It fails to crawl docs, but can't be debugged from the issue above
- The current setup has us using git to attempt to guess the recently changed files to re-crawl

What changed?

Converts the crawler from examining html to create records to being markdown to records
Moves the crawler logic to a script in source control in our repo instead of managed in Algolia dashboard
Makes updating the search records a step of the docs merges / deployments

vercel · 2026-01-13T18:42:51Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Review	Updated (UTC)
clerk-docs	Ready	Preview	Jan 16, 2026 7:22pm

…in the index

manovotny · 2026-01-14T21:49:52Z

scripts/update-algolia-records.ts

+}
+
+function getGitBranch(): string {
+  return 'main'


I don't think we want this hardcoded return. Won't this delete main records from a preview branch?

Yes this is just temp so the preview deployment works, it will be removed before merge

Allows running the script without pushing changes to Algolia. In dry run mode, outputs records to .algolia/records.json and stale record IDs to .algolia/stale.json for inspection. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

NWylynko · 2026-01-15T14:23:32Z

There's 7961 records with "content": null. Should we skip adding these to the index?

Example record:

{
    "objectID": "main-0-/docs/maintenance-mode#main",
    "branch": "main",
    "anchor": "main",
    "content": null,
    "type": "lvl1",
    "keywords": [],
    "availableSDKs": [
      "all"
    ],
    "canonical": "/docs/maintenance-mode",
    "weight": {
      "pageRank": 0,
      "level": 90,
      "position": 0
    },
    "hierarchy": {
      "lvl0": "Documentation",
      "lvl1": "Maintenance Mode",
      "lvl2": null,
      "lvl3": null,
      "lvl4": null,
      "lvl5": null,
      "lvl6": null
    },
    "distinct_group": "/docs/maintenance-mode#main",
    "record_batch": "729f829e-2658-48ab-b45e-e1e6274eff31"
  },

"content": null is an expected record, every heading in the docs, including the title will have a content of null, and Algolia uses the hierarchy object as a searchable content.

NWylynko · 2026-01-15T14:55:12Z

Tooltips are getting duplicated/smooshed together.

Example record:

{
    "objectID": "main-6-/docs/guides/customizing-clerk/email-sms-templates#revolvapp-wysiwyg-email-editor-plugin",
    "branch": "main",
    "anchor": "revolvapp-wysiwyg-email-editor-plugin",
    "content": "The email editor uses the Revolvapp WYSIWYGWYSIWYG stands for 'What You See Is What You Get'. This term is used for editors and design tools that allow you to create content or layouts in a visual manner (without requiring you to edit the underlying markup) so that you can instantly see how the result will display to your users. email template editor plugin by Imperavi. To acquaint yourself with the template markup syntax, consult Imperavi's docs.",
    "type": "content",
    "keywords": [],
    "availableSDKs": [
      "all"
    ],
    "canonical": "/docs/guides/customizing-clerk/email-sms-templates",
    "weight": {
      "pageRank": 0,
      "level": 0,
      "position": 6
    },
    "hierarchy": {
      "lvl0": "Documentation",
      "lvl1": "Email and SMS templates",
      "lvl2": "Before you start",
      "lvl3": "Revolvapp WYSIWYG email editor plugin",
      "lvl4": null,
      "lvl5": null,
      "lvl6": null
    },
    "distinct_group": "/docs/guides/customizing-clerk/email-sms-templates#revolvapp-wysiwyg-email-editor-plugin",
    "record_batch": "729f829e-2658-48ab-b45e-e1e6274eff31"
  },

I decided that since tooltip content can't be seen directly on the screen to just remove it from the record. My other thought was to create a new separate record with just the content of the tooltip, but I think that will confusing people.

NWylynko · 2026-01-15T15:08:59Z

Did we want to index properties? I can't remember what we decided in our team meeting.

Example record:

{
    "objectID": "main-4-/docs/fastify/guides/customizing-clerk/appearance-prop/variables#properties",
    "branch": "main",
    "anchor": "properties",
    "content": "string",
    "type": "content",
    "keywords": [],
    "availableSDKs": [
      "astro",
      "chrome-extension",
      "expo",
      "nextjs",
      "nuxt",
      "react",
      "react-router",
      "remix",
      "tanstack-react-start",
      "vue",
      "js-frontend",
      "fastify",
      "expressjs",
      "js-backend",
      "go",
      "ruby"
    ],
    "canonical": "/docs/:sdk:/guides/customizing-clerk/appearance-prop/variables",
    "weight": {
      "pageRank": 0,
      "level": 0,
      "position": 4
    },
    "hierarchy": {
      "lvl0": "Documentation",
      "lvl1": "Variables prop",
      "lvl2": "Properties",
      "lvl3": null,
      "lvl4": null,
      "lvl5": null,
      "lvl6": null
    },
    "distinct_group": "/docs/:sdk:/guides/customizing-clerk/appearance-prop/variables#properties",
    "record_batch": "729f829e-2658-48ab-b45e-e1e6274eff31"
  },

Example record pulling in complex types:

{
    "objectID": "main-127-/docs/guides/customizing-clerk/elements/reference/common#properties-8",
    "branch": "main",
    "anchor": "properties-8",
    "content": "({ value, status }: { value: string; status: 'none' | 'selected' | 'cursor' | 'hovered' }) => React.ReactNode",
    "type": "content",
    "keywords": [],
    "availableSDKs": [
      "all"
    ],
    "canonical": "/docs/guides/customizing-clerk/elements/reference/common",
    "weight": {
      "pageRank": 0,
      "level": 0,
      "position": 127
    },
    "hierarchy": {
      "lvl0": "Documentation",
      "lvl1": "Common components",
      "lvl2": "<Input>",
      "lvl3": "<Input type=\"otp\">",
      "lvl4": "Properties",
      "lvl5": null,
      "lvl6": null
    },
    "distinct_group": "/docs/guides/customizing-clerk/elements/reference/common#properties-8",
    "record_batch": "729f829e-2658-48ab-b45e-e1e6274eff31"
  },

I've combined the records that each property was being split in to, so now each property in a properties table gets just one record. Same as the production records.

I'm wondering if we want to weigh these records down so they need to be searched for specifically.

example:

{
    "objectID": "main-4-/docs/nuxt/guides/customizing-clerk/appearance-prop/variables#properties",
    "branch": "main",
    "anchor": "properties",
    "content": "colorDanger (string) - The color used for error states. CSS variable: --clerk-color-danger",
    "type": "content",
    "keywords": [],
    "availableSDKs": [
      "astro",
      "chrome-extension",
      "expo",
      "nextjs",
      "nuxt",
      "react",
      "react-router",
      "remix",
      "tanstack-react-start",
      "vue",
      "js-frontend",
      "fastify",
      "expressjs",
      "js-backend",
      "go",
      "ruby"
    ],
    "canonical": "/docs/:sdk:/guides/customizing-clerk/appearance-prop/variables",
    "weight": {
      "pageRank": 0,
      "level": 0,
      "position": 4
    },
    "hierarchy": {
      "lvl0": "Documentation",
      "lvl1": "Variables prop",
      "lvl2": "Properties",
      "lvl3": null,
      "lvl4": null,
      "lvl5": null,
      "lvl6": null
    },
    "distinct_group": "/docs/:sdk:/guides/customizing-clerk/appearance-prop/variables#properties",
    "record_batch": "625815f7-9153-4b43-a966-c06e2ebb0b8e"
  },

manovotny · 2026-01-15T16:14:10Z

"content": null is an expected record, every heading in the docs, including the title will have a content of null, and Algolia uses the hierarchy object as a searchable content.

Ah, gotcha. This could be a new schema improvement, but thanks for helping me understand how it currently works.

manovotny · 2026-01-15T16:16:10Z

I decided that since tooltip content can't be seen directly on the screen to just remove it from the record. My other thought was to create a new separate record with just the content of the tooltip, but I think that will confusing people.

Just to make sure we're on the same page, I would expect WYSIWYGWYSIWYG becomes WYSIWYG.

manovotny · 2026-01-15T16:17:28Z

I've combined the records that each property was being split in to, so now each property in a properties table gets just one record. Same as the production records.

Much better!

I'm wondering if we want to weigh these records down so they need to be searched for specifically.

I would say yes. They could be helpful, but they're also going to add a lot of noise. I'd deprioritize.

.gitignore

scripts/update-algolia-records.ts

#2961) Co-authored-by: Michael Novotny <manovotny@gmail.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

NWylynko added 7 commits January 10, 2026 01:43

Add a script to generate algolia search records from the dist docs

07b8494

Include the branch in the record to filter by branch in searching

0b8313c

move algolia search dep to dev deps

11dcd4a

Add transient dep

1137aef

Actually use remarkFrontmatter instead of just doing a regex find

23cf413

combine the scripts in to just one script

33deb5f

Include a batch id and then clean up old batches after updating

56f7f8a

use more environment variables to configure script

ab9f693

vercel bot had a problem deploying to Preview January 13, 2026 19:32 Failure

cut out the usage of a pushTask and just update the records directly …

5f33be8

…in the index

vercel bot had a problem deploying to Preview January 14, 2026 13:39 Failure

NWylynko added 2 commits January 15, 2026 00:43

cut down the attributes in each search record

88509c1

Merge branch 'main' into nick/algolia-search-records

eed2f91

vercel bot had a problem deploying to Preview January 14, 2026 16:45 Failure

clean up

67f575f

vercel bot had a problem deploying to Preview January 14, 2026 17:51 Failure

clean up by using the batch api

9a9bd68

vercel bot had a problem deploying to Preview January 14, 2026 18:23 Failure

Strip out backticks from titles

26d9263

vercel bot had a problem deploying to Preview January 14, 2026 19:10 Failure

fix page ranking

27b31ed

vercel bot deployed to Preview January 14, 2026 19:33 View deployment

remove tags

fad6d5b

vercel bot deployed to Preview January 14, 2026 19:56 View deployment

NWylynko marked this pull request as ready for review January 14, 2026 20:33

NWylynko requested a review from a team as a code owner January 14, 2026 20:33

NWylynko requested a review from manovotny January 14, 2026 20:33

manovotny reviewed Jan 14, 2026

View reviewed changes

Add --dry-run flag to Algolia update script

05857ab

Allows running the script without pushing changes to Algolia. In dry run mode, outputs records to .algolia/records.json and stale record IDs to .algolia/stale.json for inspection. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

NWylynko added 2 commits January 15, 2026 22:24

formatting

e4d650a

Remove callout syntax and using callout custom ids for anchors

480a5a7

vercel bot deployed to Preview January 15, 2026 14:46 View deployment

Skip over tooltip content from being included in records

0b54a2c

vercel bot deployed to Preview January 15, 2026 14:56 View deployment

combine property details in to meaningful records

b531eb9

vercel bot deployed to Preview January 15, 2026 15:07 View deployment

NWylynko requested a review from manovotny January 15, 2026 15:09

Add a new record type of property and derank it

d0214e2

vercel bot deployed to Preview January 15, 2026 16:37 View deployment

parse tables in to records

2821d0f

vercel bot deployed to Preview January 15, 2026 17:33 View deployment

manovotny requested changes Jan 16, 2026

View reviewed changes

.gitignore Outdated Show resolved Hide resolved

scripts/update-algolia-records.ts Show resolved Hide resolved

scripts/update-algolia-records.ts Show resolved Hide resolved

scripts/update-algolia-records.ts Show resolved Hide resolved

Move the env check and algolia client setup after the dry run early exit

7dc914b

vercel bot deployed to Preview January 16, 2026 18:30 View deployment

manovotny approved these changes Jan 16, 2026

View reviewed changes

switch to detecting git branch

2ae2224

vercel bot deployed to Preview January 16, 2026 19:22 View deployment

NWylynko merged commit 9dc9328 into main Jan 16, 2026
8 checks passed

NWylynko deleted the nick/algolia-search-records branch January 16, 2026 19:22

NWylynko added a commit that referenced this pull request Jan 16, 2026

New script to break markdown in to Algolia search records and publish. (

c688f17

#2961) Co-authored-by: Michael Novotny <manovotny@gmail.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

NWylynko mentioned this pull request Jan 16, 2026

Cherry pick algolia records updator script into core1 docs #2975

Merged

NWylynko added a commit that referenced this pull request Jan 16, 2026

New script to break markdown in to Algolia search records and publish. (

5a80326

#2961) Co-authored-by: Michael Novotny <manovotny@gmail.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

NWylynko mentioned this pull request Jan 16, 2026

Cherry pick algolia records updater script into core3 docs #2976

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New script to break markdown in to Algolia search records and publish.#2961

New script to break markdown in to Algolia search records and publish.#2961
NWylynko merged 27 commits intomainfrom
nick/algolia-search-records

NWylynko commented Jan 13, 2026 •

edited

Loading

Uh oh!

vercel bot commented Jan 13, 2026 •

edited

Loading

Uh oh!

manovotny Jan 14, 2026

Uh oh!

NWylynko Jan 15, 2026

Uh oh!

NWylynko commented Jan 15, 2026

Uh oh!

NWylynko commented Jan 15, 2026

Uh oh!

NWylynko commented Jan 15, 2026

Uh oh!

manovotny commented Jan 15, 2026

Uh oh!

manovotny commented Jan 15, 2026

Uh oh!

manovotny commented Jan 15, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

NWylynko commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔎 Previews:

What does this solve?

What changed?

Uh oh!

vercel bot commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

manovotny Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

NWylynko Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

NWylynko commented Jan 15, 2026

Uh oh!

NWylynko commented Jan 15, 2026

Uh oh!

NWylynko commented Jan 15, 2026

Uh oh!

manovotny commented Jan 15, 2026

Uh oh!

manovotny commented Jan 15, 2026

Uh oh!

manovotny commented Jan 15, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

NWylynko commented Jan 13, 2026 •

edited

Loading

vercel bot commented Jan 13, 2026 •

edited

Loading