Skip to content

feat: add getContents() for reading external markdown files into table descriptions#2138

Merged
kolina merged 8 commits into
dataform-co:mainfrom
AmmanuelT:read_description_from_external_file
Apr 21, 2026
Merged

feat: add getContents() for reading external markdown files into table descriptions#2138
kolina merged 8 commits into
dataform-co:mainfrom
AmmanuelT:read_description_from_external_file

Conversation

@AmmanuelT

Copy link
Copy Markdown
Contributor

Closes #1723

Allows users to source table descriptions from external markdown files, keeping .sqlx files smaller and enabling reusable, renderable descriptions across the project.

Usage:

config {
  type: "table",
  description: getContents('./my_table_description.md')
}
SELECT ...

Changes:

  • core/compilers.ts : .md files now compile to exports.contents = ...``
  • core/session.ts : new getContents(filePath) method; resolves the path relative to the calling file using the call stack, then builds an absolute path for nativeRequire
  • core/main.ts : getContents bound to the global scope
  • cli/vm/compile.ts + testing/run_core.ts: "md" added to sourceExtensions so vm2 applies the compiler to .md files
  • core/main_test.ts + core/compilers_test.ts : tests for happy path and missing file error

Notes:

  • The method is currently named getContents, inspired by the suggestion in the issue thread, but happy to rename before merge if the maintainers have a preference!
  • When getContents references a file that doesn't exist, the error surfaces as a VMError: Cannot find module '/absolute/path/to/file.md' compilation error. A cleaner message could be produced by intercepting in dataformCompile in main.ts also happy to add this if preferred, just wanted to flag the tradeoff before doing extra work.

@AmmanuelT AmmanuelT requested a review from a team as a code owner April 3, 2026 16:06
@AmmanuelT AmmanuelT requested review from krushangSk17 and removed request for a team April 3, 2026 16:06
@kolina

kolina commented Apr 11, 2026

Copy link
Copy Markdown
Contributor

/gcbrun

@kolina kolina left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much for your contribution!

Overall looks good, I've left some not very big comments to address.

Currently tests are failing due to expired BQ credentials, can you please rebase your PR onto #2139 so I can re-run them?

Comment thread core/session.ts Outdated
Comment thread docs/reference/session.md Outdated
Comment thread core/session.ts
@AmmanuelT

Copy link
Copy Markdown
Contributor Author

Thanks for the comments! Working on the fixes!

@AmmanuelT AmmanuelT force-pushed the read_description_from_external_file branch from 8cf17c1 to cf359f2 Compare April 11, 2026 17:54
@AmmanuelT

Copy link
Copy Markdown
Contributor Author

@kolina ready for another check 🫡

Comment thread core/session.ts Outdated
@kolina

kolina commented Apr 21, 2026

Copy link
Copy Markdown
Contributor

/gcbrun

@kolina kolina merged commit e7dc2b5 into dataform-co:main Apr 21, 2026
3 checks passed
kolina added a commit that referenced this pull request May 4, 2026
@kolina

kolina commented May 4, 2026

Copy link
Copy Markdown
Contributor

@AmmanuelT, unfortunately I had to revert your commit in #2155 because it broke compilation in GCP.

I think the reason for breakage is that we use pure V8 for compilation in GCP and we can't use NodeJs modules like path here. Feel free to send a new PR, I think it'll be quite feasible to remove these dependency: you use it for join call here, you can instead use our helper

@AmmanuelT

Copy link
Copy Markdown
Contributor Author

Sorry about that! I'll get that replaced!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Read table description from external files

2 participants