[RFC] feat(napi): Typed SgNode and SgRoot#1661
[RFC] feat(napi): Typed SgNode and SgRoot#1661HerringtonDarkholme merged 2 commits intoast-grep:mainfrom
Conversation
WalkthroughThe changes in this pull request enhance type definitions and functionality within the Changes
Possibly related PRs
Suggested reviewers
Tip CodeRabbit's docstrings feature is now available as part of our Early Access Program! Simply use the command Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Outside diff range and nitpick comments (4)
crates/napi/types/node-types.ts (1)
35-47: Consider renaming the type parameter 'Map' to avoid confusionUsing 'Map' as a type parameter name may cause confusion with the built-in
Maptype in JavaScript and TypeScript. Renaming it to something more descriptive like 'NodeType' or 'Schema' can improve readability and maintainability.Apply this diff to rename the type parameter:
-export type FieldTypeMeta< - Map extends NodeTypeSchema, - F extends FieldNames<Map> -> = Map["fields"] extends Record< +export type FieldTypeMeta< + NodeType extends NodeTypeSchema, + F extends FieldNames<NodeType> +> = NodeType["fields"] extends Record< string, { types: ReadonlyArray<{ type: string }> } > - ? Map["fields"][F] + ? NodeType["fields"][F] : { required: false; types: [{ type: string }]; };crates/napi/scripts/generate-types.ts (1)
59-75: Consider fetching node types in parallel to improve performanceCurrently, the
generateLangNodeTypesfunction fetches node types sequentially, which may be time-consuming. Fetching all node types in parallel usingPromise.allcan significantly improve the script's performance.Apply this diff to fetch node types in parallel:
async function generateLangNodeTypes() { - for (const [lang, url] of Object.entries(languagesNodeTypesUrls)) { - const nodeTypesResponse = await fetch(url); - const nodeTypes = (await nodeTypesResponse.json()) as NodeTypeSchema[]; - const nodeTypeMap = Object.fromEntries( - nodeTypes.map((node) => [node.type, node]) - ); - await writeFile( - path.join(dirname, "..", "types", `${lang}-node-types.ts`), - `type ${lang}NodeTypesMap = ${JSON.stringify(nodeTypeMap, null, 2)}; - -export default ${lang}NodeTypesMap; -` - ); - } + await Promise.all( + Object.entries(languagesNodeTypesUrls).map(async ([lang, url]) => { + const nodeTypesResponse = await fetch(url); + const nodeTypes = (await nodeTypesResponse.json()) as NodeTypeSchema[]; + const nodeTypeMap = Object.fromEntries( + nodeTypes.map((node) => [node.type, node]) + ); + await writeFile( + path.join(dirname, "..", "types", `${lang}-node-types.ts`), + `type ${lang}NodeTypesMap = ${JSON.stringify(nodeTypeMap, null, 2)}; + +export default ${lang}NodeTypesMap; +` + ); + }) + ); }crates/napi/src/sg_node.rs (2)
82-86: LGTM! Consider enhancing documentation.The new
ismethod provides a type-safe way to check node kinds. Consider adding an example usage in the documentation to demonstrate its benefits over direct string comparison.Add a code example to the documentation:
/// Check if the node is the same kind as the given `kind` string + /// + /// # Example + /// ```typescript + /// if (node.is("FunctionDeclaration")) { + /// // Handle function declaration + /// } + /// ``` #[napi] pub fn is(&self, kind: String) -> bool {
217-217: LGTM! Consider pre-allocating vector.The TypeScript return type
Array<SgNode<M>>correctly specifies an array of matched nodes.Consider pre-allocating the result vector to avoid reallocations:
- let mut ret = vec![]; + let mut ret = Vec::with_capacity(all_matches.len());
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
crates/napi/yarn.lockis excluded by!**/yarn.lock,!**/*.lock
📒 Files selected for processing (7)
crates/napi/index.d.ts(11 hunks)crates/napi/package.json(3 hunks)crates/napi/scripts/generate-types.ts(1 hunks)crates/napi/src/sg_node.rs(6 hunks)crates/napi/tsconfig.json(1 hunks)crates/napi/types/.gitignore(1 hunks)crates/napi/types/node-types.ts(1 hunks)
✅ Files skipped from review due to trivial changes (2)
- crates/napi/types/.gitignore
- crates/napi/tsconfig.json
🔇 Additional comments (11)
crates/napi/index.d.ts (4)
145-147: Confirm the correctness of the kind method's return type
The kind() method now returns the generic type T. Verify that all usages of this method accommodate the change from returning a string to T, and that T correctly represents the node kind.
147-147: Validate the type guard implementation in is method
The is<K extends T>(kind: K): this is SgNode<M, K> & this method uses a TypeScript type guard. Ensure that this syntax is correct and functions as intended across all supported TypeScript versions.
168-171: Check the correctness of the field and fieldChildren methods' type definitions
The methods field and fieldChildren have been updated to use generic type parameters based on FieldNames<M[T]>. Verify that these methods correctly handle optional and required fields according to the node type schema.
137-139: Ensure type parameter defaults are correctly specified
In the SgNode class definition, the default type for T is keyof M. Ensure that M is appropriately constrained to avoid possible issues with empty or incorrect mappings.
Run the following script to check if M always has keys:
✅ Verification successful
Let me gather more information about the NodeTypeSchema type to ensure it's properly constrained.
Type parameter defaults are correctly specified and safe
The type parameter defaults are properly constrained:
NodeTypesMapis defined asRecord<string, NodeTypeSchema>, ensuring it always has string keysNodeTypeSchemais a well-defined type with required fields liketypeandnamed- The code shows extensive usage across multiple language-specific implementations (JavaScript, TypeScript, Java, etc.), all extending the base
NodeTypesMap - The default
T extends keyof M = keyof Mis safe becauseMis constrained toNodeTypesMap, which guarantees string keys
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Verify that NodeTypesMap has keys and is not empty.
# Search for all declarations of NodeTypesMap and ensure they have keys
rg 'export type NodeTypesMap = Record<string, NodeTypeSchema>;' -A 5
# Check usage of SgNode to ensure M extends NodeTypesMap with keys
rg 'SgNode<\s*[^,]+,\s*[^>]+>'
Length of output: 1398
Script:
#!/bin/bash
# Check NodeTypeSchema definition and its fields
rg "type NodeTypeSchema" -A 10
# Check if there are any concrete implementations or usage of NodeTypesMap
rg "NodeTypesMap" -A 3
Length of output: 12802
crates/napi/package.json (2)
46-51: Verify the integration of the new typegen script in the build process
The build script now includes && yarn run typegen, and a new typegen script is added. Ensure that this script executes correctly in all environments and that tsx is the appropriate tool for running TypeScript scripts in this context.
55-59: Confirm compatibility of added dependencies
The new dependencies @types/node and tsx have been added. Verify that their versions are compatible with existing dependencies and that they do not introduce conflicts.
Run the following script to check for dependency issues:
✅ Verification successful
Let me try a different approach to verify the dependencies compatibility.
Let me analyze the package.json content and check for potential conflicts.
Based on the analysis of the package.json files, I can now provide a final response:
Dependencies are compatible and appropriately configured
The newly added dependencies @types/node and tsx in crates/napi/package.json are:
- Compatible with the Node.js engine requirement (">= 10")
- Properly scoped as
devDependencies - Aligned with other TypeScript-related dependencies (typescript@5.7.2, ts-node@10.9.2)
- Not conflicting with dependencies in other package.json files
@types/nodeversion is consistent with the one used in benches (^22.10.2 vs 22.9.4)
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Check for dependency conflicts and outdated packages.
# Install dependencies
yarn install
# List outdated packages
yarn outdated
# Audit for vulnerabilities
yarn audit
Length of output: 1861
Script:
#!/bin/bash
# Let's examine the package.json files to understand the dependency relationships
# First, let's see the full package.json content
cat crates/napi/package.json
# Then, let's check if there are any other package.json files that might have related dependencies
fd package.json
Length of output: 2558
Script:
#!/bin/bash
# Let's check the content of the main npm package.json and benches/package.json
# to see if there are any conflicting dependencies
cat npm/package.json
cat benches/package.json
# Also check if tsx or @types/node are used in the codebase
rg -l "from 'tsx'" .
rg -l "require\(['\"](tsx|@types/node)" .
Length of output: 1851
crates/napi/src/sg_node.rs (5)
166-166: LGTM! Type-safe children array.
The TypeScript return type Array<SgNode<M>> correctly specifies that the method returns an array of child nodes with preserved type information.
178-178: LGTM! Nullable type annotation.
The TypeScript return type SgNode<M> | null accurately represents that the method may or may not find a matching node.
284-284: LGTM! Nullable type annotation.
The TypeScript return type SgNode<M> | null accurately represents that the nth child may not exist.
367-367: LGTM! Non-nullable type annotation.
The TypeScript return type SgNode<M> correctly indicates that the root node always exists.
78-81: LGTM! Verify type parameter usage.
The addition of the TypeScript return type T improves type safety by providing more precise type information about node kinds.
Let's verify the type parameter usage in TypeScript:
✅ Verification successful
TypeScript return type annotation is correctly implemented
The kind() method's TypeScript return type T is properly used in the class definition where T is a generic type parameter constrained to be a key of NodeTypesMap. This ensures type safety by returning the specific node kind type instead of just a string, which aligns with the PR's goal of improving developer experience through better type information.
The implementation correctly maps the Rust string return type to the TypeScript generic type T, providing precise type information about possible node kinds in the TypeScript interface.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Check how the type parameter T is used in TypeScript definitions
rg -A 5 "class SgNode" crates/napi/index.d.ts
Length of output: 188
Script:
#!/bin/bash
# Let's check the full TypeScript interface to understand how T is used
rg -B 2 -A 15 "class SgNode" crates/napi/index.d.ts
# Also check if NodeTypesMap is defined in the same file
rg "type NodeTypesMap" crates/napi/index.d.ts
Length of output: 725
c281d02 to
9f4a8d9
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Outside diff range and nitpick comments (6)
crates/napi/types/node-types.ts (1)
48-54: Consider adding JSDoc comments for better documentation.The
GetSafeFieldTypeutility type plays a crucial role in type safety. Consider adding JSDoc comments to explain its purpose and usage.+/** + * Safely extracts the type of a field from the NodeTypesMap + * @template Map - The node types map + * @template K - The key type in the map + * @template F - The field name + * @template M - The field metadata + */ type GetSafeFieldType< Map extends NodeTypesMap, K extends keyof Map, F extends FieldNames<Map[K]>, M extends FieldTypeMeta<Map[K], F> = FieldTypeMeta<Map[K], F> > = M["types"][number]["type"];crates/napi/scripts/generate-types.ts (1)
78-257: Consider breaking down the updateIndexDts function.The function is quite long and handles multiple responsibilities. Consider breaking it down into smaller, focused functions:
- Type parameter updates
- Method signature updates
- Import/export generation
+async function updateTypeParameters(root: any) { + const sgRootClass = root.find({ + rule: createMatchClassDeclarationRule("SgRoot"), + }); + return { + sgRootClassTypeParametersRemovalEdit: sgRootClass! + .field("type_parameters") + ?.replace(""), + sgRootNameEdit: sgRootClass! + .field("name")! + .replace("SgRoot<M extends NodeTypesMap = NodeTypesMap>") + }; +} +async function updateMethodSignatures(root: any) { + const sgNodeClass = root.find({ + rule: createMatchClassDeclarationRule("SgNode"), + }); + // ... method signature updates +} async function updateIndexDts() { const indexDtsPath = path.join(dirname, "..", "index.d.ts"); const indexDtsSource = await readFile(indexDtsPath, "utf8"); const sgRoot = await parseAsync(Lang.TypeScript, indexDtsSource); const root = sgRoot.root(); - // ... rest of the function + const { sgRootClassTypeParametersRemovalEdit, sgRootNameEdit } = + await updateTypeParameters(root); + const methodSignatureEdits = await updateMethodSignatures(root); // ... continue with import/export generation }crates/napi/src/sg_node.rs (1)
78-78: Consider documenting TypeScript return type annotations.The TypeScript return type annotations improve type safety, but their purpose and relationship to the type system could be better documented.
+/// Returns the string name of the node kind +/// TypeScript: Returns the specific node type T from the NodeTypesMap #[napi(ts_return_type = "T")] pub fn kind(&self) -> String {Also applies to: 166-166, 178-178, 217-217, 296-296, 379-379
crates/napi/index.d.ts (3)
1-25: LGTM! Consider adding JSDoc comments for the node type maps.The imports and exports are well-organized, providing comprehensive type coverage for various languages. Consider adding JSDoc comments to document the structure and usage of these node type maps for better developer experience.
136-139: Excellent type safety improvements!The generic type parameters and type-narrowing methods significantly enhance type safety and developer experience:
M extends NodeTypesMapensures language-specific node typesT extends keyof Mensures valid node kindsis<K>()enables proper type narrowing- Field methods now provide accurate type information
Consider adding examples in JSDoc comments to demonstrate the usage of these typed methods.
Also applies to: 145-147, 171-173
197-203: Consider extracting common documentation to a shared location.The parseAsync documentation is duplicated across all language namespaces. Consider:
- Creating a shared documentation comment that can be referenced
- Using TSDoc's
@inheritDoctag to reduce duplicationAlso applies to: 219-225, 241-247, 263-269, 285-291, 307-313
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
crates/napi/yarn.lockis excluded by!**/yarn.lock,!**/*.lock
📒 Files selected for processing (7)
crates/napi/index.d.ts(12 hunks)crates/napi/package.json(3 hunks)crates/napi/scripts/generate-types.ts(1 hunks)crates/napi/src/sg_node.rs(6 hunks)crates/napi/tsconfig.json(1 hunks)crates/napi/types/.gitignore(1 hunks)crates/napi/types/node-types.ts(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (3)
- crates/napi/types/.gitignore
- crates/napi/tsconfig.json
- crates/napi/package.json
🔇 Additional comments (5)
crates/napi/types/node-types.ts (1)
3-24: LGTM! Well-structured type schema definition.
The NodeTypeSchema type provides a comprehensive and type-safe way to represent tree-sitter node types with proper generic constraints.
crates/napi/scripts/generate-types.ts (1)
59-76:
Add error handling for network and file operations.
The function lacks error handling for network requests and file operations.
crates/napi/src/sg_node.rs (1)
83-86: LGTM! Clear and concise implementation.
The is method provides a straightforward way to check node kinds.
crates/napi/index.d.ts (2)
185-187: LGTM! Type-safe root node access.
The SgRoot class properly propagates the generic type parameter to ensure type consistency with SgNode.
196-196: LGTM! Comprehensive type coverage for all languages.
Each language namespace correctly uses its specific NodeTypesMap, providing proper type information for parse, parseAsync, and findInFiles methods.
Also applies to: 204-204, 214-214, 218-218, 226-226, 236-236, 240-240, 248-248, 258-258, 262-262, 270-270, 280-280, 284-284, 292-292, 302-302, 306-306, 314-314, 324-324
b72ee45 to
82dedfb
Compare
82dedfb to
11527c1
Compare
|
Thanks for this amazing PR! This can resolve the long-standing issue #48 |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1661 +/- ##
=======================================
Coverage 87.11% 87.11%
=======================================
Files 96 96
Lines 15458 15458
=======================================
Hits 13467 13467
Misses 1991 1991 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Actionable comments posted: 4
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
crates/napi/yarn.lockis excluded by!**/yarn.lock,!**/*.lock
📒 Files selected for processing (10)
crates/napi/__test__/index.spec.ts(2 hunks)crates/napi/index.d.ts(11 hunks)crates/napi/package.json(3 hunks)crates/napi/scripts/constants.ts(1 hunks)crates/napi/scripts/generate-types.ts(1 hunks)crates/napi/scripts/rules.ts(1 hunks)crates/napi/src/sg_node.rs(6 hunks)crates/napi/tsconfig.json(1 hunks)crates/napi/types/.gitignore(1 hunks)crates/napi/types/node-types.ts(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (4)
- crates/napi/types/.gitignore
- crates/napi/tsconfig.json
- crates/napi/package.json
- crates/napi/types/node-types.ts
🔇 Additional comments (8)
crates/napi/scripts/constants.ts (1)
1-93: LGTM!
The constants are well-defined and correctly map languages to their respective crate names and node type URLs. This enhances maintainability and readability.
crates/napi/scripts/generate-types.ts (1)
54-56:
Ensure the script exits on errors during node types generation
Currently, if an error occurs while generating node types, the error is logged, but the script continues execution. This might lead to incomplete or inconsistent type definitions. Consider exiting the process upon encountering an error to ensure the issue is addressed promptly.
Apply this diff to exit the process on error:
} catch (e) {
console.error(`Error while generating node types for ${lang}:`, e);
+ process.exit(1);
}crates/napi/index.d.ts (4)
1-47: Well-organized type system with comprehensive language support!
The type system is well-structured with a clear separation of concerns. The comprehensive coverage of programming languages will enable better type safety across different language parsers.
Line range hint 158-200: Excellent type-safety improvements in SgNode!
The addition of generic type parameters and type-safe methods significantly improves the developer experience:
M extends NodeTypesMapensures type safety across different language node typesis<K extends T>provides type narrowing capabilitiesfieldandfieldChildrenmethods are now properly typed with the node's field names
Line range hint 201-208: Type-safe improvements to SgRoot class!
The addition of the generic type parameter M extends NodeTypesMap and its propagation to the root() method ensures type safety throughout the AST traversal chain.
Line range hint 209-341: Consistent type improvements across language namespaces!
The changes maintain consistency across all language namespaces (html, js, jsx, ts, tsx, css) with proper type propagation:
parsereturns language-specificSgRootfindInFilescallback receives properly typedSgNodearrays
crates/napi/__test__/index.spec.ts (2)
Line range hint 1-400: Comprehensive test coverage maintained!
The test suite provides excellent coverage of the API functionality:
- Tests both sync and async operations
- Covers node traversal, pattern matching, and file operations
- Includes edge cases like Unicode handling and range-based node finding
268-269: Verify test coverage with narrowed search paths
The test paths have been narrowed from './' to './__test__/'. While this makes the tests more focused, we should verify that this doesn't reduce test coverage.
Also applies to: 294-295
✅ Verification successful
Test coverage is maintained with narrowed search paths
After reviewing the test file, the change from './' to './__test__/' in the test paths is appropriate because:
- The test specifically verifies the functionality of
findInFileswithawait_expressionnodes, which are present in the test file itself - The test's purpose is to validate the filename retrieval functionality, not to perform a comprehensive search
- Other tests in the file already cover broader search scenarios using
'./'path
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Check for potential test files outside __test__ directory
# that might be missed by the updated paths
# Look for potential test files outside __test__
fd -e ts -e tsx --exclude '__test__' '.*\.spec\.|.*\.test\.' .
# Look for files that might need testing based on the findInFiles usage
rg -l 'findInFiles' . --type ts --type tsx --exclude '*.spec.*' --exclude '*.test.*'
Length of output: 559
Script:
#!/bin/bash
# Let's try with correct ripgrep syntax and split into multiple commands
# Look for potential test files outside __test__
fd -e ts -e tsx --exclude '__test__' '.*\.spec\.|.*\.test\.' .
# Look for files that use findInFiles
rg -l 'findInFiles' --type ts --type tsx . | grep -v '\.spec\.' | grep -v '\.test\.'
# Check the actual test file to understand the test coverage
cat crates/napi/__test__/index.spec.ts
Length of output: 11416
HerringtonDarkholme
left a comment
There was a problem hiding this comment.
LGTM, let's improve this in following PRs
When using
napi, I generally don't know what fields and kinds I'm dealing with for each language. I need to use either ast-grep or treesitter playgrounds with some sample code to know what's possible and it's generally not a good DX and might lead to (1) not covering edge cases (2) type errorsI added a script that downloads compiled treesitter node-types from the corresponding repos that's a processed version of the TS grammars that allows us to have some context about the possible kinds and fields.
I created a napi script that post-processes the index.d.ts file. Alternatively, I could introduce some proc-macros in the rust code, but that was way harder to implement and a huge headache and wasn't sure even if it's gonna work. The limitation is basically in napi not support type parameters like having it output
class SgNode<T, M>instead ofclass SgNode. And also dynamically craft the type parameters with the macro. Happy to hop on a call on Discord to explain in more details.Screen.Recording.2024-12-13.at.10.12.12.PM.mov
Summary by CodeRabbit
New Features
Bug Fixes
Documentation
.gitignoreto better manage TypeScript files.Chores
package.json.