Skip to content

fix(tsgo): align semantic data positions with AST encoder and sanitize symbol names#546

Merged
fansenze merged 1 commit intomainfrom
fix/semantic-positions-20260318
Mar 18, 2026
Merged

fix(tsgo): align semantic data positions with AST encoder and sanitize symbol names#546
fansenze merged 1 commit intomainfrom
fix/semantic-positions-20260318

Conversation

@fansenze
Copy link
Copy Markdown
Contributor

@fansenze fansenze commented Mar 18, 2026

Summary

  • Convert semantic data positions (node2sym, node2type, symbol declarations) from UTF-8 byte offsets to UTF-16 code units using positionMap.UTF8ToUTF16(), matching the AST encoder's position encoding. This fixes symbol/type lookup failures for source files containing non-BMP characters (e.g. emoji 💀) where UTF-8 and UTF-16 offsets diverge.
  • Sanitize internal symbol names by replacing the \xFE prefix (InternalSymbolNamePrefix) with __ to ensure valid UTF-8 output for consumers.
  • Fix snapshotSemantic test helper to also use UTF-16 positions for lookups.

Related Links

Checklist

  • Tests updated (or not required).
  • Documentation updated (or not required).

@gemini-code-assist
Copy link
Copy Markdown

gemini-code-assist bot commented Mar 18, 2026

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses critical issues related to character encoding and symbol naming within the tsgo semantic data generation. By aligning position encoding with UTF-16 and sanitizing internal symbol names, it ensures accurate semantic data lookup and valid output, particularly for source files containing complex characters like emojis. This improves the robustness and compatibility of the semantic analysis with downstream consumers.

Highlights

  • Position Encoding Alignment: Converted semantic data positions (node2sym, node2type, symbol declarations) from UTF-8 byte offsets to UTF-16 code units, aligning with the AST encoder's position encoding. This resolves issues with source files containing non-BMP characters where UTF-8 and UTF-16 offsets diverge.
  • Symbol Name Sanitization: Sanitized internal symbol names by replacing the \xFE prefix with __ to ensure valid UTF-8 output for consumers.
  • Test Helper Update: Fixed the snapshotSemantic test helper to also use UTF-16 positions for lookups, ensuring consistency with the new encoding.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist[bot]
gemini-code-assist bot reviewed Mar 18, 2026
View reviewed changes
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request effectively addresses issues with character encoding in semantic data positions and sanitizes internal symbol names for valid UTF-8 output. The conversion to UTF-16 code units for positions is a crucial fix for handling non-BMP characters correctly. The changes are consistently applied, and the new tests provide good coverage for the fixes. I have one suggestion regarding the implementation of the symbol name sanitization to improve clarity and align it more closely with its documented purpose.

@fansenze
fix(tsgo): align semantic data positions with AST encoder and sanitiz…
36c8879 
…e symbol names

The AST encoder converts node positions to UTF-16 code units via
positionMap.UTF8ToUTF16(), but the semantic data (node2sym, node2type)
was using raw UTF-8 byte offsets. This caused symbol/type lookup
failures for source files containing non-BMP characters (e.g. emoji)
where UTF-8 and UTF-16 offsets diverge.

Also sanitize internal symbol names by replacing the \xFE prefix
(InternalSymbolNamePrefix) with "__" to ensure valid UTF-8 output.
@fansenze fansenze force-pushed the fix/semantic-positions-20260318 branch from 99452c4 to 36c8879 Compare March 18, 2026 11:21
@fansenze fansenze requested a review from hardfist March 18, 2026 11:27
@fansenze fansenze merged commit be0f134 into main Mar 18, 2026
14 checks passed
@fansenze fansenze deleted the fix/semantic-positions-20260318 branch March 18, 2026 11:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hardfist hardfist hardfist approved these changes

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

release: bug fix

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@fansenze @hardfist