Skip to content

[model-gateway] fix tokenizer encode in golang bindings#16482

Merged
slin1237 merged 3 commits intosgl-project:mainfrom
WeiLai5432:fix_smg_bindings
Jan 5, 2026
Merged

[model-gateway] fix tokenizer encode in golang bindings#16482
slin1237 merged 3 commits intosgl-project:mainfrom
WeiLai5432:fix_smg_bindings

Conversation

@WeiLai5432
Copy link
Copy Markdown
Contributor

@WeiLai5432 WeiLai5432 commented Jan 5, 2026

fix the tokenizer encode in golang bindings

@github-actions github-actions Bot added dependencies Pull requests that update a dependency file model-gateway labels Jan 5, 2026
@gemini-code-assist
Copy link
Copy Markdown
Contributor

gemini-code-assist Bot commented Jan 5, 2026

Summary of Changes

Hello @WeiLai5432, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a bug in the tokenizer encoding within the Go bindings of the sgl-model-gateway. The core change involves enhancing the tokenizer.encode function to explicitly control the inclusion of special tokens, thereby rectifying previous incorrect tokenization behavior and providing more precise control over the encoding process.

Highlights

  • Tokenizer Function Signature Update: The tokenizer.encode function in the Rust codebase, exposed via the Go bindings, has been updated to accept an additional boolean parameter, add_special_tokens.
  • Explicit Special Token Handling: Existing calls to tokenizer.encode within client.rs and preprocessor.rs have been modified to explicitly pass false for the new add_special_tokens parameter, ensuring special tokens are not added in these contexts.
  • C FFI Type Definition: A platform-specific BooleanT type has been introduced in tokenizer.rs using the libc crate to ensure proper C Foreign Function Interface (FFI) compatibility for the new boolean parameter.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist[bot]
gemini-code-assist Bot reviewed Jan 5, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses a bug in the Go bindings where the tokenizer's encode method was not being called correctly. The fix involves passing add_special_tokens: false to the encode method for chat-related tokenization, which is the correct behavior after a chat template has been applied. Additionally, the sgl_tokenizer_encode FFI function is updated to expose the add_special_tokens parameter, providing more control to the Go layer. The changes are correct, well-implemented, and effectively resolve the tokenization issue. The introduction of the libc dependency for FFI-compatible boolean types is appropriate.

@WeiLai5432
Copy link
Copy Markdown
Contributor Author

WeiLai5432 commented Jan 5, 2026

Hello @whybeyoung ,
Please see this - the tokenizer trait was changed some days ago, but the corresponding Go bindings code wasn't updated to match these changes. This has resulted in compilation errors.

I've made the necessary changes to fix the issues. Please review to confirm the changes are correct.

@slin1237 slin1237 added the run-ci label Jan 5, 2026
@slin1237 slin1237 merged commit 23849eb into sgl-project:main Jan 5, 2026
71 of 72 checks passed
jamesjxliu pushed a commit to jamesjxliu/sglang that referenced this pull request Jan 6, 2026
@WeiLai5432 @jamesjxliu
[model-gateway] fix tokenizer encode in golang bindings (sgl-project#…
b1418b9 
…16482)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@slin1237 slin1237 slin1237 approved these changes

@CatherineSue CatherineSue Awaiting requested review from CatherineSue CatherineSue is a code owner

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

dependencies Pull requests that update a dependency file model-gateway run-ci

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@WeiLai5432 @slin1237