feat: support zai as inference provider#1766
Conversation
we need to allowlist your org on the Hub side. (we can add a note about this to the doc maybe @SBrandeis @Wauplin) |
Wauplin
left a comment
There was a problem hiding this comment.
Haven't tested yet but implementation looks good. Left a comment regarding text-generation vs chat-completion.
| export class ZaiTextGenerationTask extends BaseTextGenerationTask { | ||
| constructor() { | ||
| super("zai", ZAI_API_BASE_URL); | ||
| } | ||
|
|
||
| override prepareHeaders(params: HeaderParams, binary: boolean): Record<string, string> { | ||
| const headers = super.prepareHeaders(params, binary); | ||
| headers["x-source-channel"] = "hugging_face"; | ||
| return headers; | ||
| } | ||
|
|
||
| override makeRoute(): string { | ||
| return "/chat/completions"; | ||
| } | ||
| } |
There was a problem hiding this comment.
The text-generation task is meant for non-conversational LLMs. Do you plan to serve any? Usually Inference Providers are starting with conversational ones only i.e. "chat completion".
Note that a model cannot be served through Inference Providers both with the Chat Completion and the Text Generation APIs. If a model is tagged as "conversational" on the Hub (https://huggingface.co/models?other=conversational&sort=trending&search=zai-org), then it must be served as Chat Completion.
There was a problem hiding this comment.
If you do want to serve some non-conversational text-generation models, then I suppose the API route must be updated?
There was a problem hiding this comment.
hi @Wauplin Thanks for the reminder. We’re providing conversational models, I’ll update the pr.
There was a problem hiding this comment.
hi, i have removed the text-generation. PTAL, thanks
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
hi @julien-c , could you help add our team zai-org to the allowlist? we would like to register the provider as |
|
we try to have provider id === organization namespace on the Hub Would you rather have zai-org as the provider id? Or we can look into moving your org to hf.co/zai Let me and @SBrandeis know |
hanouticelina
left a comment
There was a problem hiding this comment.
thanks @tomsun28! this looks good
Co-authored-by: célina <hanouticelina@gmail.com>
hi @julien-c @SBrandeis , got it, Is this mandatory? If zai cannot be used without changing the organization, we can use zai-org as the provider. Thanks. |
SBrandeis
left a comment
There was a problem hiding this comment.
Sorry, overseen this
Took https://github.com/tomsun28/huggingface.js/blob/3a4b909e563ef7e86ffc600045e22be25f0579b9/packages/inference/src/providers/novita.ts for reference
Co-authored-by: Simon Brandeis <33657802+SBrandeis@users.noreply.github.com>
Co-authored-by: Simon Brandeis <33657802+SBrandeis@users.noreply.github.com>
hi @SBrandeis Thanks, the base_url has been updated. |
|
@SBrandeis Hi and could you help add us to the allowlist? so we can proceed with the model mapping registration process. |
|
Hey @tomsun28
Merging this PR is a pre-requisite before we can enable the model mapping API
It would be nice if we could match the provider name with the name of the org yes! |
Got it thanks, i will update the provider id. |
|
@tomsun28 you sure you don't want us to rename your org to hf.co/zai instead (all links will be redirected)? |
|
Hi @julien-c @SBrandeis the provider id has updated to zai-org, PTAL thanks |
Hi 🤗 team,
We’d like to apply to register ZAI as inference provider. Thanks
One more question: when we tried to register the model mapping with the endpoint
POST api/partners/zai/models, we received a 404 response. https://huggingface.co/docs/inference-providers/register-as-a-provider#3-model-mapping-api Does this action only work after the PR has been merged?