Skip to content

llama : Add IBM granite template#10013

Merged
ngxson merged 11 commits intoggml-org:masterfrom
arch-btw:master
Oct 28, 2024
Merged

llama : Add IBM granite template#10013
ngxson merged 11 commits intoggml-org:masterfrom
arch-btw:master

Conversation

@arch-btw
Copy link
Contributor

@arch-btw arch-btw commented Oct 23, 2024

Hi @ggerganov and @ngxson,

I'd like to contribute the IBM granite template to llama.cpp.

I've made my best effort but I'm new to this, so could you please be so kind to review? Maybe you can see if it's alright.

I've followed the wiki instructions but feel free to make changes or provide feedback on how to improve.

The model is here: https://huggingface.co/ibm-granite/granite-3.0-8b-instruct

Here is the current output of llama-cli:

main: chat template example:
<|start_of_role|>system<|end_of_role|>
You are a helpful assistant<|end_of_text|>
<|start_of_role|>user<|end_of_role|>
Hello<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>
Hi there<|end_of_text|>
<|start_of_role|>user<|end_of_role|>
How are you?<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>

--

granite

@github-actions github-actions bot added the testing Everything test related label Oct 23, 2024
ngxson
ngxson requested changes Oct 23, 2024
View reviewed changes
@arch-btw @ngxson
Update src/llama.cpp
26f0911 
Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>
@arch-btw arch-btw changed the title Add IBM granite template llama : Add IBM granite template Oct 26, 2024
@arch-btw @ngxson
Update tests/test-chat-template.cpp
8fe174d 
Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>
@arch-btw
Copy link
Contributor Author

arch-btw commented Oct 26, 2024 •
edited
Loading

@ngxson thank you so much for your help, I appreciate your time and effort!

I have applied your review 👍

I have a quick question, when I run test-chat-template is it normal that it looks like this (screenshot below).
The part that confuses me is that <end_of_sentence> is not escaped but the other tags are.
And then user/assistant are on the same line, twice.

I'm not sure what it should look like in the context of this script, considering the other prompts have things like additional spacing as well.

Edit: sorry, I'm looking at the deepseek template. I guess mine failed:

terminate called after throwing an instance of 'std::length_error'
what(): vector::_M_default_append
fish: Job 1, './tests/test-chat-template' terminated by signal SIGABRT (Abort)

test-chat-template

@ngxson
Copy link
Collaborator

ngxson commented Oct 26, 2024 •
edited
Loading

You need to add an example of an unformatted chat template in test-chat-template. Please look carefully in the file.

@arch-btw
Copy link
Contributor Author

arch-btw commented Oct 27, 2024

Hi @ngxson

Thank you again for your guidance!
I finally managed to get it to work! 😅

Here is the output of make tests/test-chat-template && ./tests/test-chat-template:

outputgranite

And here is the output of ./llama-cli -m granite-3.0-8B-instruct-F32-Q5_K_M.gguf -t 4 -p 'You are granite, an AI model.' --conversation --color --chat-template granite:

outputgranite2

And a prompt example:

outputgranite3

@arch-btw arch-btw requested a review from ngxson October 27, 2024 01:22
ngxson
ngxson reviewed Oct 28, 2024
View reviewed changes
Copy link
Collaborator

@ngxson ngxson left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small changes before I can merge

arch-btw and others added 2 commits October 28, 2024 04:20
@arch-btw @ngxson
Add code space &
50ef6ca 
Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>
@arch-btw
Fix spacing
839cf4c
@arch-btw
Copy link
Contributor Author

arch-btw commented Oct 28, 2024

@ngxson Thank you again and sorry about that!

ngxson
ngxson approved these changes Oct 28, 2024
View reviewed changes
Copy link
Collaborator

@ngxson ngxson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, merging once the CI passes

ngxson
ngxson reviewed Oct 28, 2024
View reviewed changes
@ngxson ngxson merged commit 61715d5 into ggml-org:master Oct 28, 2024
gabe-l-hart added a commit to gabe-l-hart/llamafile that referenced this pull request Nov 4, 2024
@gabe-l-hart
feat(granite*): Add granite chat template
473e3fc 
Branch: GraniteThreeSupport

This is a port of the work done in llama.cpp with a slight tweak for the
tool call response:
ggml-org/llama.cpp#10013

Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart gabe-l-hart mentioned this pull request Nov 4, 2024
Merged
gabe-l-hart added a commit to gabe-l-hart/llamafile that referenced this pull request Nov 5, 2024
@gabe-l-hart
feat(granite*): Add granite chat template
bbe64fe 
Branch: GraniteThreeSupport

This is a port of the work done in llama.cpp with a slight tweak for the
tool call response:
ggml-org/llama.cpp#10013

Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart gabe-l-hart mentioned this pull request Nov 5, 2024
Merged
4 tasks
arthw pushed a commit to arthw/llama.cpp that referenced this pull request Nov 15, 2024
@arch-btw @ngxson @arthw
llama : Add IBM granite template (ggml-org#10013)
d8a469d 
* Add granite template to llama.cpp

* Add granite template to test-chat-template.cpp

* Update src/llama.cpp

Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>

* Update tests/test-chat-template.cpp

Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>

* Added proper template and expected output

* Small change to \n

Small change to \n

* Add code space &

Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>

* Fix spacing

* Apply suggestions from code review

* Update src/llama.cpp

---------

Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>
arthw pushed a commit to arthw/llama.cpp that referenced this pull request Nov 18, 2024
@arch-btw @ngxson @arthw
llama : Add IBM granite template (ggml-org#10013)
b20b8bf 
* Add granite template to llama.cpp

* Add granite template to test-chat-template.cpp

* Update src/llama.cpp

Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>

* Update tests/test-chat-template.cpp

Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>

* Added proper template and expected output

* Small change to \n

Small change to \n

* Add code space &

Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>

* Fix spacing

* Apply suggestions from code review

* Update src/llama.cpp

---------

Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>
gabe-l-hart added a commit to gabe-l-hart/llamafile that referenced this pull request Dec 10, 2024
@gabe-l-hart
feat(granite*): Add granite chat template
f2557e3 
Branch: GraniteThreeSupport

This is a port of the work done in llama.cpp with a slight tweak for the
tool call response:
ggml-org/llama.cpp#10013

Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
gabe-l-hart added a commit to gabe-l-hart/llamafile that referenced this pull request Mar 14, 2025
@gabe-l-hart
feat(granite*): Add granite chat template
f9ab15a 
Branch: GraniteThreeSupport

This is a port of the work done in llama.cpp with a slight tweak for the
tool call response:
ggml-org/llama.cpp#10013

Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ngxson ngxson ngxson approved these changes

Assignees

No one assigned

Labels

testing Everything test related

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@arch-btw @ngxson