Fix Path Handling by Using Special Symbol Replacement#1005
Fix Path Handling by Using Special Symbol Replacement#1005FredericVAN wants to merge 2 commits intoShishirPatil:mainfrom
Conversation
| --num-gpus 1 \ | ||
| --gpu-memory-utilization 0.9 | ||
| ``` | ||
|
|
There was a problem hiding this comment.
We have recently updated the workflow for adding new models (#999). Please refer to README.md and CONTRIBUTING.md for more information on running new models, including locally hosted ones.
| continue | ||
|
|
||
| model_name_escaped = model_name.replace("_", "/") | ||
| model_name_escaped = model_name.replace("#S#L#A#S#H#", "/") |
There was a problem hiding this comment.
Perhaps "__" could be a better choice than "#S#L#A#S#H#". Also don't forget to search the code base for all occurences of replace("/", "_") and replace("_", "/").
There was a problem hiding this comment.
You are right, I will improve it later.Thanks
|
Hi @FredericVAN Thanks a lot for the PR! I’ve left two comments regarding your changes. Please let me know if you have any thoughts or feedback on my review as well 😃 |
Thank you for your comments, I will consider and modify my previous PR |
|
Hey @FredericVAN just wanted to check in on the status of this PR. Thanks |
|
This PR updates the inference mechanism for Google Gemini models, replacing the use of Google Vertex AI with Google AI Studio. In addition, this PR downgrades `tenacity` from 9.0.0 → 8.5.0 because `google-genai` currently pins `tenacity<9.0`. ---- **Compatibility note on tenacity** Our code does exercise the retry path affected by [jd/tenacity #425](jd/tenacity#425), but the issue has no functional impact on our evaluation accuracy. Therefore, the temporary downgrade is considered safe. We will revert to tenacity ≥9.0 once python-genai removes the <9.0 pin (tracked in [googleapis/python-genai #1005](googleapis/python-genai#1005)).
Description:
This Pull Request addresses an issue #1004 where model paths containing underscores were incorrectly split during processing. The solution involves using a special symbol replacement technique to avoid conflicts with existing underscores in model names.
Changes Made:
Modified the code ：ShishirPatil/gorilla.git/berkeley-function-call-leaderboard/bfcl/eval_checker/eval_runner.py
Reason for Change:
The previous implementation was causing incorrect path splitting when underscores were present in model names. By using a unique symbol sequence (#S#L#A#S#H#), we ensure that slashes are correctly restored without affecting other characters.
Testing:
Verified that model paths are correctly processed without splitting underscores.
Confirmed that the special symbol replacement does not interfere with other parts of the code.