You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
My own task or dataset (give details below)
Reproduction
Run transformers add-new-model-like
Select a model which is inside TOKENIZER_MAPPING_NAMES such as qwen2_5_vl
Provide the new model name and fully cased name
Error:
root@51e8d26e057b:/# transformers add-new-model-like
What model would you like to duplicate? Please provide it as lowercase, e.g. `llama`): qwen2_5_vl
What is the new model name? Please provide it as snake lowercase, e.g. `new_model`? test_model
What is the fully cased name you would like to appear in the doc (e.g. `NeW ModEl`)? [TestModel]
Traceback (most recent call last):
File "/usr/local/bin/transformers", line 33, in<module>
sys.exit(load_entry_point('transformers', 'console_scripts', 'transformers')())
File "/transformers/src/transformers/cli/transformers.py", line 39, in main
app()
File "/usr/local/lib/python3.10/dist-packages/typer/main.py", line 1152, in __call__
raise e
File "/usr/local/lib/python3.10/dist-packages/typer/main.py", line 1135, in __call__
return get_command(self)(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1485, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/typer/core.py", line 795, in main
return _main(
File "/usr/local/lib/python3.10/dist-packages/typer/core.py", line 188, in _main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1873, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1269, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 824, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/typer/main.py", line 1514, in wrapper
return callback(**use_params)
File "/transformers/src/transformers/cli/add_new_model_like.py", line 104, in add_new_model_like
) = get_user_input()
File "/transformers/src/transformers/cli/add_new_model_like.py", line 719, in get_user_input
if old_model_infos.tokenizer_class is not None:
AttributeError: 'ModelInfos' object has no attribute 'tokenizer_class'. Did you mean: 'fast_tokenizer_class'?
The issue is due to the following code, which in the case self.lowercase_name in TOKENIZER_MAPPING_NAMES == True does not set self.tokenizer_class. self.tokenizer_class is later accessed inside of add-new-model-like
System Info
transformersversion: 5.3.0.dev0Who can help?
@ydshieh
Information
Tasks
examplesfolder (such as GLUE/SQuAD, ...)Reproduction
transformers add-new-model-likeTOKENIZER_MAPPING_NAMESsuch asqwen2_5_vlError:
The issue is due to the following code, which in the case
self.lowercase_name in TOKENIZER_MAPPING_NAMES == Truedoes not setself.tokenizer_class.self.tokenizer_classis later accessed inside ofadd-new-model-liketransformers/src/transformers/cli/add_new_model_like.py
Lines 144 to 150 in ca960f0
This was changed in #40936, which removed setting
self.tokenizer_class.Happy to make a PR, fixing this but not sure what is the expected behaviour:
self.tokenizer_classas before rm slow tokenizers #40936self.tokenizer_classtoNoneadd-new-model-likeExpected behavior
The script runs successfully.