python /opt/NeMo/scripts/checkpoint_converters/convert_llama_nemo_to_hf.py \
    --input_name_or_path ${base_dir}/KD/distill_trainings/megatron_llama_distill/checkpoints/distilled_4b_model_extracted \
    --output_path ${base_dir}/ckpt/distilled_4b_model_llama3_8b_inst/pytorch_model.bin \
    --hf_input_path ${base_dir}/ckpt/distilled_4b_model_llama3_8b_inst \
    --hf_output_path ${base_dir}/ckpt/distilled_4b_model_llama3_8b_inst

converting layer 31
done layer 31
[NeMo I 2024-11-08 13:34:37 convert_llama_nemo_to_hf:228] Weights saved to /<base_dir>/ckpt/distilled_4b_model_llama3_8b_inst/pytorch_model.bin
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00,  3.55it/s]
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. 
The tokenizer class you load from this checkpoint is 'PreTrainedTokenizerFast'. 
The class this function is called from is 'LlamaTokenizer'.
[rank0]: Traceback (most recent call last):
[rank0]:   File "/opt/NeMo/scripts/checkpoint_converters/convert_llama_nemo_to_hf.py", line 275, in <module>
[rank0]:     replace_hf_weights_and_tokenizer(
[rank0]:   File "/opt/NeMo/scripts/checkpoint_converters/convert_llama_nemo_to_hf.py", line 249, in replace_hf_weights_and_tokenizer
[rank0]:     tokenizer = LlamaTokenizer.from_pretrained(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py", line 2213, in from_pretrained
[rank0]:     return cls._from_pretrained(
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py", line 2447, in _from_pretrained
[rank0]:     tokenizer = cls(*init_inputs, **init_kwargs)
[rank0]:   File "/opt/NeMo/nemo/lightning/io/mixin.py", line 577, in wrapped_init
[rank0]:     original_init(self, *args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/tokenization_llama.py", line 169, in __init__
[rank0]:     self.sp_model = self.get_spm_processor(kwargs.pop("from_slow", False))
[rank0]:   File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/tokenization_llama.py", line 199, in get_spm_processor
[rank0]:     with open(self.vocab_file, "rb") as f:
[rank0]: TypeError: expected str, bytes or os.PathLike object, not NoneType

python /opt/NeMo/scripts/checkpoint_converters/convert_llama_nemo_to_hf.py \
    --input_name_or_path ${base_dir}/KD/distill_trainings/megatron_llama_distill/checkpoints/distilled_4b_model_extracted \
    --output_path ${base_dir}/ckpt/distilled_4b_model_llama3_8b_inst/pytorch_model.bin \
    --hf_input_path ${base_dir}/ckpt/distilled_4b_model_llama3_8b_inst \
    --hf_output_path ${base_dir}/ckpt/distilled_4b_model_llama3_8b_inst_2 \
    --input_tokenizer ${base_dir}/ckpt/Meta-Llama-3-8B-Instruct \
    --hf_output_tokenizer ${base_dir}/ckpt/distilled_4b_model_llama3_8b_inst_2

"""
Script to convert a llama checkpoint in nemo (mcore path) into a HuggingFace checkpoint.
This script can be used to 1) generate only the HF weights, or 2) generate an entire HF model folder.

1) Generate only HF weights from a nemo file:

    python convert_llama_nemo_to_hf.py \
    --input_name_or_path /path/to/file.nemo or /path/to/extracted_folder \
    --output_path /path/to/pytorch_model.bin
    
2) Generate the full HF model folder

    python convert_llama_nemo_to_hf.py \
    --input_name_or_path /path/to/file.nemo or /path/to/extracted_folder \
    --output_path /path/to/pytorch_model.bin \
    --hf_input_path /path/to/input_hf_folder \
    --hf_output_path /path/to/output_hf_folder

3) Generate the full HF model folder with a custom tokenizer

    python convert_llama_nemo_to_hf.py \
    --input_name_or_path /path/to/file.nemo or /path/to/extracted_folder \
    --output_path /path/to/pytorch_model.bin \
    --hf_input_path /path/to/input_hf_folder \
    --hf_output_path /path/to/output_hf_folder \
    --input_tokenizer /path/to/custom_nemo_tokenizer.model \
    --hf_output_tokenizer /path/to/output_tokenizer

    Use the --cpu-only flag if the model cannot fit in the GPU (e.g. Llama2 70b). 
    However this option makes the conversion script significantly slower.
"""

    if tokenizer_path:
        try:
            tokenizer = LlamaTokenizer.from_pretrained(
                tokenizer_path,
                local_files_only=True,
                legacy=False,
            )
            tmp_tokenizer = convert_slow_tokenizer.convert_slow_tokenizer(tokenizer)
            fast_tokenizer = LlamaTokenizerFast(tokenizer_object=tmp_tokenizer)
            tokenizer_length = len(fast_tokenizer)
            model.resize_token_embeddings(tokenizer_length)
        except:
            tokenizer = None
            logger.warning("Could not load custom tokenizer, proceeding with default tokenizer")

    if tokenizer_path and (tokenizer is not None):
        fast_tokenizer.save_pretrained(output_hf_tokenizer)
        tokenizer.save_pretrained(output_hf_tokenizer)
        logging.info(f"Tokenizer saved to {output_hf_tokenizer}")