Skip to content

Out of Memory error ... NEMO MISTRAL error on saving as GGUF #2193

@CAISAMPS

Description

@CAISAMPS

Here is the ERROR.

nsloth: Merging 4bit and LoRA weights to 16bit...
Unsloth: Will use up to 24.38 out of 50.99 RAM for saving.
Unsloth: Saving model... This might take 5 minutes ...
55%|█████▌ | 22/40 [00:31<00:25, 1.43s/it]

OutOfMemoryError Traceback (most recent call last)
in <cell line: 0>()
1 # Save to multiple GGUF options - much faster if you want multiple!
2 if True:
----> 3 model.push_to_hub_gguf(
4 "AlSamCur123/NemoFineTune", # Change hf to your username! Nemo-merge-14b DeepSeek-R1-Distill-Qwen-14B
5 tokenizer,

3 frames
/usr/local/lib/python3.11/dist-packages/unsloth/save.py in merge_lora(layer, name)
175 # W.addmm
(A.t().to(W.dtype), B.t().to(W.dtype), alpha = s)
176 # if not torch.isfinite(W).all():
--> 177 maximum_element = torch.max(W.min().abs(), W.max())
178 if not torch.isfinite(maximum_element).item():
179 raise ValueError(f"Unsloth: Merge failed.\n{name} has some elements = infinity.")

OutOfMemoryError: CUDA out of memory. Tried to allocate 280.00 MiB. GPU 0 has a total capacity of 14.74 GiB of which 258.12 MiB is free. Process 38882 has 14.46 GiB memory in use. Of the allocated memory 14.11 GiB is allocated by PyTorch, and 199.47 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Colab ...

%%capture
import os
if "COLAB_" not in "".join(os.environ.keys()):
!pip install unsloth
else:
# Do this only in Colab notebooks! Otherwise use pip install unsloth
!pip install --no-deps bitsandbytes accelerate xformers==0.0.29.post3 peft trl triton cut_cross_entropy unsloth_zoo
!pip install sentencepiece protobuf datasets huggingface_hub hf_transfer
!pip install --no-deps unsloth
!pip install xformers --upgrade

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions