Skip to content

[BigModeling] Add missing check for quantized models#1652

Merged
younesbelkada merged 4 commits intomainfrom
fix-to-int8
Jun 28, 2023
Merged

[BigModeling] Add missing check for quantized models#1652
younesbelkada merged 4 commits intomainfrom
fix-to-int8

Conversation

@younesbelkada
Copy link
Contributor

@younesbelkada younesbelkada commented Jun 28, 2023

What does this PR do?

Fixes huggingface/transformers#24540 and the failing test: https://github.com/huggingface/accelerate/actions/runs/5396594202/jobs/9800396637

Currently on the main branch loading a quantized model on a single GPU fails:

from transformers import AutoModelForCausalLM, AutoConfig, AutoTokenizer
import torch

model_path="facebook/opt-350m"

config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True, load_in_8bit=True, device_map="auto")

tokenizer = AutoTokenizer.from_pretrained(model_path)

input_text = "Describe the solar system."
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")

outputs = model.generate(input_ids, max_length=100)
print(tokenizer.decode(outputs[0]))

In #1648 it seems that a check was missing before calling .to to the model in case of single GPU model dispatching. Adding a small check circunvemts this issue

cc @sgugger @SunMarc

younesbelkada
younesbelkada commented Jun 28, 2023
View reviewed changes
src/accelerate/big_modeling.py Outdated
else:
device = list(device_map.values())[0]
if device != "disk":
if device != "disk" and not getattr(model, "is_quantized", False):
Copy link
Contributor Author

@younesbelkada younesbelkada Jun 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should maybe check with the attributes is_loaded_in_8bit or is_loaded_in_4bit as is_quantized has been only recently introduced. WDYT?

@younesbelkada younesbelkada mentioned this pull request Jun 28, 2023
Closed
4 tasks
@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Jun 28, 2023 •
edited
Loading

The documentation is not available anymore as the PR was closed or merged.

@younesbelkada younesbelkada requested review from SunMarc and sgugger June 28, 2023 06:46
younesbelkada
younesbelkada commented Jun 28, 2023
View reviewed changes
src/accelerate/big_modeling.py
device = list(device_map.values())[0]
if device != "disk":
# for backward compatibility
is_quantized = getattr(model, "is_quantized", False) or getattr(model, "is_loaded_in_8bit", False)
Copy link
Contributor Author

@younesbelkada younesbelkada Jun 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getattr(model, "is_loaded_in_8bit", False) --> for backward compatilibty for users that have an old version of transformers (before 4bit integration)

sgugger
sgugger approved these changes Jun 28, 2023
View reviewed changes
Copy link
Collaborator

@sgugger sgugger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your PR, added a small comment.

src/accelerate/big_modeling.py Outdated
Comment on lines 394 to 396
elif is_quantized:
pass
else:
Copy link
Collaborator

@sgugger sgugger Jun 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
elif is_quantized:
pass
else:
elif not is_quantized:

Copy link
Contributor Author

@younesbelkada younesbelkada Jun 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah added it already b93d93c before your review :D

SunMarc
SunMarc approved these changes Jun 28, 2023
View reviewed changes
Copy link
Member

@SunMarc SunMarc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work!

@younesbelkada younesbelkada merged commit a9d43cd into main Jun 28, 2023
@younesbelkada younesbelkada deleted the fix-to-int8 branch June 28, 2023 14:07
@younesbelkada younesbelkada mentioned this pull request Jun 28, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sgugger sgugger sgugger approved these changes

@SunMarc SunMarc SunMarc approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

4 participants

@younesbelkada @HuggingFaceDocBuilderDev @sgugger @SunMarc