Handle Transformers 5 vLLM import errors#3908
Conversation
Summary of ChangesHello @danielhanchen, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly improves the compatibility and user experience of the Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request addresses compatibility issues with Transformers v5 by adding error handling for vLLM imports and cleaning up unused imports.
In unsloth/import_fixes.py, you've added try-except blocks to catch ImportError when importing vllm, providing a clearer error message to users when an incompatible version of transformers is detected. This is a great improvement for user experience. However, there is significant code duplication in the error handling logic. I've suggested refactoring this into a helper function to improve maintainability.
The changes in unsloth/models/vision.py to remove the unused HybridCache import are clean and help prevent potential import errors with newer transformers versions.
Overall, good work on improving compatibility and error handling.
| try: | ||
| import vllm | ||
| except ImportError as e: | ||
| error_text = str(e) | ||
| if ( | ||
| "ALLOWED_LAYER_TYPES" in error_text | ||
| or "transformers.configuration_utils" in error_text | ||
| ): | ||
| try: | ||
| vllm_version = importlib_version("vllm") | ||
| except Exception: | ||
| vllm_version = "unknown" | ||
| raise RuntimeError( | ||
| "Unsloth: vLLM with version " | ||
| f"{vllm_version} does not yet support transformers>=5.0.0. " | ||
| "Please downgrade to transformers==4.57.3 via " | ||
| 'pip install --force-reinstall "transformers==4.57.3". ' | ||
| f"Original error: {e}" | ||
| ) from e | ||
| raise | ||
|
|
||
| try: | ||
| from vllm.sampling_params import GuidedDecodingParams | ||
| except ImportError: | ||
| except ImportError as e: | ||
| error_text = str(e) | ||
| if ( | ||
| "ALLOWED_LAYER_TYPES" in error_text | ||
| or "transformers.configuration_utils" in error_text | ||
| ): | ||
| try: | ||
| vllm_version = importlib_version("vllm") | ||
| except Exception: | ||
| vllm_version = "unknown" | ||
| raise RuntimeError( | ||
| "Unsloth: vLLM with version " | ||
| f"{vllm_version} does not yet support transformers>=5.0.0. " | ||
| "Please downgrade to transformers==4.57.3 via " | ||
| 'pip install --force-reinstall "transformers==4.57.3". ' | ||
| f"Original error: {e}" | ||
| ) from e | ||
| if not hasattr(vllm, "sampling_params") or not hasattr( | ||
| vllm.sampling_params, "StructuredOutputsParams" | ||
| ): | ||
| raise | ||
| vllm.sampling_params.GuidedDecodingParams = ( | ||
| vllm.sampling_params.StructuredOutputsParams | ||
| ) |
There was a problem hiding this comment.
There's significant code duplication in the except blocks for handling ImportError. This duplicated logic can be extracted into a helper function to improve code clarity and maintainability. Defining a local helper function within fix_vllm_guided_decoding_params would be a good approach to keep the scope limited.
def _handle_vllm_transformers_import_error(e):
error_text = str(e)
if (
"ALLOWED_LAYER_TYPES" in error_text
or "transformers.configuration_utils" in error_text
):
try:
vllm_version = importlib_version("vllm")
except Exception:
vllm_version = "unknown"
raise RuntimeError(
"Unsloth: vLLM with version "
f"{vllm_version} does not yet support transformers>=5.0.0. "
"Please downgrade to transformers==4.57.3 via "
'pip install --force-reinstall "transformers==4.57.3". '
f"Original error: {e}"
) from e
try:
import vllm
except ImportError as e:
_handle_vllm_transformers_import_error(e)
raise
try:
from vllm.sampling_params import GuidedDecodingParams
except ImportError as e:
_handle_vllm_transformers_import_error(e)
if not hasattr(vllm, "sampling_params") or not hasattr(
vllm.sampling_params, "StructuredOutputsParams"
):
raise
vllm.sampling_params.GuidedDecodingParams = (
vllm.sampling_params.StructuredOutputsParams
)* Handle Transformers 5 vLLM import errors * Deduplicate vLLM transformers mismatch handling --------- Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com>
Summary
Testing
from unsloth import version
print("unsloth", version)
PY