Skip to content

rocmPackages.rocblas: rocblaslt error: Cannot read TensileLibrary_lazy_gfx1100.dat: No such file or directory #434280

@pshirshov

Description

@pshirshov

Nixpkgs version

  • Unstable (25.11)

Describe the bug

transformers library fails with

Device set to use cuda:0

rocblaslt error: Cannot read /nix/store/i3salxiqdvryyq67703ffx8lk701xhgi-hipblaslt-6.3.3/lib/hipblaslt/library/TensileLibrary_lazy_gfx1100.dat: No such file or directory

rocblaslt error: Could not load /nix/store/i3salxiqdvryyq67703ffx8lk701xhgi-hipblaslt-6.3.3/lib/hipblaslt/library/TensileLibrary_lazy_gfx1100.dat
Traceback (most recent call last):
  File "/home/pavel/work/safe/playq/sentiments/./test-env.py", line 11, in <module>
    result = pipe("I love using transformers on GPU!")
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/transformers/pipelines/text_classification.py", line 168, in __call__
    result = super().__call__(*inputs, **kwargs)
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/transformers/pipelines/base.py", line 1458, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/transformers/pipelines/base.py", line 1465, in run_single
    model_outputs = self.forward(model_inputs, **forward_params)
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/transformers/pipelines/base.py", line 1365, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/transformers/pipelines/text_classification.py", line 199, in _forward
    return self.model(**model_inputs)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 917, in forward
    distilbert_output = self.distilbert(
        input_ids=input_ids,
    ...<5 lines>...
        return_dict=return_dict,
    )
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward
    return self.transformer(
           ~~~~~~~~~~~~~~~~^
        x=embeddings,
        ^^^^^^^^^^^^^
    ...<4 lines>...
        return_dict=return_dict,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward
    layer_outputs = layer_module(
        hidden_state,
    ...<2 lines>...
        output_attentions,
    )
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/transformers/modeling_layers.py", line 94, in __call__
    return super().__call__(*args, **kwargs)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward
    sa_output = self.attention(
        query=x,
    ...<4 lines>...
        output_attentions=output_attentions,
    )
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward
    q = shape(self.q_lin(query))  # (bs, n_heads, q_length, dim_per_head)
              ~~~~~~~~~~^^^^^^^
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/nix/store/kxwcpngmyi1c9gn0kgf3krk6pylsvj0b-python3-3.13.5-env/lib/python3.13/site-packages/torch/nn/modules/linear.py", line 125, in forward
    return F.linear(input, self.weight, self.bias)
           ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: HIPBLAS_STATUS_INVALID_VALUE when calling `hipblasLtMatmulAlgoGetHeuristic( ltHandle, computeDesc.descriptor(), Adesc.descriptor(), Bdesc.descriptor(), Cdesc.descriptor(), Cdesc.descriptor(), preference.descriptor(), 1, &heuristicResult, &returnedResult)`

Steps to reproduce

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
          config = {
            allowUnfree = true;
            rocmSupport = true;
          };
        };

        pythonEnv = pkgs.python313.withPackages (ps: with ps; [
          torchWithRocm

          requests
          datasets

          ollama
          transformers
          pandas
          scikit-learn

          # unsloth

          # peft
          # trl
          # bitsandbytes

          # accelerate
          # xformers
        ]);
      in
      {
        devShells.default = pkgs.mkShell {
          buildInputs = with pkgs; [
            pythonEnv
            ollama
          ];
        };
      });
}
#!/usr/bin/env python3

import torch

print(f"Torch has GPU: {torch.cuda.is_available()}" )


from transformers import pipeline

pipe = pipeline("sentiment-analysis", device=0)
result = pipe("I love using transformers on GPU!")

print(f"Transformers see GPU: {result}")

Expected behaviour

The script should print something

Screenshots

No response

Relevant log output

Additional context

No response

System metadata

➜ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.15.9, NixOS, 25.11 (Xantusia), 25.11.20250814.c5e2e42`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.28.4`
 - nixpkgs: `/nix/store/fllg601b0yavarsnyfib571r3wxx9852-source`

Notify maintainers

@LunNova @GZGavinZhao


Note for maintainers: Please tag this issue in your pull request description. (i.e. Resolves #ISSUE.)

I assert that this issue is relevant for Nixpkgs

Is this issue important to you?

Add a 👍 reaction to issues you find important.

Metadata

Metadata

Labels

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions