Skip to content

[Models] Llama2 fix#332

Closed
Aalanli wants to merge 8 commits intohidet-org:mainfrom
Aalanli:llama2
Closed

[Models] Llama2 fix#332
Aalanli wants to merge 8 commits intohidet-org:mainfrom
Aalanli:llama2

Conversation

@Aalanli
Copy link
Copy Markdown
Contributor

@Aalanli Aalanli commented Jul 27, 2023

Fix llama2 and add test for num_heads != num_key_value_heads.

@Aalanli Aalanli closed this Jul 27, 2023
@Aalanli Aalanli deleted the llama2 branch September 27, 2023 18:10
vadiklyutiy pushed a commit that referenced this pull request Jul 22, 2024
…332)

[Edit: The issue was encountered while attempting to compile the model
`yolov3`]

Currently the
[`setitem`](https://github.com/CentML/hidet/blob/566f0fe55f441326c3034b7eed44b3fa0b03f38d/python/hidet/graph/frontend/torch/register_functions.py#L280)
function in Hidet will fail on two special scenarios when `setvalue` is
a tensor:

1. When `setvalue` and `x` are of different dtypes, currently there will
be an error that looks like:
> RuntimeError: If-then-else operand 1 and 2 have different types
(hidet.float16 vs hidet.float32) ((((v < 0) || (2 <= v)) ? false :
(((v_1 < 0) || (3 <= v_1)) ? false : (((v_2 < 0) || (3 <= v_2)) ? false
: true))) ? setvalue[v_2, v_1, v] : data[v_2, v_1, v]), occurred when
interpreting operator.setitem with
>   setitem(tensor(...), (Ellipsis, slice(None, 2, None)), tensor(...))

Whereas in PyTorch `setvalue` appears to be casted to the same datatype
as `x` if possible.

2. When `setvalue` and `x` are on different devices, currently this will
result in an error:

> RuntimeError: All inputs of an operator must be on the same device,
occurred when interpreting operator.setitem with
>   setitem(tensor(...), (Ellipsis, slice(None, 2, None)), tensor(...))

Whereas in PyTorch the `setvalue` is moved to the same device as `x`.
vadiklyutiy pushed a commit that referenced this pull request Jul 23, 2024
…332)

[Edit: The issue was encountered while attempting to compile the model
`yolov3`]

Currently the
[`setitem`](https://github.com/CentML/hidet/blob/566f0fe55f441326c3034b7eed44b3fa0b03f38d/python/hidet/graph/frontend/torch/register_functions.py#L280)
function in Hidet will fail on two special scenarios when `setvalue` is
a tensor:

1. When `setvalue` and `x` are of different dtypes, currently there will
be an error that looks like:
> RuntimeError: If-then-else operand 1 and 2 have different types
(hidet.float16 vs hidet.float32) ((((v < 0) || (2 <= v)) ? false :
(((v_1 < 0) || (3 <= v_1)) ? false : (((v_2 < 0) || (3 <= v_2)) ? false
: true))) ? setvalue[v_2, v_1, v] : data[v_2, v_1, v]), occurred when
interpreting operator.setitem with
>   setitem(tensor(...), (Ellipsis, slice(None, 2, None)), tensor(...))

Whereas in PyTorch `setvalue` appears to be casted to the same datatype
as `x` if possible.

2. When `setvalue` and `x` are on different devices, currently this will
result in an error:

> RuntimeError: All inputs of an operator must be on the same device,
occurred when interpreting operator.setitem with
>   setitem(tensor(...), (Ellipsis, slice(None, 2, None)), tensor(...))

Whereas in PyTorch the `setvalue` is moved to the same device as `x`.
vadiklyutiy pushed a commit that referenced this pull request Dec 26, 2024
…332)

[Edit: The issue was encountered while attempting to compile the model
`yolov3`]

Currently the
[`setitem`](https://github.com/CentML/hidet/blob/566f0fe55f441326c3034b7eed44b3fa0b03f38d/python/hidet/graph/frontend/torch/register_functions.py#L280)
function in Hidet will fail on two special scenarios when `setvalue` is
a tensor:

1. When `setvalue` and `x` are of different dtypes, currently there will
be an error that looks like:
> RuntimeError: If-then-else operand 1 and 2 have different types
(hidet.float16 vs hidet.float32) ((((v < 0) || (2 <= v)) ? false :
(((v_1 < 0) || (3 <= v_1)) ? false : (((v_2 < 0) || (3 <= v_2)) ? false
: true))) ? setvalue[v_2, v_1, v] : data[v_2, v_1, v]), occurred when
interpreting operator.setitem with
>   setitem(tensor(...), (Ellipsis, slice(None, 2, None)), tensor(...))

Whereas in PyTorch `setvalue` appears to be casted to the same datatype
as `x` if possible.

2. When `setvalue` and `x` are on different devices, currently this will
result in an error:

> RuntimeError: All inputs of an operator must be on the same device,
occurred when interpreting operator.setitem with
>   setitem(tensor(...), (Ellipsis, slice(None, 2, None)), tensor(...))

Whereas in PyTorch the `setvalue` is moved to the same device as `x`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant