Skip to content

[Model Support] Add support for wav2vec#303

Merged
yaoyaoding merged 2 commits intohidet-org:mainfrom
yaoyaoding:wav2vec
Jul 5, 2023
Merged

[Model Support] Add support for wav2vec#303
yaoyaoding merged 2 commits intohidet-org:mainfrom
yaoyaoding:wav2vec

Conversation

@yaoyaoding
Copy link
Copy Markdown
Member

@yaoyaoding yaoyaoding commented Jul 4, 2023

Major:

  1. Enhance the hidet dynamo backend to support the pytorch model that used weight_norm.
  2. Add resolve rule for conv1d.

The example code that use hidet could be found at https://github.com/egorsmkv/wav2vec2-hidet.

Currently, hidet is still slower than pytorch (hidet 30ms vs. pytorch 25ms on RTX 4090). Will optimize these models when we have more hands.

@yaoyaoding yaoyaoding merged commit 6ebb6ec into hidet-org:main Jul 5, 2023
@yaoyaoding yaoyaoding deleted the wav2vec branch July 5, 2023 17:28
@egorsmkv
Copy link
Copy Markdown

egorsmkv commented Sep 28, 2023

Nice, thanks

@egorsmkv
Copy link
Copy Markdown

egorsmkv commented Oct 4, 2023

@yaoyaoding hi

My little feedback on this issue - https://github.com/egorsmkv/wav2vec2-hidet/blob/main/logs_hidet_0.3.0_and_python3.11.txt

@yaoyaoding
Copy link
Copy Markdown
Member Author

yaoyaoding commented Oct 4, 2023

Hi @Aalanli, could you take a look at this? Thanks!

@egorsmkv
Copy link
Copy Markdown

egorsmkv commented Jan 26, 2024

Hello! Any update on it? @Aalanli

@egorsmkv
Copy link
Copy Markdown

egorsmkv commented Jan 26, 2024

Just tried hidet with wav2vec2-bert model recently released. Also I have some problems.

I made a colab to share what's going wrong: https://colab.research.google.com/drive/1qsw0Tu9ENS_q3ao0YUbh4CDTyWqUxLWq?usp=sharing

In short:

BackendCompilerFailed: backend='hidet' raised:
NotImplementedError: The following modules/functions are not supported by hidet yet:
  torch.rand
  torch.nn.GLU
  torch.softmax

@Aalanli
Copy link
Copy Markdown
Contributor

Aalanli commented Jan 26, 2024

Hi @egorsmkv, sorry for the long delay. It turns out that implementing rand would require some invasive changes to the runtime, which we held off doing until our AMD support is ready.
As a potential work-around, one can pass any random state as an input to the model, kind of like what is done in Jax.
The other functions should be a quick fix.

@egorsmkv
Copy link
Copy Markdown

egorsmkv commented Jul 30, 2024

An update with hidet 0.4.1:

https://github.com/egorsmkv/wav2vec2-hidet-0.4.0/blob/master/logs_hidet_0.4.1_and_python3.12.txt

In short:

NotImplementedError: The following operators are not supported or mapped by hidet yet:
  torch.nn.GLU

@vadiklyutiy
Copy link
Copy Markdown
Collaborator

vadiklyutiy commented Jul 30, 2024 •
edited
Loading

An update with hidet 0.4.1:

https://github.com/egorsmkv/wav2vec2-hidet-0.4.0/blob/master/logs_hidet_0.4.1_and_python3.12.txt

In short:

NotImplementedError: The following operators are not supported or mapped by hidet yet:
  torch.nn.GLU

How to reproduce?

PS actually it should be supported.

@egorsmkv
Copy link
Copy Markdown

egorsmkv commented Jul 31, 2024

An update with hidet 0.4.1:
https://github.com/egorsmkv/wav2vec2-hidet-0.4.0/blob/master/logs_hidet_0.4.1_and_python3.12.txt
In short:

NotImplementedError: The following operators are not supported or mapped by hidet yet:
  torch.nn.GLU

How to reproduce?

PS actually it should be supported.

https://github.com/egorsmkv/wav2vec2-hidet-0.4.0?tab=readme-ov-file#install here is the instruction to reproduce that issue.

@vadiklyutiy
Copy link
Copy Markdown
Collaborator

vadiklyutiy commented Aug 2, 2024

@egorsmkv
Sorry, it was my foul. torch.nn.GLU is not supported yet.

@egorsmkv
Copy link
Copy Markdown

egorsmkv commented Aug 2, 2024

@egorsmkv
Sorry, it was my foul. torch.nn.GLU is not supported yet.

Oh, okay. I'm waiting for final support of it. Luck to the team :) You do awesome work 👏

vadiklyutiy pushed a commit that referenced this pull request Dec 19, 2024
@xiaocenxiaocen @vadiklyutiy
Hexcute base branch (All related PRs will be merged into this base PR…
62057b8 
…. ) (#294)

[Ir][Primitives] add vectorized conversion instructions
[Ir][CuTe] add reduce primitives in cute (#295)
[Ir][CuTe] add mma primitives (#296)
[Ir][CuTe] add other primitives in cute (#297)
[Transforms][CuTe] add instruction selection pass (#298)
[Transforms][CuTe] add resolve bank conflict pass (#299)
[Transforms][CuTe] add resolve auto keywords pass (#300)
[Transforms][CuTe] add shared memory allocation pass (#301)
[Transforms][CuTe] add vectorize elementwise operation pass (#302)
[Transforms][CuTe] add analysis pass (#303)
[Transforms][CuTe] add canonicalization pass (#304)
[Transforms][CuTe] add deadcode elimination pass (#305)
[Transforms][CuTe] refactor cute lowering pass (#306)
[Graph][Ops] matmul cute (#307)
[Ir] cute miscs (#308)
[Tests] cute tests (#309)
[Chore] fix ci (#313)
---------

Co-authored-by: xiaocenxiaocen <xiao.zhang@centml.ai>
vadiklyutiy pushed a commit that referenced this pull request Dec 20, 2024
@xiaocenxiaocen @vadiklyutiy
Hexcute base branch (All related PRs will be merged into this base PR…
b1fdf17 
…. ) (#294)

[Ir][Primitives] add vectorized conversion instructions
[Ir][CuTe] add reduce primitives in cute (#295)
[Ir][CuTe] add mma primitives (#296)
[Ir][CuTe] add other primitives in cute (#297)
[Transforms][CuTe] add instruction selection pass (#298)
[Transforms][CuTe] add resolve bank conflict pass (#299)
[Transforms][CuTe] add resolve auto keywords pass (#300)
[Transforms][CuTe] add shared memory allocation pass (#301)
[Transforms][CuTe] add vectorize elementwise operation pass (#302)
[Transforms][CuTe] add analysis pass (#303)
[Transforms][CuTe] add canonicalization pass (#304)
[Transforms][CuTe] add deadcode elimination pass (#305)
[Transforms][CuTe] refactor cute lowering pass (#306)
[Graph][Ops] matmul cute (#307)
[Ir] cute miscs (#308)
[Tests] cute tests (#309)
[Chore] fix ci (#313)
---------

Co-authored-by: xiaocenxiaocen <xiao.zhang@centml.ai>
vadiklyutiy pushed a commit that referenced this pull request Dec 26, 2024
@xiaocenxiaocen
Hexcute base branch (All related PRs will be merged into this base PR…
49e5740 
…. ) (#294)

[Ir][Primitives] add vectorized conversion instructions
[Ir][CuTe] add reduce primitives in cute (#295)
[Ir][CuTe] add mma primitives (#296)
[Ir][CuTe] add other primitives in cute (#297)
[Transforms][CuTe] add instruction selection pass (#298)
[Transforms][CuTe] add resolve bank conflict pass (#299)
[Transforms][CuTe] add resolve auto keywords pass (#300)
[Transforms][CuTe] add shared memory allocation pass (#301)
[Transforms][CuTe] add vectorize elementwise operation pass (#302)
[Transforms][CuTe] add analysis pass (#303)
[Transforms][CuTe] add canonicalization pass (#304)
[Transforms][CuTe] add deadcode elimination pass (#305)
[Transforms][CuTe] refactor cute lowering pass (#306)
[Graph][Ops] matmul cute (#307)
[Ir] cute miscs (#308)
[Tests] cute tests (#309)
[Chore] fix ci (#313)
---------

Co-authored-by: xiaocenxiaocen <xiao.zhang@centml.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@yaoyaoding @egorsmkv @Aalanli @vadiklyutiy