[Iluvatar] llama2 7b case #343
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* 【kunlunxin】Longformer update config * update memory used for kunlunxin and solve file conflict. * update 2x8 memory used. * update kunlunxin docs. * update kunlunxin 1x1 memory docs. * Change to a larger batch size. * Remove empty line in test_config.py * Update README. * Update test_conf.py, add comma --------- Co-authored-by: zhangsanfeng2022 <zhangsanfeng2022@github.com> Co-authored-by: root <root@szzj-isa-ai-chip0.szzj.baidu.com> Co-authored-by: Zhou Yu <zycosmos@gmail.com>
* update * update * update * update * update * update --------- Co-authored-by: 魏杰 <jie.wei@iluvatar.com>
…agPerf into iluvatarllama27BNew merge .
yuzhou03
reviewed
Nov 29, 2023
yuzhou03
reviewed
Nov 29, 2023
yuzhou03
reviewed
Nov 29, 2023
yuzhou03
reviewed
Nov 29, 2023
yuzhou03
reviewed
Nov 29, 2023
shh2000
reviewed
Nov 29, 2023
|
|
||
| - ##### 软件环境 | ||
| - OS版本：Ubuntu 20.04 | ||
| - Docker 版本：20.10.18 |
Collaborator
There was a problem hiding this comment.
20.10.21，请参考flagperf自动打印出来的sys_info.log
shh2000
reviewed
Nov 29, 2023
| seqlength = 2048 | ||
| batchsize = 4 | ||
| datafilename = "openwebtext_llama2_100M.npy" | ||
| theoryflops = 312000000000000.0 |
Collaborator
There was a problem hiding this comment.
此值目前填的是312Tflops，为A100的值。请修改为天数bf16算力。此值影响MFU计算，不影响吞吐量计算
shh2000
reviewed
Nov 30, 2023
shh2000
approved these changes
Dec 15, 2023
upvenly
approved these changes
Dec 15, 2023
shh2000
reviewed
Dec 15, 2023
| epochs = 1 | ||
| flashattn = True | ||
| gradient_checkpointing = True | ||
| use_cache = false No newline at end of file |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.