Skip to content

feat: testbed#39

Merged
McPatate merged 59 commits intomainfrom
feat/test_framework
Nov 6, 2023
Merged

feat: testbed#39
McPatate merged 59 commits intomainfrom
feat/test_framework

Conversation

@McPatate
Copy link
Copy Markdown
Member

testbed will be evaluating completion quality

@McPatate McPatate force-pushed the feat/test_framework branch from b66825b to 54da8aa Compare October 24, 2023 16:17
@github-actions
Copy link
Copy Markdown

github-actions bot commented Oct 30, 2023

Repository name Source type Average hole completion time (s) Pass percentage
simple local 3.11255 93%
gcanti/io-ts github 109.3835 0%
lancedb/lance github 96.7374 69.46386%
lancedb/lancedb github 78.153206 90%
mmaitre314/picklescan github 4.4968 100%
smol-rs/async-executor github 10.5044 50%
colinhacks/zod github 52.4536 80%
helix-editor/helix github 86.479706 70%
encode/starlette github 37.8284 89.73298%
tkaitchuck/constrandom github 2.444 10%
tiangolo/fastapi github 88.779205 80%
huggingface/huggingface_hub github 20.571901 100%
jaemk/cached github 34.3235 100%
Total -- 29.694595 80.41803%

Note: The "hole completion time" represents the full process of:

  • replacing the code from the file with a completion from the model
  • building the project
  • running the tests

@huggingface huggingface deleted a comment from github-actions bot Oct 30, 2023
@huggingface huggingface deleted a comment from github-actions bot Oct 30, 2023
@huggingface huggingface deleted a comment from github-actions bot Oct 30, 2023
@McPatate McPatate force-pushed the feat/test_framework branch from b675e52 to 1198992 Compare November 3, 2023 14:20
@McPatate McPatate force-pushed the feat/test_framework branch from 1644939 to 61ad1a4 Compare November 3, 2023 19:18
@McPatate McPatate merged commit c7affd0 into main Nov 6, 2023
@McPatate McPatate deleted the feat/test_framework branch November 6, 2023 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants