Skip to content

model: NightOwl-CodeEmbedding#4791

Merged
Samoed merged 6 commits into
embeddings-benchmark:mainfrom
Shun0212:add-nightowl-model
Jun 10, 2026
Merged

model: NightOwl-CodeEmbedding#4791
Samoed merged 6 commits into
embeddings-benchmark:mainfrom
Shun0212:add-nightowl-model

Conversation

@Shun0212

Copy link
Copy Markdown
Contributor

Summary

Adds Shuu12121/NightOwl-CodeEmbedding to the MTEB model registry.

NightOwl-CodeEmbedding is a Sentence Transformers-compatible ModernBERT model specialized for code retrieval. It uses CLS pooling, cosine similarity, and does not require query or document prefixes.

Evaluation

The model was evaluated on 12 representative code-retrieval tasks using MTEB. The macro-average NDCG@10 was 0.70240.

Task Split NDCG@10
AppsRetrieval test 0.36361
COIRCodeSearchNetRetrieval test 0.84063
CodeEditSearchRetrieval train 0.74720
CodeFeedbackMT test 0.76277
CodeFeedbackST test 0.85137
CodeSearchNetCCRetrieval test 0.91646
CodeSearchNetRetrieval test 0.89187
CodeTransOceanContest test 0.74091
CodeTransOceanDL test 0.35802
CosQA test 0.41207
StackOverflowQA test 0.86031
SyntheticText2SQL test 0.68354
Macro average 0.70240

There is currently no original paper associated with this model. Detailed benchmark results are available in the model card.

For CodeEditSearch-like training data, I used a custom dataset derived from bigcode/commitpackft. Rows overlapping with cassanof/CodeEditSearch were excluded using content-, diff-, commit-, and repo/commit-based hashes. If CodeEditSearch should still be listed in the model metadata as related training data, I would be happy to add it.

Checklist

  • I have filled out the ModelMeta object to the extent possible

  • I have ensured that my model can be loaded using

    • mteb.get_model(model_name, revision)
    • mteb.get_model_meta(model_name, revision)
  • I have tested the implementation works on a representative set of tasks

  • The model is public, i.e., is available either as an API or the weights are publicly available to download

  • I reproduced results from the original paper (if applicable) on at least one benchmark, and I am including the results in the PR description

    • Not applicable: there is currently no original paper associated with this model.

@Samoed Samoed added the new model Questions related to adding a new model to the benchmark label Jun 10, 2026
@Samoed Samoed enabled auto-merge (squash) June 10, 2026 05:54
@Samoed Samoed merged commit b691769 into embeddings-benchmark:main Jun 10, 2026
18 of 19 checks passed
@Shun0212

Copy link
Copy Markdown
Contributor Author

Thank you for merging this PR!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new model Questions related to adding a new model to the benchmark

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants