Score: **0.422**
Writing report to /tmp/humaneval_mistralai_Mixtral-8x7B-Instruct-v0.1.html
{'pass@1': 0.42195121951219516, 'pass@1:std': 0.47524689096990586, 'pass@2': 0.4445121951219513, 'pass@2:std': 0.4832881593555032, 'pass@5': 0.4695121951219512, 'pass@5:std': 0.499069628161961, 'score:std': 0.47524689096990586, 'score': 0.42195121951219516}

[2025-12-18 17:51:32 TP0] Decode batch, #running-req: 2, #token: 3282, token usage: 0.00, cuda graph: True, gen throughput (token/s): 204.54, #queue-req: 0, 
[2025-12-18 17:51:32 TP0] Decode batch, #running-req: 1, #token: 1733, token usage: 0.00, cuda graph: True, gen throughput (token/s): 169.94, #queue-req: 0, 
[2025-12-18 17:51:32 TP0] Decode batch, #running-req: 1, #token: 1773, token usage: 0.00, cuda graph: True, gen throughput (token/s): 123.76, #queue-req: 0, 
100%|█████████████████████████████████████████████████████████████████████████████████████████| 5000/5000 [02:59<00:00, 27.92it/s]
Total latency: 179.171 s
Score: 0.633
Writing report to /tmp/mmlu_mistralai_Mixtral-8x7B-Instruct-v0.1.html
{'other': 0.7285223367697594, 'other:std': 0.4447218699336571, 'score:std': 0.4820967122891422, 'stem': 0.6374321880650995, 'stem:std': 0.48074150401607635, 'humanities': 0.5173237753882916, 'humanities:std': 0.49969979668426523, 'social_sciences': 0.7045454545454546, 'social_sciences:std': 0.45624681590647115, 'score': 0.6326}
Writing results to /tmp/mmlu_mistralai_Mixtral-8x7B-Instruct-v0.1.json
WARNING:root:GITHUB_STEP_SUMMARY environment variable not set
.
----------------------------------------------------------------------
Ran 3 tests in 406.697s

OK

Total latency: 848.003 s
Score: **0.421**
Writing report to /tmp/humaneval_mistralai_Mixtral-8x7B-Instruct-v0.1.html
{'pass@1': 0.42073170731707316, 'pass@1:std': 0.4867106630046756, 'pass@2': 0.4292682926829269, 'pass@2:std': 0.493491247081309, 'pass@5': 0.4329268292682927, 'pass@5:std': 0.49548076629471227, 'score:std': 0.4867106630046756, 'score': 0.42073170731707316}
Writing results to /tmp/humaneval_mistralai_Mixtral-8x7B-Instruct-v0.1.json
WARNING:root:GITHUB_STEP_SUMMARY environment variable not set

Total latency: 183.573 s
Score: 0.640
Writing report to /tmp/mmlu_mistralai_Mixtral-8x7B-Instruct-v0.1.html
{'other': 0.7285223367697594, 'other:std': 0.4447218699336571, 'score:std': 0.4801164858656699, 'stem': 0.6482820976491862, 'stem:std': 0.4775064601832704, 'humanities': 0.5286738351254481, 'humanities:std': 0.4991771340708613, 'social_sciences': 0.7083333333333334, 'social_sciences:std': 0.45452967144315476, 'score': 0.6396}
Writing results to /tmp/mmlu_mistralai_Mixtral-8x7B-Instruct-v0.1.json
WARNING:root:GITHUB_STEP_SUMMARY environment variable not set
.
----------------------------------------------------------------------
Ran 3 tests in 1096.271s

OK