By Anna St. Clair and Alayna Smith
This project investigates whether GPT-4o's response quality varies across typologically distinct languages. The same multiple-choice questions are posed in seven languages representing different grammatical word orders (SVO, SOV, VSO), and accuracy is compared across them.
| Language | Word Order |
|---|---|
| English | SVO |
| French | SVO |
| Japanese | SOV |
| Korean | SOV |
| Bengali | SOV |
| Arabic | VSO |
| Swahili | SVO |
Questions are drawn from the openai/MMMLU dataset on Hugging Face — a multilingual version of MMLU where each language config contains the same questions in the same order, enabling fair cross-language comparison.
| Section | Description |
|---|---|
| 0 — Preamble & Imports | Install and import all dependencies |
| 1 — Configuration | Experiment parameters (languages, model, sample size) |
| 2 — Dataset Loading | Load and verify cross-language alignment |
| 3 — Tokenization Utilities | Compute token fertility and subword fragmentation metrics |
| 4 — Prompt Runner | Send prompts to GPT-4o and collect results |
| 4b — Failure Mode Analysis | Inspect questions where the model failed in one language but not another |
| 5 — Scoring & Confidence Intervals | Per-language accuracy with 95% CIs |
| 6 — Visualizations | Six charts saved to figures/ |
results/— per-language CSVs and a combinedall_results.csvfigures/— six PNG charts including accuracy by language, token fertility, and correlation plots
- Python 3.10+
- OpenAI API key (stored as
OPENAI_API_KEYin Colab Secrets) - Dependencies are auto-installed in Section 0:
openai,datasets,tiktoken,pandas,numpy,scipy,matplotlib,seaborn
- Open the notebook in Google Colab
- Add your
OPENAI_API_KEYto Colab Secrets - Run all cells in order