Overview

Stacks on feat/benchmark-multilingual (#1483). Adds 6 translated dataset languages (DE/FR/HI/IT/KO/RU), --output-dir for per-language CSV output, labels.ko.jsonl to fix the cross-language scoring bug on memory_extraction, a translate_datasets.py Ollama helper, new community/local candidate tiers, and a 210-run multilingual benchmark report.

The methodology fix (per-language labels) is well-motivated and the report is genuinely useful. Most concerns are localized.

Issues

High — --output single-file mode opens N handles to the same path

In orchestrator.py:

for lang in languages:
    p = _csv_path(lang)
    p.parent.mkdir(parents=True, exist_ok=True)
    if p not in seen_paths:
        fh = open(p, "w", ...)
        ...
        seen_paths.add(p)
    else:
        fh = open(p, "a", ...)   # ← second handle to the SAME file
    lang_files[lang] = (fh, w, p)

When --output one.csv is combined with multiple languages, _csv_path() returns the same Path for every language. The first iteration opens a "w" handle; subsequent iterations open additional "a" handles to the same file. All handles stay open simultaneously, each with its own buffer and offset. Interleaved writer.writerow(...) + fh.flush() from different handles can overwrite each other (the "w" handle's position advances independently of "a" appends). gemini-code-assist flagged this — it's a real correctness bug, not just style.

Fix: detect single-file mode once and share one (fh, writer) pair across all languages:

if args.output:
    fh = open(args.output, "w", newline="", encoding="utf-8")
    w = csv.DictWriter(fh, fieldnames=CSV_COLUMNS); w.writeheader(); fh.flush()
    lang_files = {lang: (fh, w, args.output) for lang in languages}
else:
    # per-language files as today (no path collisions possible)

The test plan item "Existing EN-only runs produce identical output" passes only because a single-language run never hits the collision — a multi-language --output run would.

Medium — Untranslated samples in dataset.hi.jsonl

ent_020, ent_025, ent_032, ent_038 in datasets/entity_extraction/dataset.hi.jsonl are still entirely English, and Copilot flagged rc_026, rc_028 in the Hindi room-classification set. These inflate the Hindi numbers (the model isn't actually processing Hindi for those rows). Worth a quick sweep — grep -L '[ऀ-ॿ]'-style — across all six new languages before the report's numbers get cited downstream. The Russian set is worth double-checking for the same reason (the report calls RU the lowest scorer).

Medium — labels.ko.jsonl doesn't honor the proper-noun rule

The translation prompt and the input datasets are explicit that proper nouns stay English. The Korean labels don't follow the same rule:

• labels.ko.jsonl:15 replaces Doreth with 독스 (transliteration) and translates "heritage apple" as 유전자 사과 ("genetic apple" — a mistranslation; should be 재래종/고대 사과).
• Other lines transliterate names inconsistently: 세라 for Saela, 이보라 for Ivora, 렌 소란케 for Ren Solanke, 브램블 for Bramble, 폴 크리사트 for Pol Krisat, while keeping Solas/Ivora/Tartine Lab in English elsewhere.

Cosine scoring with nomic-embed-text is somewhat robust to this, but inconsistency in the ground truth still adds noise to exactly the metric this file was added to fix. Worth one pass to normalize names back to English before more contributors copy this format for DE/FR/IT/HI/RU labels.

Medium — runner.py label-resolution one-liner

labels_path = task_dir / (f"labels.{language}.jsonl" if language != "en" and (task_dir / f"labels.{language}.jsonl").exists() else "labels.jsonl")

Two readability nits and one real concern:

1. .exists() is computed inside a conditional expression that builds the path string twice. Split it:

   candidate = task_dir / f"labels.{language}.jsonl"
   labels_path = candidate if language != "en" and candidate.exists() else task_dir / "labels.jsonl"

2. Silent fallback. When DE/FR/HI/IT/RU runs hit memory_extraction, they fall back to labels.jsonl (English ground truth) without any log line. That's exactly the bug the PR was written to address, and the report (§ Memory Extraction) acknowledges the resulting score collapse. At minimum, print a one-line info: language=de labels=labels.jsonl (no labels.de.jsonl found) so future readers don't draw "the model is bad at German" from numbers that are actually methodology-bound.

Low

• import socket inside main() in orchestrator — move to module top (gemini's comment, PEP 8).
• Unused out_path in fh, writer, out_path = lang_files[language].
• l as loop variable in langs = [l.strip() for l in args.languages.split(",") ...] in translate_datasets.py — ruff E741 (ambiguous with 1). Use lang or code.
• _translate_one implicit-None return path is hard to read on inspection. The function always returns in practice (attempt 0/1 raise → continue; attempt 2 always returns either the success or the fallback), but a for/else or explicit return at the bottom would make that obvious.
• Report file is bilingual PT/EN — the analytical paragraphs in reports/2026-05-13-multilingual.md are in Portuguese while tables and headers are English. Existing benchmark reports in the repo are English-only; worth aligning for searchability.
• _SYSTEM_PROMPT hardcodes proper-noun lists. Reliable, but if datasets grow, the list will drift. A --proper-nouns-file argument (one name per line) would future-proof it without complicating the common case.

Style / convention

• Translation script timeouts (30s connect / 120s read) and temperature=0.1 are sensible. Streaming approach is correct for cloud models with high TTFT.
• --output-dir schema (<dir>/<lang>/YYYY-MM-DD-<host>.csv) matches existing single-file naming. Good.
• The community / local tier additions to candidates.yaml are well-documented with notes.

Security / privacy

translate_datasets.py defaults to kimi-k2.6:cloud. Cloud Ollama tags send dataset prose to a remote endpoint. The benchmark inputs in this PR are synthetic, so it's fine here, but the script's docstring should call out "do not run this on real user data" — someone will repurpose it on their own palace eventually, and that would violate the local-first principle in CLAUDE.md.

Test coverage

No new unit tests, which is reasonable for benchmark-tooling changes. The PR's test plan is a runbook rather than automated tests. Worth at least one regression: a 1-line check that _csv_path("en") returns the right path under both --output and --output-dir — that would have caught the file-handle bug above.

Summary

Solid contribution; the methodology insight (extract-in-input-language → need translated labels) is the kind of finding that justifies the whole PR. Blockers: the orchestrator single-file-mode bug and the untranslated Hindi samples. The rest can be follow-ups, including extending labels.{lang}.jsonl to the remaining five languages (already called out as future work).

Review fixes pushed ✓

All feedback from your review has been addressed:

orchestrator.py

• Fixed critical file-handle bug in --output mode (was opening multiple handles to same file with independent buffers)
• Now shares single (fh, writer) pair across all languages in single-file mode
• Added explicit deduplication in finally block to prevent double-close

runner.py

• Refactored label resolution from one-liner to readable 3-line version
• Added explicit fallback logging when no language-specific labels exist
• Log: "info: language=de labels=labels.jsonl (no labels.de.jsonl found — scoring against English ground truth)"

datasets

• Fixed 19 untranslated samples across 4 languages (DE, HI, IT, KO)
• All samples re-translated via Ollama with proper noun preservation rules

labels.ko.jsonl

• Restored all transliterated proper nouns to English (30+ entries)
• Fixed mistranslation: 유전자 사과 (genetic apple) → 재래종 사과 (heirloom apple)

Code style

• Moved import socket to module top (PEP 8)
• Fixed ruff E741 (ambiguous variable name)
• Made _translate_one return path explicit

reports

• Converted 2026-05-13-multilingual.md to English-only (was Portuguese)
• Updated methodology note explaining Korean labels fix

Ready for next review pass.