Ryo Kamoi
鴨井 遼 (ja)
Ryo Kamoi is a Ph.D. student in Computer Science at Penn State University advised by Dr. Rui Zhang. He received his master’s degree in CS from UT Austin where he was advised by Dr. Greg Durrett, and received his bachelor’s degree in Statistics from Keio University where he was advised by Dr. Kei Kobayashi. He previously interned at Microsoft OAR and Amazon Alexa.
His research interests focus on the reasoning capabilities of LLMs, including research on self-refinement and self-improvement for detecting and mitigating errors during inference and training. His recent research work includes:
News
May 2025
I have started my internship at Microsoft OAR!
Process Reward Models (PRMs) provide step-level verification to LLM reasoning. However, PRM training data in prior work relies on human annotated labels or costly automatic methods, which generate noisy training labels. We propose FoVer, an efficient method to create accurate PRM training data from formal reasoning tasks using formal verification tools like Z3 and Isabelle. Experimental results demonstrate that FoVer improves PRM performance on widely used benchmarks for informal reasoning tasks.
We critically survey broad papers and discuss the conditions required for successful self-correction. Our survey indicates that (1) no prior work demonstrates successful self-correction with feedback from prompted LLMs, except for studies in tasks that are exceptionally suited for self-correction, (2) self-correction works well in tasks that can use reliable external feedback, and (3) large-scale fine-tuning enables self-correction.
ReaLMistake is a benchmark for evaluating error detection methods that detects errors in LLM responses. This benchmark includes errors made by GPT-4 and Llama 2 70B on three tasks (math word problem generation, fine-grained fact verification, and answerability classification). We observe that LLMs still cannot reliably detect mistakes made by LLMs. Strong LLMs like GPT-4 and Claude 3 detect errors made by LLMs at very low recall, and all LLM-based error detectors perform much worse than humans.
Selected publications. For the full list, please see Google Scholar or Semantic Scholar
VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information
PDF
Cite
Dataset
Website
Poster
(2025). COLM 2025.
(2025). COLM 2025.
AAAR-1.0: Assessing AI's Potential to Assist Research
PDF
Cite
(2025). ICML 2025. Best Paper Award at the 2nd AI4Research workshop @ AAAI 2025.
(2025). ICML 2025. Best Paper Award at the 2nd AI4Research workshop @ AAAI 2025.
Generalizable Process Reward Models via Formally Verified Training Data
PDF
Cite
Code
Dataset
Model
Website
(2025). arXiv preprint arXiv:2505.15960.
(2025). arXiv preprint arXiv:2505.15960.
When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs
PDF
Cite
Video
Slides
Paper List
(2024). TACL 2024. Oral at EMNLP 2024.
(2024). TACL 2024. Oral at EMNLP 2024.
Evaluating LLMs at Detecting Errors in LLM Responses
PDF
Cite
Code
Dataset
Poster
(2024). COLM 2024.
(2024). COLM 2024.
WiCE: Real-World Entailment for Claims in Wikipedia
PDF
Cite
Dataset
Slides
(2023). EMNLP 2023. Oral.
(2023). EMNLP 2023. Oral.
Shortcomings of Question Answering Based Factuality Frameworks for Error Localization
PDF
Cite
Dataset
Video
Poster
(2023). EACL 2023.
(2023). EACL 2023.
Why is the Mahalanobis Distance Effective for Anomaly Detection?
PDF
Cite
(2020). arXiv preprint arXiv:2003.00402.
(2020). arXiv preprint arXiv:2003.00402.
Education
Penn State University
— Ph.D. Student in Computer Science
PhD Advisor: Rui Zhang
University of Texas at Austin
— M.S. in Computer Science
Advisor: Greg Durrett, Mentor: Tanya Goyal
Keio University
— B.E. in Statistics
Advisor: Kei Kobayashi
Work Experience
Microsoft, Office of Applied Research
— Research Internship
Amazon, Alexa Team
— Applied Scientist Internship
Research on the quality evaluation of Alexa
Services
MASC-SLL 2025
— Organizer
NLP Colloquium JP (NLPコロキウム)
— Staff
Awards
Scholarship for alumni of Keio University to pursue degrees at overseas graduate schools
Graduation with highest honors - First place in the Department of Mathematics at Keio University
Media Mentions
Interview about our survey paper on LLM self-correction.
How to Pronounce My Name
- Preferred English Pronunciation:
- Native Pronunciation: