In recent discussions about language model construction, particularly with the rise of large language models (LLMs), there has been growing concern over the phenomenon of model collapse—where LLMs trained on synthetic data begin to diverge from real-world language. A recent article in Communications of the ACM, titled “The Collapse of GPT” (Savage, 2025), explores this issue and highlights its relevance to the future of AI systems. This article brought to mind work I co-authored with my colleague Peter Wyard in the late 1990s, titled “An Internet Agent for Language Model Construction.” Peter was a member of the BT Labs NLP research group, and I was working there on a Research Fellowship. At the time, we were grappling with similar challenges—how to build domain-specific language models for speech recognition systems when collecting enough in-domain data was both costly and time-consuming.
(more…)
Information Interaction 