The Curse Of Recursion: Training On Generated Data Makes Models Forget
Large language models may "forget" unique elements when trained on generated data, leading to irreversible issues & loss of diversity.
This is a Plain English Papers summary of a research paper called The Curse of Recursion: Training on Generated Data Makes Models Forget. If you like these kinds of analysis, you should subscribe to the AImodels.fyi newsletter or follow me on Twitter. Overview The paper explores the potential impact of large language models (LLMs) like GPT-3 and ChatGPT on the future of online content and the models themselves. It introduces the concept of "Model Collapse," where using model-generated content in training can lead to irreversible issues in the resulting models. The paper aims to build...