LLMs Show Promise In PBE But Struggle With New Problem Types
LLMs show promise in PBE tasks but struggle with new problem types, fine-tuning improves performance but out-of-distribution generalization remains a challenge.
This is a Plain English Papers summary of a research paper called LLMs Show Promise in Programming by Example, But Struggle with New Problem Types. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter. Overview This paper investigates the ability of Large Language Models (LLMs) to solve Programming-by-Examples (PBE) tasks. PBE aims to generate algorithms from input-output examples, which is important both practically and theoretically. The researchers experiment on classic domains like lists and strings, as well as an uncommon graphics programming...