LLMs And JSON Syntax: Reliably Generating Code With Json_mode
LLMs generate responses token by token, probabilistically. Influence these probabilities to discourage incorrect tokens & get code snippets in JSON, CSV or Python with correct syntax using "json_mode", structured output or function calling.
How does an LLM reliably produce responses that strictly follow JSON syntax when using features like "json_mode" or "function calling"? These options actually provide an answer to the question: "How can we get an LLM to generate responses exactly the way we want?" You're probably familiar with the fact that LLMs generate responses token by token, step-by-step. But what's not commonly known, especially outside technical circles, is that each token is generated probabilistically. Then, what if we could influence or adjust the probabilities, discouraging tokens that don't match our desired format...