I've wondered if it would be possible (and beneficial) to make LLM's deterministic via a seed. Like how PRNG can specify a seed for repeatable deterministic pseudorandom numbers.
Theoretically, if you could specify a seed and the exact version of the model the output should always be the same. I wonder if this is possible with any open-weight models today?
---
On a more practical level, scripts (small programs) are deterministic so having the coding agent write (and possibly reuse) scripts might help.
Leftium
Anyone telling you they have tamed LLMs into producing 100% deterministic answers has either scoped the problem space so narrowly as to border on meaningless (e.g. "Is earth flat?" with a structured output schema of a single JSON boolean value), hasn't done robust statistical validation to actually confirm truly deterministic outputs, or both.
LLMs are fundamentally non-deterministic. Trying to use them to solve deterministic problem spaces is selecting the wrong tool for the job, and expecting them to be 100% reliable is the wrong mindset for working with them.
comments (2)
Theoretically, if you could specify a seed and the exact version of the model the output should always be the same. I wonder if this is possible with any open-weight models today?
---
On a more practical level, scripts (small programs) are deterministic so having the coding agent write (and possibly reuse) scripts might help.
Leftium
LLMs are fundamentally non-deterministic. Trying to use them to solve deterministic problem spaces is selecting the wrong tool for the job, and expecting them to be 100% reliable is the wrong mindset for working with them.
anonym29