## blog entry
Stop Asking the LLM to Remember the Rules
A binary decision tree lets code ask an LLM one narrow question at a time, making large-scale classification more reliable and mistakes easier to inspect.
The problem
More instructions didn’t make my LLM reliable. Letting code pick one question at a time made its mistakes easier to spot.
We needed to classify over 18k base-word + prefix/suffix records.
Ex: sunny + er can form a valid word, but “sunnyer” should be “sunnier”.
And then do that 18 thousand times!
Of course, asking an LLM the naive question---does this combination form a valid word, and if so, what’s the correct spelling?---is unreliable. It’s actually wildly unreliable.
So then the next stage is to give the LLM a rubric to help it reason about the correctness of the words. Better results but still unreliable.
Note: if you’re thinking, why not use a dictionary? Cost was a constraint for us, so we were OK with relying on the LLM’s training data.
Introducing Binary Decision Trees
Then I had an insight---why not let the LLM focus on one thing at a time? Take the rubric and turn it into a binary decision tree that guides the LLM through the data set.
The complex rubric collapsed into a few narrow questions, with many branches being completely deterministic!
Each word part follows only the path it needs---at most a few questions to reach a decision, plus a possible spelling follow-up. An answer choice picks the next branch.
If it’s not clear, the LLM gets the shared instructions and current question, not the entire decision tree. So it’s context per question is super tiny.
We ask each decision question five times, take a majority vote, and save disagreements for review.
Why this works well with LLMs
The LLM doesn’t have to remember the entire process or decide what to do next. It does the task it’s better suited to: judge one specific case against one specific question.
Code handles the rules about which question comes next.
It does use more tokens than the previous iterations, so we gained more accuracy for higher cost. We also get a trail we can inspect. It’s not hallucination-proof——five votes can still be wrong——but a bad label no longer has to be a mystery.