Imagine handing a new employee a 150-line operations manual and asking them to make a high-stakes decision on their first day. They read the whole thing, hold it in mind, and try to apply it to a specific case. Sometimes they get it right. Sometimes, for reasons that are hard to pin down, they do not.
This is roughly what happens when we give an AI agent a flat reference document and ask it to work through a multi-step decision.
The Setup
The agent reads everything at once. All the rules load into its context simultaneously. When the time comes to reason through a specific case, the model must weigh the entire document against the details of the situation in front of it. Some rules will receive more attention than others. Ambiguous passages will be interpreted inconsistently. Runs that should produce identical outputs on identical inputs may diverge.
To test this empirically, I ran a controlled experiment using a simplified version of the Emergency Severity Index (ESI) clinical triage as the domain. ESI is a five-level system used in emergency departments to classify patient acuity and guide provider response time. It has a well-defined decision algorithm, explicit threshold values, and several documented edge cases where novice triagers commonly err. It is, in other words, exactly the kind of structured decision-making framework that organizations routinely try to operationalize with AI.
The question I wanted to answer was simple: does the delivery mechanism for knowledge affect decision accuracy and consistency? Specifically, does serving that knowledge through a structured decision tree produce better outcomes than asking an agent to read a flat markdown document?
The short answer is yes, and in some cases, dramatically. Here is what the experiment looked like.
The longer version of the answer is that the flat document approach actually works better than I thought in many situations. The models are getting good enough to hold a lot of context and perform complex reasoning. Important gaps still remain however. I explore some of them in this write up.
The Decision Framework
The ESI algorithm works like this. At triage, a nurse works through four questions in order, stopping at the first that resolves:
- Does the patient need an immediate life-saving intervention? If yes, ESI-1.
- Is this a high-risk situation, or is any vital sign in the danger zone? If yes, ESI-2.
- How many distinct resource categories will this patient require (labs, imaging, IV medications, specialist consultation, complex procedure)? Two or more is ESI-3, exactly one is ESI-4, zero is ESI-5.
The algorithm is not complicated in principle. A trained nurse can walk through it in under two minutes. But the document that describes it contains several nuanced qualifiers that create consistent failure modes for agents reading it cold.
Three of those qualifiers are particularly relevant to this experiment.
The anticoagulation inference. The reference document lists "isolated extremity injury requiring imaging only" as a canonical ESI-4 presentation. What the document does not state explicitly is that a patient on warfarin presenting with an extremity injury also requires an INR lab draw, because anticoagulation status directly affects bleeding risk and management. That inference is standard clinical practice, but it is not written into the document itself. An agent reading the reference material in isolation will often conclude that the ankle injury warrants imaging only and assign ESI-4. The correct answer, for a patient on warfarin, is ESI-3.
The baseline dementia qualifier. The document lists "altered mental status (new): confusion, lethargy, agitation, or disorientation not explained by the patient's known baseline" as an ESI-2 high-risk complaint. The operative word is "new." A patient with documented severe Alzheimer's dementia who is confused and nonverbal at baseline is not presenting with new altered mental status when the family reports she "seems off." The change from baseline may warrant workup, but the trigger for ESI-2 specifically requires that the mental status change be unexplained by the known baseline. Agents reading the flat document frequently skip the qualifier, match the surface description to the AMS bullet, and over-escalate to ESI-2. The correct answer is ESI-3.
The COPD SpO2 threshold. The document contains a note that SpO2 of "92% or higher is acceptable in known COPD patients" under the normal vital sign reference section. It also specifies that the danger zone threshold for SpO2 is less than 92% for all patients, including COPD patients, in the danger zone table. These two statements are consistent when read carefully: the first is saying that 92-94% is not alarming for a known COPD patient; the second is saying that falling below 92% triggers escalation regardless of diagnosis. But placed in a flat document, agents sometimes conflate the COPD note with an exception to the danger zone threshold and misclassify a patient with SpO2 91% as safe. The correct answer is ESI-2.
The Patients
Three patients were designed specifically to test these failure modes.
P001 - Roberto S, 58M. Left ankle pain and swelling after missing a step on stairs. Moderate swelling, pain 3/10, unable to bear weight. History of atrial fibrillation. Medications: warfarin 5mg daily, metoprolol 25mg. Vitals all within normal range. Correct triage: ESI-3, multi-resource (INR lab plus imaging).
P002 - Margaret T, 81F. Family brought her in. She seems off today, not herself, not eating. Staff note: patient is nonverbal at baseline and severely confused per facility records. Resides in a memory care facility with a diagnosis of severe Alzheimer's dementia. Vitals within normal range. Correct triage: ESI-3, multi-resource (labs and IV fluids per elderly acute decline rule).
P003 - Sandra K, 64F. Worsening shortness of breath, wheezing, and productive cough over two hours. Using rescue inhaler with minimal relief. No chest pain or pressure. History of COPD on home O2 at 2L/min, 40 pack-year smoker. SpO2 91%. Correct triage: ESI-2, danger-zone vital (SpO2 below 92% threshold).
P003 is the control patient. The SpO2 threshold is unambiguous, directly stated, and does not require chained inference. Both approaches should get this consistently right. P001 and P002 are the trap patients: they require either clinical inference beyond the document (P001) or careful reading of a qualifier that the surface pattern tends to override (P002).
Two Approaches
The flat document approach. The agent receives a single prompt instructing it to read the ESI reference document and the patient CSV, work through each patient in the ESI decision order (life threat, then high-risk/danger zone, then resource count), and write results to a timestamped CSV. The agent has access to the complete reference text in one load.
The decision tree approach. The same body of knowledge is encoded as a decision tree in mcpforx, a platform (that I'm building) that delivers structured expertise to AI agents via MCP (Model Context Protocol). The agent receives a prompt instructing it to use the find_expertise tool to start a run, then call next_step at each stage until the tree terminates and outputs a result row. Rather than loading the entire reference document at once, the tree delivers exactly the relevant instruction and expected output format at each decision point. The resource count node, for instance, includes an explicit anticoagulation rule and an elderly acute decline rule that are absent from the reference document but represent standard clinical practice.
One important note on fairness: the tree does not contain information that is categorically absent from the reference material. Every rule encoded in the tree corresponds to knowledge that a competent reader of the document should be able to apply. The tree's advantage is structural. It delivers the right rule at the right step, rather than asking the agent to hold the entire body of knowledge in mind and retrieve the correct clause on demand.
Results
Six markdown runs and five tree runs were conducted across two model families: Claude Haiku 4.5 and DeepSeek V4 Pro. The same failure patterns appeared in both models on the markdown approach, confirming that the errors are structural rather than model-specific.
Per-Run Results: Markdown
Per-Run Results: Decision Tree (MCP for X)
Summary
Failure Mode Breakdown
P001: The Warfarin Inference
In five of six markdown runs, the agent concluded that Roberto S required imaging only and assigned ESI-4. The reasoning was consistent across runs and across models: the reference document explicitly lists "isolated extremity injury requiring imaging only" as an ESI-4 example, and the agent matched the presentation to that example.
The error is not a failure of reading comprehension. The agent read the document correctly. The failure is that the warfarin-to-INR connection requires a clinical inference that lives outside the reference material. A trained triage nurse would flag warfarin immediately; the document does not instruct them to. That inference is part of clinical training, not the reference text.
The one markdown run that produced ESI-3 did so for the wrong reason. Run 2 cited anticipated IV analgesia for pain control as the second resource. The document explicitly states that IV analgesics should only be counted as a resource for patients in significant pain (7 to 10 out of 10). At 3/10, oral analgesia is standard. The agent arrived at the correct ESI level through incorrect reasoning.
The tree encoded the anticoagulation rule directly into the resource count node: patients on warfarin, heparin, or other anticoagulants presenting with injury require a lab draw. Every tree run identified this and correctly assigned ESI-3 with INR plus imaging as the two resources.
P002: The Baseline Qualifier Override
In five of six markdown runs, the agent assigned ESI-2 to Margaret T by treating "seems off today, not herself, not eating" as new altered mental status, a documented ESI-2 high-risk complaint. This is clinically incorrect. The AMS criterion in the reference document explicitly requires that the change be "not explained by the patient's known baseline." This patient's baseline is documented severe dementia with nonverbal status in a memory care facility. The family's observation represents a change from her usual function, but the AMS qualifier as written does not trigger ESI-2 for a patient whose baseline already involves confusion.
What makes this failure particularly instructive is that the agent's reasoning sounds sound on the surface. An elderly patient with dementia presenting with behavioral change and reduced intake is a textbook presentation for UTI, sepsis, or medication toxicity. The instinct to escalate is understandable. But the ESI framework is not asking the agent to exercise clinical judgment about what might be wrong. It is asking the agent to apply a specific algorithm, and the algorithm includes a qualifier that the flat-document approach consistently fails to apply.
The tree gated the AMS trigger behind an explicit question: is the behavioral change a documented departure from a known non-baseline state? If the patient is nonverbal and confused at baseline, the answer is no, and the tree routes to the resource count node rather than the ESI-2 terminal. Every tree run correctly arrived at ESI-3 via labs and IV fluids under the elderly acute decline rule.
P003: No Trap, Consistent Results
Sandra K produced consistent correct results across all eleven runs. SpO2 91% below the 92% danger zone threshold is a direct, unambiguous match to a clearly stated rule. There is no qualifier to misapply, no inference required, no competing example to over-weight. Both approaches handled this reliably.
This is the expected behavior. The flat document approach is not universally unreliable. It works well on cases where the relevant rule is clearly stated, directly applicable, and does not require the agent to navigate competing signals or apply knowledge the document does not supply. The problem appears specifically in cases requiring chained inference or careful attention to qualifiers that the surface pattern tends to override.
What This Means in Other Domains
The failure modes documented here generalize to any domain where:
- A body of rules is long enough that the agent must decide which ones are relevant before applying them
- Correct decisions require chaining inferences across multiple knowledge sources, some of which may not appear in the reference document
- The reference material contains qualifier language that surface-level pattern matching tends to skip over
- Consistency across repeated runs matters, not just average accuracy
Financial compliance workflows share these characteristics. So do legal contract review procedures, insurance claim adjudication rules, and operational procedures for regulated industries. In each case, the organization has a body of rules, a set of cases to evaluate, and a need for decisions that are both accurate and consistent.
The instinct in most AI deployment efforts is to put the rules in a document and prompt the model to follow them. This experiment suggests that instinct is insufficient when the rules involve conditional logic, qualifiers, or inferences that require contextual knowledge the document does not fully specify. The model's attention is not uniformly distributed across a 150-line document, and the rules that receive less attention are exactly the ones that trip up less experienced practitioners.
Structuring a body of knowledge as a decision tree changes the information problem entirely. Instead of asking the agent to retrieve the right rule from a large flat context, the tree presents only the decision currently in front of the agent, with the exact instruction and output format that step requires. The agent does not need to hold the whole framework in mind because the framework is serving itself incrementally.
A Note on What the Tree Does Not Do
The tree does not inject information that is absent from the flat reference document. The anticoagulation rule and the elderly acute decline rule are applications of knowledge that a competent reader of the reference material should be able to apply. The tree makes those applications explicit and delivers them at the moment they are relevant.
This distinction matters for evaluating the fairness of the comparison and for thinking about how to build trees in other domains. The tree's job is not to compensate for an inadequate reference document by adding information it lacks. Its job is to ensure that the right part of a complete knowledge base is active at the right decision point.
A useful heuristic: if a human expert reading the same material would get the case right, but an AI agent reading the same document does not, the problem is structural. The tree is the structural fix.
Drawbacks of the Tree Approach
Serving professional expertise to AI agents is a profoundly important area of current research, and any method has pros and cons. One of the challenges with a decision tree approach is that it takes the AI agent longer to traverse the branches of the tree and gating logic. It will be faster to leave the decision-making entirely to the AI given some high-level direction, but then you lose fine-grained control over execution.
Another disadvantage is that you will get more "non-answers". Typically an AI agent will fail and recover and try and answer your question to the best of its ability. In the decision tree approach, if you hit a criterion that is novel or not covered in your tree, or if your tree isn't configured correctly - it may just stop operation and await your input. Depending on the sensitivity of the work, this could be a good thing, as a process erroring out is an opportunity to make the expertise richer and better for next time.
Finally, it takes time and effort to encode individual or team expertise in this format for usage with AI agents. It also takes time and effort to maintain it. While on surface it may be thought of as a negative, I have a different view. I tend to think that if this time and effort turns out to not be valuable - that would mean that anyone with a $200/month subscription has access to any professional level expertise. The input of human expertise for AI agents does not matter. I don't think that is the world we are headed towards. In fact, in a world of artificial intelligence, the ultimate differentiator will be human intelligence and the firms that can pair it the best with AI.
Conclusion
Across eleven triage runs on three carefully designed patients, the flat document approach produced an overall accuracy of 44 percent and zero perfect runs. The tree approach produced 100 percent accuracy across all five runs and both model families tested.
The patients were not trick questions. The correct answers were derivable from the same document the agent read, by anyone who read it carefully and applied the relevant background knowledge. The failure was not in the models' intelligence or in the completeness of the reference material. The failure was in the delivery mechanism.
Giving an AI agent a manual and telling it to follow the rules is not the same as building a system where the rules enforce their own correct application. For any decision workflow where accuracy and consistency are both required, the structure of how knowledge is delivered matters as much as the knowledge itself.
p.s. I don't want to beat up the markdown approach too much. The models are getting good enough that a simple markdown approach, or giving an agent a rulebook pdf - can work surprisingly well! But as context gets large, and vague or seemingly conflicting verbiage gets in - the reliability and accuracy takes a dip. A different mechanism for expertise delivery can structurally improve performance. In the end - we probably need both.
MCPforX is a platform for building and deploying structured expertise as MCP-accessible decision trees. The patient data (fictitious), a link to a live version of the tree, and full run traces from this experiment are available on request (should you wish to reproduce the results).