An AI agent read a prior authorization policy document. The document was long, about 580 lines. It had an FAQ section, CPT code tables, a policy change log, forwarded emails from a medical director, and buried references to specialty addenda. The agent read it carefully. It processed four authorization requests. It reached four decisions.
Three of those decisions were wrong.
Same document, same model, four minutes later. The agent read the policy again. Same four requests. This time two were wrong. Same document, same model, five minutes after that. Two wrong again, but a different pair. The request that had been correctly approved in the first run was denied in the third. The request that had been incorrectly approved for inpatient psychiatric admission was instead incorrectly denied.
Three runs. Three different answers to the same question. Same policy, same model, same instruction. The error rate was only part of the story. The instability was the rest of it.
This is the fourth experiment in a series testing whether the delivery mechanism for knowledge affects how AI agents make structured decisions. The previous experiments covered claim adjudication, clinical triage, and financial due diligence. Each found that flat documents produced errors that decision trees eliminated. This experiment finds something more fundamental: when the flat-document approach fails, it does not fail the same way twice.
The solution is not a better model. It is the same one that solved the other three: encoding expertise as a traversable structure that controls what the agent sees and in what order. But the nature of the errors here makes a stronger case than accuracy alone. A system that is sometimes wrong and sometimes right is harder to fix than a system that is consistently wrong. A system that is wrong in different ways on different runs is harder still.
The Domain
Prior authorization is the process by which a health insurer determines whether a requested medical service is covered. A reviewer works through a sequence: verify the member has active coverage on the date of service, confirm the procedure requires authorization, apply clinical necessity criteria, check for step therapy requirements and applicable exceptions, and issue a decision: approved, denied, or pended.
The rules live in policy documents. The real ones run hundreds of pages with addenda, bulletins, and quarterly updates. I built a simplified set with the structural features that cause trouble: a primary policy memo, specialty addenda for neurological and sleep service exceptions, a clinical criteria manual, a benefit summary, and a collection of worked examples from prior authorization decisions, some of which are designed to be misleading. The test set consisted of four authorization requests.
The question was not whether the document contained all the information needed to reach the correct decisions. It did. The question was whether an agent reading it cold would find it.
General Note: this isn't representative of how such workflows work in the real world, nor is this article prescriptive on how such a process should work or even whether AI should be used. I tried to construct a scenario that is somewhat interesting and close to real workflows. The goal of this example is expose failure modes than LLMs have when applied in a typical way, and how to address them.
The Four Requests
R001. Lumbar spine MRI, documented foot drop. A 52-year-old male with chronic low back pain rated 7 out of 10 for eight weeks. Conservative therapy was not initiated. The clinical notes document progressive foot drop and difficulty walking on heels for two weeks. Physical examination confirms inability to dorsiflex the right foot. No bowel or bladder dysfunction.
Foot drop is a neurological red flag. It waives the conservative therapy requirement. The correct decision is APPROVED. The policy manual's spine imaging section states that lumbar MRI requires six weeks of conservative therapy and that failure to complete step therapy results in denial. The exception for neurological presentations is documented in a separate addendum, neuro_exceptions.md, whose existence is referenced only in a policy change log entry at the end of a 580-line document and in a now-deleted addenda cross-reference section. An agent reading the main policy document alone sees only the denial rule.
R002. Polysomnography, documented CPAP intolerance. A 61-year-old female with suspected obstructive sleep apnea. No prior home sleep test. Body mass index of 28.4. A prior CPAP trial was initiated nine months ago for clinically diagnosed OSA and discontinued after six weeks due to severe aerophagia and claustrophobia. CPAP intolerance is documented in physician notes dated July 2025.
Per addendum_73.md, polysomnography is directly authorized for members with documented CPAP intolerance within the preceding twelve months, without requiring a home sleep test. The correct decision is APPROVED. The main policy's sleep services section says home sleep tests are mandatory before PSG and that requests without HST results should be pended. The addendum is referenced only in the policy change log.
R003. Inpatient psychiatric admission, passive suicidal ideation. A 34-year-old female referred by an outpatient therapist. Reports passive suicidal ideation: thoughts of not wanting to be here, without a specific plan or identified means. No active intent. The patient denies any plan or means when asked directly. Family members are requesting inpatient placement due to general concern for safety. The safety assessment documents passive ideation only.
The clinical criteria distinguish between active and passive ideation. Active ideation with a specific plan and means or intent meets the inpatient danger-to-self criterion. Passive ideation without plan or means does not. The appropriate level of care is intensive outpatient program evaluation. The correct decision is PEND, with referral to IOP.
The main policy document contains a tension on exactly this point. The primary psychiatric criteria section describes the danger-to-self criterion in broad terms: suicidal ideation documented on a safety assessment, combined with family concern for the patient's safety at home, is generally sufficient. A forwarded email from the medical director reinforces this: do not downgrade to IOP just because the notes use the word passive. It is a descriptive term, not a coverage exclusion. The same document also contains a note, in the psychiatric section, that less acute presentations, specifically thoughts of not wanting to be alive without a method or plan, should trigger consideration of IOP. An agent has to weigh the medical director's explicit instruction against the clinical nuance embedded deeper in the same section.
R004. Total knee arthroplasty, lapsed coverage. A 67-year-old male requesting elective right knee replacement. Coverage terminated February 15, 2026. The requested date of service is April 10, 2026, 54 days after termination. Clinical documentation is complete: Grade 4 osteoarthritis, conservative therapy completed, functional impairment documented.
The correct decision is DENIED. Coverage ended before the date of service. Section 1 of the policy is unambiguous: if the coverage end date is populated and precedes the DOS, the decision is an automatic eligibility denial. Stop there. Clinical merits do not matter. R004 is the control. Both approaches should catch it every time.
Two Approaches
Flat document. The agent receives a minimal prompt: read the policy document main_policy.md, then process each request in data.csv and output decisions. It is told to read only that file and data.csv and to not search for or read any other files. The main policy document is approximately 580 lines in an email-memo hybrid format with interleaved forwarded messages, a fifteen-question FAQ, CPT code tables, a policy change log, and clinical criteria sections for spine imaging, sleep services, psychiatric admissions, and orthopedic surgery. The specialty addenda containing the neurological and CPAP exceptions are not mentioned in the prompt and are not discoverable from the main policy without reading to the document's final sections.
Decision tree. The agent is told to use the Healthcare Prior Authorization Screener (code F56DCU) through the MCPforX platform. It initiates a run with a natural language query matching each procedure and follows the tree step by step using next_step calls. The tree gates every decision explicitly: eligibility check first, then procedure classification, then branch-specific criteria. For musculoskeletal imaging, neurological red flags are checked before step therapy. For sleep studies, the CPAP failure exception is checked before standard HST criteria. For psychiatric admissions, the active versus passive ideation qualifier is applied as an explicit triage rule with an all-or-nothing output format. The agent does not discover rules. The tree delivers them.
Both approaches used the same model, DeepSeek V4 Flash. Both had access to the same underlying policy rules. The only difference was delivery.
Results
Four flat-document runs, four tree runs. Same model, same requests, same information baseline.
Flat Document
Overall accuracy: 37.5%. Zero perfect runs.
Decision Tree
Overall accuracy: 100%. Four perfect runs.
By Request
What Each Failure Reveals
R002: The Distributed Criteria Problem
Four flat runs, four identical wrong answers. The agent read the main policy's sleep services section, saw that home sleep tests are required before polysomnography, and issued PEND with instructions to order the HST first. Not once did the agent discover addendum_73.md.
This is not a reading comprehension error. The document mentioned the addendum. The mention was buried in the policy change log at the end of a 580-line document, in a bullet point alongside other quarterly updates about DME formulary changes and provider webinar notes. The sleep services section itself stated the HST requirement unequivocally, with no footnote or cross-reference. An agent that reads the section header, the criteria, and the decision rule has no reason to look further. It found the rule that matched the service category and applied it.
The real world is worse. A health plan's CPAP failure exception might live in a clinical policy bulletin issued six months after the main criteria manual, referenced in a quarterly update memo that the reviewer's department receives but does not integrate into the primary reference. The agent sees the primary reference. The exception is somewhere else. The structure produces the error.
The tree resolved this by placing the CPAP failure check before the standard sleep study criteria. The agent never reached the HST requirement node because the tree routed it through the addendum_73 check first, where documented aerophagia and claustrophobia within twelve months triggered approval directly. The correct rule was applied not because the tree was smarter than the document, but because the tree placed the exception gate ahead of the general rule. A human designed the control flow once. Every subsequent run inherited it.
R003: The Inconsistent Guidance Problem
Four flat runs, four wrong answers. But they were not the same wrong answer. Runs 1, 2, and 4 approved the inpatient psychiatric admission. Run 3 denied it.
The difference is instructive. The document contained two signals about passive suicidal ideation. The psychiatric criteria section described SI with safety assessment and family concern as generally sufficient for the danger-to-self criterion. The medical director's forwarded email reinforced this explicitly: do not downgrade to IOP because the notes say passive. The same section also contained an additional considerations note, further in, describing less acute presentations, thoughts of not wanting to be alive without method or plan, as candidates for IOP referral.
An agent processing this document must weigh these signals. Three runs gave more weight to the medical director's explicit instruction and the broad framing in the danger-to-self criterion. One run gave more weight to the additional considerations note and the clinical nuance it encodes. Neither answer was correct. Both answers were defensible from the text. The document contained a contradiction, and the model resolved it differently across runs.
This is the kind of failure that is hardest to detect in production. A reviewer looking at any individual decision would find supporting policy language. The contradiction would only be visible by comparing decisions across runs and noticing that the same language, applied to the same facts, produced opposite results. The tree eliminated the contradiction by encoding the active versus passive qualifier as an explicit triage rule: all three conditions, active ideation, specific plan, and means or intent, must be true for the inpatient criterion to be met. If any is false, the default path routes to IOP. No weighing required. No contradictory signals to resolve.
R001: The Domain Knowledge Instability Problem
R001 was the only flat result that was sometimes correct. The document's spine imaging section stated that lumbar MRI requires six weeks of conservative therapy and that failure to complete step therapy results in denial. The neurological exceptions addendum was not referenced in the section. The policy change log mentioned neuro_exceptions.md but without describing its contents.
Two runs approved R001. Two runs denied it. The approving runs filled the gap with domain knowledge: foot drop is a well-known neurological red flag, inability to dorsiflex on examination is an objective finding, and progressive motor deficits warrant imaging without waiting for conservative therapy. The model recognized the clinical pattern and overrode the strict policy language. The denying runs followed the document literally: step therapy not completed, standard determination is denial, neurological exceptions are not described in this document. Both reasoning paths were sound given what was in front of the agent.
This is not a failure of either approach. It is a failure of the whole model. A system that is right half the time on the same input is not a system. It is a coin flip. The instability cannot be fixed by clarifying the document because the two runs that got it right were the ones that ignored the document. Making the document clearer about the absence of an exception would make all four runs wrong. Making it clearer about the exception would make all four runs right. Neither change addresses the underlying issue: the flat document approach delegates to the model a decision about whether to follow the text or follow its training, and the model decides differently across runs.
The tree did not face this ambiguity. The neurological red flags node explicitly listed foot drop as a qualifying condition. The agent was told what to check and what to produce. The question was not whether foot drop qualifies. The question was whether the clinical notes document it. They did. The answer was APPROVED, four times out of four.
The Instability Argument
Every experiment in this series has found that decision trees outperform flat documents on accuracy. This experiment found the same: 100 percent versus 37.5 percent. But the accuracy gap, while large, is not the main finding.
The main finding is the instability. R001 flipped correct to incorrect across runs. R003 flipped between two different wrong answers across runs. Given the same documents, the same prompt, the same model, and the same requests, the flat-document approach produced three different answer sets across four runs. No run got all four right. No two runs agreed completely.
This matters for organizations deploying AI on structured decisions. A system that is consistently wrong on a particular case type can be patched. You identify the failure, you fix the document or the prompt, you retest. The fix sticks because the error is deterministic. A system that is inconsistently wrong cannot be patched. You identify a failure in one run, you make an adjustment, you test again, and the failure has disappeared not because the adjustment fixed it but because the model happened to choose a different reasoning path on the second attempt. You think you solved the problem. You did not. The next run might reinstate it.
The tree eliminates this class of problem by eliminating the model's discretion over what to attend to. The model does not decide which rules are relevant. The tree decides, by controlling the sequence of instructions. The model does not resolve contradictions between policy sections. The tree resolves them in advance, by encoding the resolution as conditional logic. The model's job is to extract the relevant facts from the clinical notes and produce structured output. That job is narrow enough that it is performed consistently.
Better models will narrow the gap on tasks where the model must synthesize information from multiple sources. They will not narrow the gap on tasks where the model must decide, across runs, which of two contradictory signals to prioritize. A more powerful model might prioritize the right signal more often. It might be right 65 percent of the time instead of 37.5 percent. It might be right 80 percent. But it will never be right 100 percent of the time in a system where correctness depends on a choice the model makes anew on every invocation. Consistency requires removing the choice.
What the Tree Changes
The tree does not change the information. The neurological red flags node in the tree contains the same list of conditions described in neuro_exceptions.md. The CPAP failure exception node contains the same criteria described in addendum_73.md. The psychiatric ideation triage node applies the same active versus passive qualifier encoded in the clinical criteria manual. The tree has no access to information the flat-document agent lacks.
What it has is sequence. The tree presents one instruction at a time. Each instruction asks for a structured output. Each output routes to the next instruction. The model is never asked to hold the full policy framework in mind and retrieve the correct rule. It is asked to answer the question currently in front of it using the instruction that accompanies it.
This is what structured expertise looks like. A human expert does not hold a 580-line policy document in working memory and scan for the relevant passage on each case. They work through a sequence: eligibility first, then service classification, then exception gates, then criteria. The tree makes that sequence explicit and gives it to the model. The investment is human. The return is mechanical.
Conclusion
Across eight authorization runs on four requests, the same model was right 37.5 percent of the time when asked to navigate a flat policy document and 100 percent of the time when guided through a decision tree. The correct answers were in the policy. The model was capable of reaching them with the right structure. The limiting factor was not model capability. It was the absence of a delivery mechanism that controls what the agent reads, in what order, with what instruction.
The R002 result is the consistency failure worth sitting with. Four flat runs, four identical wrong answers. Zero tree runs wrong. The addendum was in the shared drive, mentioned in a changelog at the end of a long document. Every run found the rule that matched the service category and applied it. No run found the exception that should have overridden it. The structure, not the information, determined the outcome.
The R001 and R003 results are the instability failures worth sitting with. R001 flipped correct to incorrect between runs because the model sometimes deferred to its domain knowledge and sometimes followed the literal text. R003 flipped between two different wrong answers because the document contained contradictory signals and the model resolved them differently on each invocation. A system that is unpredictably wrong on the same input is not a system. It is a lottery with policy language.
MCPforX is a platform for building and deploying structured expertise as MCP-accessible decision trees. The authorization data is fictitious. Full run traces are available on request.