I've been building pieces towards a goal: Can we get AI to do complex knowledge work with a high level of determinism and competence.
This goal does two things simultaneously. It increases productivity by scaling knowledge work using AI, and at the same time making human experts more valuable, not less. The ultimate value is in the creation and maintenance of expertise.
Over the next few days I'll be doing a detailed test of what I've built and sharing results. What better topic to pick than actuarial risk adjustment. Here is the flow of what I'd like my AI agent to accomplish.
Overall objective: Risk score MEPS database so I can do the really interesting analyses.
Details:
- Set up my workspace. I like a certain folder structure for my work.
- Download latest ACA risk adjustment model, set it up
- MEPS has a whole bunch of data limitations that make risk scoring very difficult, sort through all that and configure the model to work with it
- Risk score and test the results for reasonableness in a comprehensive way.
- Provide me with email updates at critical points of the entire workflow, with a summary at the end.
No matter how expensive of a model you use or how many tokens you burn, giving an AI the list above is not going to accomplish the task. And even if you get something resembling an output, running it repeatedly would get you wildly unpredictable results. I've already demonstrated this ad nauseum in earlier posts.
So what do you do?
Follow along for the full recipe on how to accomplish this exact workflow completely autonomously. Hint: human expertise encoded as composable decision trees guiding the AI agent of your choice, using MCP for X.