What a Thousand Patient Simulations Helped Us Notice About Medical AI Interaction

A thousand completed patient simulations helped us notice a limit in the interaction, not a truth about clinicians or real patients. The sessions were synthetic. Learners spoke or typed, finished a structured encounter, and received formative feedback. As of 31 August 2026 that use had passed a thousand completed sessions among more than 300 users. That snapshot is early commercial operation of a formative product. It is not product-market fit, not clinical validation, and not evidence of real-world care performance. Nothing in it proves how a clinician would act, how a real patient would respond, or whether anyone got better.

What the volume made hard to ignore was structural. You can review questions, omitted information, explanation, trust, and feedback inside one session. You cannot watch those things matter later if the patient is thrown away when the session ends.

Questions only count if the world can withhold an answer

Inside a structured case, a question can be asked or left unasked, and the case can allow or withhold what follows. A session transcript can show whether the question was asked. Formative feedback can mark it against a rubric. The mark points at a turn. It does not, by itself, give that turn a later life.

The thousand sessions did not yield a public statistic about which questions learners skip. We are not publishing one. What they helped us notice is that a question's importance is partly about the next encounter. If the fact stays hidden and the session then resets, the simulation never has to live with the gap. The learner can be told they missed an item. The patient world does not change, because there is no patient world left. Repeated practice made that difference obvious: feedback about a question is not the same object as a state that still lacks the fact.

This is a product-learning signal. It is not a claim that real clinicians under-ask, or that synthetic patients reveal how people behave in clinics. The signal is about the software. A system that only stores the transcript can grade the question. A system that stores state can make the unanswered question still true later.

Omitted information needs somewhere to remain omitted

An omission in a single session is a gap in a document. An omission in a longitudinal simulation is a fact the world still holds and the actor still has not been shown. Those are different. The first can be listed at the bottom of a score report. The second has to survive the report.

Session-level formative scoring, which is what the current MediTutor product does, can tell a learner what the rubric says was missed in that encounter. It does not, by itself, carry the miss into another encounter with the same patient. Cross-encounter state is the design direction, not a feature we are claiming was already present in those sessions. The notice is the absence: after a thousand resets, "missed" was a comment on a transcript, not a property of a patient that the next session was required to respect.

From MediTutor to Patientic is the longer account of that limit. The evaluation stance that follows from it, looking at the path rather than only the final answer, is in How to Evaluate AI Agents in a Patient World. The same stance applies to a human learner. The final paragraph of an encounter is a weak place to see an omission.

Explanation is not the same thing as a state change

Learners explain. The transcript records the explanation. A rubric can say whether the explanation covered what the case asked for. None of that tells you whether the simulated situation changed because of the explanation. Understanding, agreement, and confusion are easy to perform in prose and hard to keep as state.

What repeated sessions helped us notice is the temptation to treat a fluent explanation as the outcome. Fluency is in the transcript. An outcome that later encounters could depend on would have to be written down as a change in the patient world: what was disclosed, what was understood well enough to store, what was left unresolved. Without that write, every explanation is a fresh performance. The next session cannot tell a careful explanation from a vague one except by rereading, and rereading is not the same as continuity.

We do not have a measured result that says a particular style of explanation changed the simulated situation across those thousand sessions. The sessions were not built to carry that result. The product could hear an explanation. It could not make the explanation consequential.

Trust does not live in a single tone of voice

Trust shows up in the design of a longitudinal case as something that can change and then matter at a later encounter. In a session that ends, trust is at most a tone in the dialogue or a line in the feedback. It is not a field the next encounter must read.

The thousand simulations helped us notice that gap without measuring anyone's trust. We are not reporting that learners gained or lost rapport, and we are not reporting how real patients trust clinicians. We are reporting that a resettable encounter has nowhere to put trust so that it constrains the future. If trust cannot change the next state, the simulation can talk about trust and still be unable to show it. That is a reason to want longitudinal state. It is not a finding from a trust study.

Feedback is only as longitudinal as the record under it

Formative feedback in the current product is session-level. It can be grounded in what was said and what the rubric marks as missing. Institutions, in the design we are building toward, should be able to review that feedback, not merely accept a model's adjective. Transcript-grounded feedback is valuable exactly because it points at events. It becomes incomplete when the events have no state to update.

Any use of those transcripts for training or evaluation is subject to applicable consent, data rights, de-identification, and governance controls. The sessions were not used to train a foundation model, and this essay is not a claim that they should be. The use that informed Patientic was product design: seeing that a score at the end of a transcript cannot describe a patient who is supposed to continue.

Repeated use mattered here in a modest way. People came back for more than one synthetic encounter. A return visit that starts from a blank case is still a new session. A return visit that starts from stored state would be a different product. The return itself does not prove learning. It shows that practice was repeated often enough for the missing continuity to be a design problem rather than a hypothetical one.

What we will not say the sessions proved

They did not prove how medical AI behaves in care. They did not prove which questions are clinically essential. They did not rank models, learners, or explanations. They helped us notice that questions, omissions, explanations, trust, and feedback are all stunted when the patient is disposable. The next design problem is to give those signals a state, a clock, and a history. That problem is why a transcript, on its own, is not enough, which is the subject of the companion essay on longitudinal state.

Related reading

• From MediTutor to Patientic: Why a Patient Cannot Be a Disposable Chat Session
• Why Transcript Feedback Needs Longitudinal Patient State