Agent evaluation
AI Agents Need Environments, Not Just Context Windows
Patientic currently uses synthetic, educational, or otherwise permitted scenarios. It is not intended to diagnose, treat, or replace professional medical judgment.
An agent that only receives a larger context window is still guessing inside a document. An agent that can work needs an environment: inspectable state, permissions, actions with consequences, and a history that outlasts the call. Context is how a model sees. The environment is what is there to see, what may be done, and what remains true afterward.
The tempting fix, whenever an agent fails, is to show it more text. Paste the chart. Paste the policy. Paste the last attempt. Raise the window size and hope the next completion notices what the last one missed. Sometimes that helps, because the model literally could not see a sentence. Often it does not help, because the failure was not blindness. The failure was that nothing in the setup could say which facts were available, which actions were allowed, or what changed because of what the agent did. More text cannot invent those rules. It can only hope the model infers them from prose.
Patientic is being built as that missing environment for patient-shaped work. The design is a governed world, not a smarter paste buffer. External agents meet the world through a permissioned surface. They propose. The world records what was permitted, what was committed, and what state followed. This essay is about why that split matters. It is not a claim that any particular agent is already good at the work.
A window is a view
A context window is a bounded view assembled for one model call. It has a budget. Something, somewhere, chooses which strings fit. When the budget is tight, the usual remedies are truncation and summary. Both are lossy. Truncation drops the tail or the head. Summary replaces events with a paraphrase, and the paraphrase can omit the one fact the next action depended on.
Even a huge window is still a view. It does not know which sentences are hidden facts, which are already disclosed, and which are the model's own earlier guesses that got copied back in. Once a guess is in the transcript, the next call treats it as text like any other text. The environment, if it exists, can refuse that promotion. The window cannot. The window's only operation is to be full of characters.
What an environment has to hold
A useful environment for this kind of work has five properties. They are ordinary in software and easy to skip when the interface is a chat box.
State is inspectable. A reviewer can read what the world currently holds without asking the model to recall it. The state is structured enough that two observers can point at the same field and mean the same thing. Prose can sit beside that structure. Prose cannot replace it, because prose does not diff cleanly and does not name permissions.
Permissions bound action. An agent may be allowed to ask, to read a disclosed fact, or to propose a commit. It may not be allowed to rewrite history, to open a hidden fact, or to treat a suggestion as already true. The permission is checked by the environment. It is not a sentence in the system prompt that the model is trusted to obey.
Actions are discrete. "Think about the case" is not an action the world can record. Asking for a specific unavailable fact, committing a decision, or advancing time are actions. Discrete actions can be accepted, rejected, or recorded. A continuous stream of tokens cannot, except by guessing where one intention ended and the next began.
Consequences follow from accepted actions. If a fact is disclosed, later views may include it and earlier views still must not. If a decision is committed, the clock and the branch move. If an action is rejected, the state does not change and the rejection itself is visible. A consequence that exists only inside the model's next paragraph is not a consequence. It is a continuation.
History keeps the sequence. The environment stores who acted, what they attempted, what was allowed, and when. Later evaluation reads that history. It does not ask the agent for a sincere summary of its own behavior. Agents are poor witnesses to their own process, because the process they report is another generation, not the log.
Permissions are the product
In a patient world, the sensitive design choice is not the tone of the reply. It is the gate around the action. A fluent agent can sound careful while doing something the setup did not allow, such as treating a hidden fact as known or skipping ahead in time because the story would be shorter. The environment has to make the illegal action fail in the state, not merely look impolite in the prose.
Patientic's design puts that gate outside the model. The agent proposes. Policy decides. The result is an event: accepted, rejected, or answered with a narrower disclosure than the agent asked for. That event is what a later reviewer grades. The paragraph the model wrote to justify itself is evidence of how it talked, not proof of what the world allowed.
This is also why a tool list stapled onto a chat is only half an environment. Tools matter. They give the model verbs. They do not, by themselves, supply a patient state those verbs must respect, a clock, or a rule that hidden facts stay hidden until a permitted action reveals them. A tool that returns "whatever the model needs" has collapsed the permission into the tool's implementation, where nobody can see it. The world described in What Is a Persistent Virtual Patient? is the object those verbs should act on.
Consequences you can point at
An environment is only real if the next state is a function of the previous state and the accepted action, not of the model's mood. That sounds severe. It is the minimum requirement for comparison. If two agents face the same state and take different permitted actions, you should be able to show two resulting states and say what differed. If the only difference is two essays, you compared writing samples.
agent
sees a view of state (not the whole world)
proposes an action
|
v
environment
checks permission + policy
accepts or rejects
writes an event
produces the next state
|
v
history: actor, action, decision, time
Read the diagram from the agent downward. The agent never holds the world. It holds a view. The environment is the only writer of the next state. History is the byproduct of that write, not a diary the agent keeps for itself.
History is the evaluation surface
Once actions have consequences, evaluation can move off the final answer. You can ask whether the agent requested information it was allowed to request, whether it committed before it had a disclosed basis, and whether a second run from the same state produced a different event sequence. Those questions are the core of How to Evaluate AI Agents in a Patient World. They are unanswerable if the only artifact is a completion, because a completion does not distinguish "the world disclosed this" from "the model invented this."
History also makes permissions auditable after the fact. A prompt that says "be careful" leaves no trace of care except the adjectives in the reply. An event that says the disclosure was refused leaves a trace a person can read without trusting the prose. That is the difference between an agent demo and an environment.
None of this requires claiming that a simulated action is clinically correct. The environment can be strict and still be a simulation. Strictness here means the state machine does what it says. It does not mean the scenario encodes the right medical decision. Those are separate questions, and the second one is not what this system is for.
Patientic is being built so external agents operate a patient world under those constraints, through the same permissioned surface a person can inspect. The longer context window can still matter, as a better view. It is not a substitute for the world. The product those agents would meet is Patientic.
Related reading
See the patient world this writing is about.
Explore Patientic