Writing a good instruction for a model is useful to one person. Designing the instructions of a system is what makes the result repeatable.
The difference is the same as the one between knowing how to write a query and designing a database schema. The first is a personal skill that improves the work of whoever has it. The second is a component of the architecture, with requirements for versioning, testing and maintenance.
Many organizations have invested in the first and neglected the second. The result is that people work better individually, while the systems in production remain fragile.
The difference between individual use and a system
In individual use, the instruction is ephemeral. If the answer doesn’t satisfy, you rephrase. The correction loop is immediate, the cost of an error is zero, the only judge is whoever asked the question.
In a system, the conditions invert.
The instruction is fixed and gets applied to thousands of different requests, many of which whoever wrote it never anticipated.
Whoever receives the answer never saw the instruction and can’t correct it.
An error isn’t an isolated case: it’s a systematic behavior that repeats until it’s caught.
And the behavior needs to be documented, because in a regulated context the instructions that govern a system are part of its technical description.
Three requirements follow from this that don’t exist in individual use: versioning, testing, regression management.
Why an untested change can make a thousand answers worse
This is the risk specific to this component, and it isn’t intuitive.
A system’s instructions are interdependent. Adding a rule to handle a problematic case changes behavior on cases that were working too.
A recurring example. The system answers too briefly on some complex questions. An instruction is added requiring more detailed answers. The side effect is that the system becomes verbose on simple questions and, more importantly, tends to extend its inference beyond what the documents actually state, because it needs to fill more space.
The fix solved one case in a hundred and made quality worse on the other ninety-nine. Without a test set, the effect surfaces weeks later, through scattered reports that are hard to trace back to the change.
This behavior is the reason changing a production system’s instructions needs to be treated like a software change: proposed, tested, approved, versioned, released. Not as an adjustment applied directly.
How to structure a governed instruction library
Five elements.
Separation by layer. A system’s instructions aren’t a single block. There are general behavioral rules, valid across the whole system: how to cite sources, when to abstain, what register to use. There are task-specific instructions: classify a document, draft a summary, extract fields. And there are instructions that depend on the context of use.
Separating them makes it possible to change one without touching the others, and to reuse the general rules across multiple systems.
Versioning. Every instruction has a version, a date, an author and a reason for the change. This is essential for establishing whether an error found today also affects output produced in the past.
An associated test set. Every instruction has a corresponding set of cases with expected outcomes. A change is validated by running the full set, not just the cases that motivated the change.
An approval path. Who can propose a change, who verifies it, who authorizes it. In systems that affect processes with formal accountability, approval involves the process owner, not just the technical contact.
Readable documentation. The instructions contain governance decisions expressed in operational language: when the system abstains, what information it must not provide, when it asks for confirmation. They need to be understandable to whoever is accountable for the process and whoever checks compliance.
The link with continuous evaluation
These two elements are the same mechanism seen from two sides.
A test set without governed instructions measures a system that changes without control. Versioned instructions without a test set produce traceability of changes without knowing whether they improved or worsened behavior.
The operating cycle has five steps. A problem is detected, from a report or from monitoring rejected outputs. A change is formulated. The full test set is run, comparing results against the previous version by metric and by case category. It’s assessed whether the gain on the problematic cases outweighs any loss elsewhere. It’s released with version and rationale recorded.
The third step is the one most frequently skipped, and it’s the one that keeps the system reliable over time.
One more observation, on model changes: instructions calibrated on one version can behave differently on another. Every model update requires a new run of the test set, not just a functional check.
What this approach doesn’t solve
It doesn’t compensate for inadequate document retrieval. No instruction can make a model produce a correct answer from the wrong material.
It doesn’t eliminate variability. Language models are, by construction, non-deterministic: the same request can produce different wordings. Instructions reduce the spread, they don’t eliminate it, and systems with rigid output requirements need to be designed with this in mind.
It doesn’t replace deterministic rules. A constraint that can be expressed as a verifiable condition should be implemented as one.
And it isn’t a one-off activity. The library needs to be maintained like any other component.
In closing
The quality of a production system depends on its instructions as much as on its retrieval architecture, and instructions are the one component that tends to get changed without a process.
Treating them as code — versioned, tested, approved — isn’t a formality. It’s what makes it possible to change them without degrading behavior.
If you have a system in operation whose instructions get changed without a verification cycle, we’re available for a conversation about how to set up the library and the test set.
