You can tell a good vendor by the questions they ask about your process, not the technologies they list.
It’s a simple criterion, and it discriminates more than it seems to. A vendor who, after an hour-long meeting, knows the architecture they’re proposing but doesn’t know how many cases you process a month, who’s accountable for that process, or what state your documents are in, hasn’t yet understood what you need.
The four areas below are the ones that, in our experience, decide how a project turns out.
1. Ask about the method, not the tools
Tools change. Within eighteen months the technically correct choice today can be superseded, and a vendor tied to one platform will tend to propose it regardless of the problem.
Method is more stable and more informative.
How they arrive at the definition of scope. A vendor who accepts the scope the client proposes without discussing it isn’t adding expertise.
What gets checked before design begins: data quality, system accessibility, clarity of the rules, existence of an internal owner.
How unforeseen cases are handled, where the system stops, who decides the thresholds.
How the rollout is organized: a single delivery, or verifiable phases.
One question that produces revealing answers: which projects have you advised against, and why. A vendor who’s never advised against anything has a different case-selection process than the one they claim to have.
The reverse question is worth asking too: what about our process concerns you the least. If the answer is that everything looks fine, the analysis hasn’t been done.
2. Ask which indicators they propose and how they measure them
A competent vendor arrives with a proposal for indicators; they don’t wait for the client to ask.
Four checks.
Which indicators they propose, and whether they relate to the process or to tool usage. Active users and number of queries don’t measure a return.
How they intend to record the baseline, and when. It has to be before development starts: afterward, it’s no longer possible.
How they measure the system’s accuracy: with what set of cases, built by whom, including which categories. The correct answer includes hard cases and questions with no answer in the archive.
What threshold they propose for authorizing go-live, and how they handle it if it isn’t reached.
A concrete request: ask to see the format of the evaluation report that will be delivered. A vendor who has one shows it within minutes.
3. Ask what happens when the contract ends
This is the part least discussed during negotiation and the most costly if neglected.
Six points to define in writing.
Ownership of custom-developed code and the right to use it if the relationship ends.
Ownership and return of the data, indexes and configurations, with the format specified. “Return of the data” without a specified format can end up meaning an unusable archive.
Ownership of the instructions and libraries developed, which in these systems represent a significant share of the work done.
Ownership of any specialized models and of the example sets built for training.
Technical documentation: what gets delivered, at what level of detail, current as of what date.
Migration support: duration, content, cost.
The topic of data generated during operation should be added too: rejected-case logs, operator corrections, an enriched test set. This is material produced by your organization and it has value for any future transition.
4. Ask for a verifiable reference
Not a list of logos. A project in operation, with the possibility of talking to whoever uses it.
There are five questions to ask the reference contact, and none of them is about technology.
Is the project still in daily use, and for how long.
Which indicators were measured, and what changed compared to the starting situation.
What didn’t work as expected, and how it was handled. This is the most informative question: every project has had difficulties, and the answer reveals how the vendor behaves when things get complicated.
How much internal time the project required, beyond the fee. This is the cost that almost never shows up in preliminary evaluations.
Would you make the same choice again.
If the vendor can’t provide references for confidentiality reasons, that’s a legitimate circumstance, common in the public and financial sectors. In that case it’s reasonable to ask for a detailed anonymized description, with numbers and difficulties encountered, and to judge the depth of the answer.
What these questions don’t guarantee
They don’t guarantee the project will succeed. A significant part of the outcome depends on the commissioning organization: data availability, the presence of an internal owner, the time of operational staff.
They don’t replace a security review, which requires its own set of questions on data location, permissions, logs, deletion and incident accountability.
They don’t compensate for a poorly defined scope. If the problem to be solved isn’t clear, no vendor will clarify it for you, though a good vendor will flag it.
In closing
Selecting a vendor in this field is more like choosing a designer than buying a license. What you’re buying is a method applied to your process, and the check needs to be on the method.
The most reliable signal remains the first one: the quality and specificity of the questions they ask you about your work.
If you’re evaluating a project and want a preliminary technical conversation about your process, with no commitment, we’re available. We’ll ask the first questions.
