A system that extracts data from a document has a correct answer: the dimensional callout on the technical drawing is what it is, and it can be verified. A music recommendation engine doesn’t have that luxury. There is no “right album” to recommend, so there’s no reference answer to compare the output against and count errors.
That changes the evaluation method, not just the final number.
The problem the project solves
Music recommendation has traditionally relied on the collaborative signal: what people similar to you have listened to. That signal is missing exactly where it would matter most — a rarely played album, a new release, a user who just arrived — and on a catalog that combines a local library with streaming, that’s often the most interesting part of the catalog left invisible.
The system built indexes the catalog’s descriptive content — metadata, genres, relationships between artists, editorial content — and retrieves by semantic similarity, using the same retrieval architecture described for an enterprise RAG system, applied here not to corporate documents but to a music catalog. An album can be recommended even without any listening history: that’s exactly the point where collaborative filtering has nothing to say.
Why evaluation needs a person
A test set with correct answers, the way you’d build one for a document system, assumes a correct answer exists and is known in advance. Here it doesn’t: recommendation quality isn’t accuracy. That’s why evaluation requires real people judging whether a suggestion is relevant, not just metrics computed on historical data.
Two numbers emerge from this evaluation, and they need to be read together.
Relevance: 85% of recommendations were judged relevant by users in the evaluated sample.
Coverage: 65% of the catalog reached by the new logic, compared with the previous recommendation logic.
Why two numbers, not one
An engine that maximizes relevance alone ends up always recommending the same safe titles: the easiest way to get high relevance is to suggest what almost everyone likes. Coverage measures whether the system is actually reaching the long tail of the catalog, which was the original problem.
The two quantities move in opposite directions, the same way accuracy and coverage do in a RAG system: optimizing one without watching the other produces a result that looks good on one dimension and empty on the other. There’s a third quantity monitored separately — the diversity of the recommended catalog — that doesn’t yet have a published figure: high relevance obtained by always recommending the same titles wouldn’t be progress, which is why it stays watched even without a number to show for it.
What the system doesn’t do
It doesn’t invent titles. Generation stays constrained to items actually present in the catalog and available to the user: recommending a record that doesn’t exist or can’t be played is worse than recommending nothing. Every suggestion traces back to the content that produced it, so it’s explainable rather than opaque.
In closing
When there’s no correct answer to compare against, the temptation is to stop measuring and trust the impression instead. The more solid answer is the opposite: more people in the evaluation, more quantities read together, not a single figure that hides the trade-off.
The full case study describes the architecture and implementation. If you’re evaluating a recommendation or search system over a large, heterogeneous catalog, we’re available to talk through the evaluation method best suited to your case.
