Prove It
Trying a few questions by hand proves nothing. 65 questions with known answers, run through six versions of the bot, checked automatically. Every answer here is real.
The results
I wrote 65 questions with known answers and ran six versions of the bot through all of them. The AI is the same in every row. Only the way it looks things up changes.
Same AI, six ways to find the pages
65 questions with an answer keyWhere each version finds the right page
| Product codes | In other words | Old prices | Two facts at once | From a table | Trick questions | Only in one email or note | Other languages | |
|---|---|---|---|---|---|---|---|---|
| Equal pieces · words | 5/5 | 4/8 | 8/8 | 4/5 | 2/4 | 5/5 | 13/19 | 1/5 |
| Equal pieces · meaning | 4/5 | 7/8 | 8/8 | 4/5 | 2/4 | 5/5 | 19/19 | 5/5 |
| Sections · meaning | 4/5 | 8/8 | 8/8 | 4/5 | 3/4 | 5/5 | 19/19 | 5/5 |
| Sections · both | 5/5 | 6/8 | 8/8 | 4/5 | 3/4 | 5/5 | 18/19 | 5/5 |
| Sections · both · second look | 5/5 | 8/8 | 8/8 | 5/5 | 4/4 | 5/5 | 19/19 | 5/5 |
| Sections · both · second look · skip old | 5/5 | 8/8 | 8/8 | 5/5 | 4/4 | 5/5 | 19/19 | 5/5 |
What the quick version said
Word for word, from the test runs. The quick version gives up, hedges, or misses the one number you needed. The careful one finds it.
Der Studio-Plan kostet laut dem aktuellen Preisblatt für 2026 einen bestimmten Betrag pro Monat, der in den Quellen jedoch nicht vollständig angegeben ist. Die Quelle [4] nennt die Preise für 2026, doch der Auszug, der mir vorliegt, enthält nur den Starter-Plan mit €2.400 pro Monat [4]. Den genauen Preis des Studio-Plans kann ich daher nicht bestätigen.
Der Studio-Plan (Code NW-RET-M) kostet €5.800 pro Monat, exklusive MwSt. [3][5] Er umfasst 64 Design-Stunden und 2 aktive Anfragen gleichzeitig. [5]
The Partner plan is €11,200 per month in the current reference matrix [5], which would be €134,400 per year before any prepayment discount. The draft for 2026 proposes a Partner retainer of €12,000 per month and removal of the annual prepayment discount, but that is a draft and not in force [2][3]. I can't give a confirmed annual prepaid price based on these sources.
The Partner plan paid annually in advance costs €120,960 per year, which is 10% off the monthly price of €11,200 [1][2]. This prepayment discount applies to the retainer only and does not combine with any other discount [2][3].
Clause 14 governs how long files are kept, but the sources provided do not state the retention period itself. Source 3 says Figma access is removed 30 days after the engagement ends and that removal is not deletion of files. I don't know based on the Northwind documents.
Northwind keeps project files for 24 months after the last invoice, then deletes them, and you can ask in writing for earlier deletion, which is confirmed within 10 working days [2][3][5]. Invoices and their supporting accounting records are kept for 7 years and are not deleted along with the project files [3][5]. Separately, Figma access for your team is removed 30 days after the engagement ends, so export your files before then [1][3].
Check every answer
Every square is a real answer from the test runs: green right, pink wrong. Click one to read it.
How much would the Partner plan cost per year if paid in advance?
Must mention “120,960”.
- 1Equal pieces · words
- 2Equal pieces · meaning
- 3Sections · meaning
- 4Sections · both
- 5Sections · both · second look
- 6Sections · both · second look · skip old
How to test an AI
Ask the hard questions
Easy questions pass on every version. The useful ones are the tricky kinds: changed prices, facts hidden in one email, other languages, and questions it should refuse.
Check with a script, not a feeling
A script checks each answer for the facts it must contain. Same answer, same verdict, every time. When the key is wrong, fix it and re-check everything in seconds.
Find where it broke
Two scores: did it find the right page, and did it answer right. A wrong answer with the right page is a different problem from never finding the page.
The code
How an answer gets checked, in one sentence. The code is one click away.
Each test question comes with the facts a right answer must contain and the old numbers it must not lead with. A short script checks every answer against that list.
Show the code anyway
export function grade(q: EvalQuestion, text: string) { const declined = DECLINE.test(text.trim()); if (q.kind === "unanswerable") return declined ? ok("declined") : wrong("answered a question the documents don't cover"); if (declined) return wrong("said it doesn't know"); const missing = q.facts.find((f) => firstAt(text, f) === Infinity); // "5,800" | ["5.800", …] if (missing) return wrong(`missing ${JSON.stringify(missing)}`); // Mentioning the 2025 price as history is fine. Leading with it is not. const firstFact = Math.min(...q.facts.map((f) => firstAt(text, f))); const bad = q.forbid?.find((f) => at(text, f) >= 0 && at(text, f) < firstFact); return bad ? wrong(`led with ${JSON.stringify(bad)}`) : ok("all facts present");}// lib/rag/evalset.ts{ id: "s1", kind: "stale", q: "How much is the Studio retainer per month?", facts: ["5,800"], forbid: ["5,200"] },Still curious?
A few more questions people ask. Open what you need.
Why not let another AI do the checking?
When there's a right answer, checking for it with a script is cheaper and never changes its mind. Another AI is useful for judging tone or style, but then you have to test that AI too.
Doesn't a near-perfect score sound too good?
Yes. It's a score on my 65 questions, not a promise about yours, and it can move by one answer between runs. The point of the list is to notice when a change makes things worse.
How long does checking everything take?
About 20 seconds and two cents per version. Cheap enough to run after every change.