Cost and Speed
Where the time and money of one AI answer go. Most of the money went to a helper service, not the AI itself, and one small change cut the bill by a third without more wrong answers.
Try it
Ask the company's help bot a question. It answers twice: the careful way, and with one shortcut. See where the time goes and what each answer costs.
- Starts typing
- —
- 1,000 answers cost
- —
- Text it reads
- —
The answer appears here.
- Starts typing
- —
- 1,000 answers cost
- —
- Text it reads
- —
The answer appears here.
- Turn the question into numbers
- Search the documents
- Second opinion
- AI starts answering
- AI finishes
Every shortcut, measured
Every question was asked through every setup, three times. Time is the typical wait and the slow one-in-twenty. Cost is per thousand answers.
| Setup | Typical wait | Slow one | Starts typing | 1,000 answers | Search services | AI answer | Right answers |
|---|---|---|---|---|---|---|---|
| The careful setupThe setup from the RAG chapters: two kinds of search, then a second AI picks the best five passages. | 1.73 s | 2.52 s | 1.22 s | $0.94 | $0.64 | $0.30 | 193/195 |
| A shorter list for the second opinionThe second AI only re-sorts the ten most promising passages, not every one the search found. | 1.64 s | 2.28 s | 1.14 s | $0.58 | $0.26 | $0.33 | 193/195 |
| Skip the second opinionUse the search results as they come, without a second AI re-sorting them. | 1.38 s | 2.34 s | 0.92 s | $0.41 | $0.00 | $0.41 | 187/195 |
| Fewer pages to readGive the AI three passages to read instead of five. | 1.67 s | 2.52 s | 1.18 s | $0.84 | $0.64 | $0.20 | 192/195 |
| Plain word searchFind passages by matching words, like a basic site search. No extra AI services. | 1.15 s | 3.68 s | 0.70 s | $0.68 | $0.00 | $0.68 | 172/195 |
| Remember repeat questionsIf someone asked the same thing before, skip one step and reuse the earlier lookup. | 1.51 s | 2.16 s | 1.02 s | $0.94 | $0.64 | $0.30 | 191/195 |
Same instructions, every step
An AI assistant that takes several steps re-sends the same instructions every step. A cache lets the provider charge a tenth for the part it has already seen.
- Re-reading, per step
- $0.22
- One whole task
- $1.79
- Time per step
- 2.34 s
- Steps per task
- 4.7
- Re-reading, per step
- $0.07
- One whole task
- $1.08
- Time per step
- 2.27 s
- Steps per task
- 4.8
With the cache, re-reading the instructions got 67% cheaper per step, and a whole task 39% cheaper. The AI's writing isn't cached, which is why the saving on a whole task is smaller.
Costs are per thousand, at the provider's list price.
What actually mattered
The second opinion cost more than the answer
68% of what an answer costs goes to the second AI that re-sorts the search results, not to the AI that writes the answer. Giving it a shorter list cut the cost by 38% with 193 of 195 answers right, against 193 of 195.
The AI is slow at writing, not at reading
Giving the AI four times more to read barely changed how long it took. Shorter reading saves money, not time. What makes it feel faster is showing the answer as it's written: the first words after 1.22 s, the whole answer after 1.73 s.
The quickest shortcut gave the most wrong answers
Plain word search was the fastest and needs no extra AI service, but got 21 more answers wrong out of 195. And it wasn't cheaper: the passages it finds are longer, so the AI reads more.
The code
What changed, in one sentence. The code itself is one click away.
Before, every passage the search found went to the second AI for re-sorting, and it charges for every word of every passage. Now only the ten most promising go.
Show the code anyway
const fused = rrf(keywordTop20, vectorTop20); // up to 40 chunksconst order = await voyage.rerank({ model: "rerank-3", query: question, documents: fused.map((c) => c.text), // all of them top_k: 5,});const fused = rrf(keywordTop20, vectorTop20);const order = await voyage.rerank({ model: "rerank-3", query: question, documents: fused.slice(0, 10).map((c) => c.text), top_k: 5,});Still curious?
A few more questions people ask. Open what you need.
Where were these timed?
From my own computer, using the same services the site uses. A real server is closer to them, so its times are lower. What matters is the comparison. The demo above runs on the real server.
Do a couple of extra wrong answers matter?
Not by themselves. The same setup asked the same questions three times scores a little differently each time. Only differences bigger than that count.
Why not use the cache for the help bot too?
The help bot's fixed instructions are too short to be cached, and the passages it reads change with every question. The assistant re-sends the same long instructions every step, which is exactly what the cache is for.
Isn't the search service free?
The first part of the usage is free, roughly 16 thousand answers at this size. The prices above are what it costs after that.
Why does the second opinion cost so much?
It's charged for every word of every passage it looks at, and the search hands it dozens of long passages. Most of the good ones are already near the top, so a shorter list loses little.
Was the shorter list tested the same way?
After the others, once I saw where the money went. It was run next to a fresh copy of the careful setup, so the times are compared fairly.