1 · Quality: which model finds the facts
24 synthetic notes with 68 known facts, same prompt for every model, answers judged blind (the judge did not know which model wrote what). Includes one empty note (should give no facts) and one prompt-injection note (should be refused). Measured 30 Sep 2026.
| Model | Runs on | Facts found | Wrong | Empty note skipped | Injection refused | Median s | p95 s | Licence | Our use |
|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet (reference) | Cloud | 100 % | 0 | yes | yes | — | — | Commercial | Yardstick only — never sees company data |
| gpt-oss-120b | Groq cloud | 99 % | 0 | yes | yes | 0.6 | 1.4 | Apache-2.0 | First choice for synthetic test accounts |
| Qwen 27B | Groq cloud | 97 % | 0 | yes | yes | 0.3 | 0.5 | Apache-2.0 | Candidate |
| gpt-oss-20b | Groq cloud | 85 % | 1 | yes | yes | 0.4 | 0.6 | Apache-2.0 | Not chosen (2 failed calls) |
| Qwen3 4B Instruct 2507 | Our server (CPU) | 94 % | 1 | yes | yes | 4.5 | 6.1 | Apache-2.0 | Production default for every account |
| Granite 3.3 2B | Our server (CPU) | 96 % | 5 | no | no | 2.9 | 4.6 | Apache-2.0 | Not chosen (more wrong facts) |
| Gemma 3n E4B | Our server (CPU) | 94 % | 4 | no | yes | 7.5 | 9.9 | Gemma terms | Not chosen (slower, licence) |
| Gemma 3 4B | Our server (CPU) | 90 % | 1 | yes | no | 5.6 | 7.0 | Gemma terms | Earlier default (licence) |
| Llama 3.2 3B | Our server (CPU) | 87 % | 3 | yes | yes | 3.7 | 15.4 | Llama licence | Not chosen |
| Phi-4 mini | Our server (CPU) | 85 % | 0 | yes | yes | 3.8 | 5.7 | MIT | Not chosen (misses facts) |
| Qwen3 4B (thinking) | Our server (CPU) | 82 % | 6 | yes | yes | 5.4 | 8.4 | Apache-2.0 | Not chosen |
| Gemma 3 1B | Our server (CPU) | 82 % | 5 | no | no | 1.9 | 2.6 | Gemma terms | Not chosen (too many errors) |
Highlighted rows are in use. Company data is only ever processed by the local model on our server; the cloud model (Groq) only sees the synthetic test account until the company approves a provider. If the cloud is down or slow, the test account falls back to the local model automatically.
2 · Timing by text size
The same synthetic notes joined into longer texts (1, 4, 10 and 22 notes, then 22 notes three times), each run 3 times per model; the table shows the median. The orange line is real production work: about 506 extraction jobs from bulk imports on 30 Sep – 2 Oct, measured as the time between one job and the next (it includes saving and indexing, so it is a little higher than the model alone). Measured 2 Oct 2026.
Local Qwen3 4B — testGroq gpt-oss-120bGroq gpt-oss-20bLocal Qwen3 4B — production
| Text | Local Qwen3 4B — test | Groq gpt-oss-120b | Groq gpt-oss-20b |
|---|---|---|---|
| 1 note34 words | 6.4 s4 of 4 facts · 3/3 ok | 0.9 s4 of 4 facts · 3/3 ok | 0.4 s4 of 4 facts · 3/3 ok |
| 4 notes107 words | 13.7 s9 of 14 facts · 3/3 ok | 1.4 s11 of 14 facts · 3/3 ok | 0.6 s5 of 14 facts · 3/3 ok |
| 10 notes246 words | 10.8 s5 of 34 facts · 3/3 ok | 2.9 s19 of 34 facts · 3/3 ok | 0.9 s20 of 34 facts · 3/3 ok |
| 22 notes508 words | 12.6 s5 of 68 facts · 3/3 ok | 4.8 s40 of 68 facts · 3/3 ok | 1.4 s44 of 68 facts · 3/3 ok |
| 22 notes x31,656 words | 94.9 s50 of 68 facts · 3/3 ok | failed | 1.9 s46.5 of 68 facts · 2/3 ok |
Production: local model on real work, by text size
| Text size | Jobs | Avg words | Typical (p50) | Slow (p90) | Facts per text |
|---|---|---|---|---|---|
| ≤ 30 words | 142 | 22 | 7.4 s | 13.8 s | 3.3 |
| 31–80 words | 206 | 52 | 12.2 s | 17.0 s | 5.1 |
| 81–200 words | 74 | 119 | 17.7 s | 30.7 s | 11.9 |
| 201–1,000 words | 74 | 617 | 50.2 s | 111.8 s | 14.8 |
| > 1,000 words | 10 | 1,261 | 131.3 s | 164.7 s | 27.1 |
Long texts lose facts on the small model. "x of y facts" counts the facts returned against the facts known to be in the text (not judged one by one; for the tripled text the known facts repeat, so the y is the unique count). The local 4B model returns most facts for one or a few notes, but on long texts it keeps only a handful, and the time stops growing because it writes less. The large cloud model keeps far more. For long company texts, this argues for splitting them into parts before extraction or, in Phase 2, a larger approved model.
What this means. On our CPU the local model takes about 6–12 seconds for a short note and grows roughly in step with the text: a long Jira issue with many comments takes 1–2 minutes. Groq stays around one to a few seconds at every size. Because extraction runs in the background, nobody waits for it — a saved note is searchable at once, and its facts appear a little later.
3 · Search and sync timing by data size
Searching does not use the language model, so it stays fast as data grows.
| What | Data size | Typical (p50) | Slow (p95) | Quality / note |
|---|---|---|---|---|
| Search, live system (full request incl. sign-in and permission check) | 568 facts, 24 questions | 70 ms | 109 ms | right fact first 21/24, in top 5 22/24 · target 100 ms |
| Context pack for an AI app (live system) | 108 items, 2,000 tokens | 23 ms | 61 ms | target 150 ms |
| "Is my memory up to date?" check (live system, HTTP) | one person | 5.3 ms | 6.2 ms | target 20 ms |
| Search engine alone, one store per account (spike) | 6,068 facts in 6 scopes | 27 ms | 36 ms | first 15/21, top 5 19/21 · no leaks across clients |
| Search engine alone, one store per scope (spike) | 6,068 facts | 62 ms | 120 ms | first 18/21 |
| Same, with a re-ranking model on CPU (spike) | 6,068 facts | 469–2,740 ms | 694–3,842 ms | first 21/21 — too slow, switched off |
| Up-to-date check, database only (spike) | 220 people → 10,000 people | 0.05 → 0.77 ms | 0.05 → 0.83 ms | grows slowly with headcount |
| Company-wide rule change reaches everyone (spike) | 220 people → 10,000 people | 2 → 55 ms | — | every person's memory marked for refresh |
| Rebuild the search index from our records (spike) | 1,017 facts | 23 s in total | same top-5 results before and after | |
| Loading facts into the search index (spike) | 6,068 facts | 13–44 facts per second | no language model needed | |
6 · On a GPU server: Hetzner GEX45 (estimate)
The GEX45 has one NVIDIA RTX PRO 4000 Blackwell SFF Edition (24 GB GDDR7, 432 GB/s, 770 AI TOPS, 70 W), an Intel Core i5-13500 (14 cores), 64 GB RAM and 2 × 512 GB NVMe, in Helsinki, for €214 a month plus €209 setup, with no minimum term. Nothing in this section is measured yet. It scales a published benchmark of the larger RTX PRO 4000 to this card and applies it to the same text sizes as section 2.
| Model on the GEX45 GPU | 34 words | 107 words | 246 words | 508 words | 1,656 words | Our quality test |
|---|---|---|---|---|---|---|
| Qwen3 4B Instruct (today's model)2.5 GB on the GPU · ~110 tok/s | 1.2 s | 2.1 s | 3.8 s | 7.0 s | 21.2 s | 94 %, 1 wrong (CPU run) |
| Qwen3 14B9.0 GB on the GPU · ~37 tok/s | 2.9 s | 5.6 s | 10.7 s | 20.3 s | 62.2 s | not tested yet |
| Qwen3 30B-A3B (MoE, 3B active)18.6 GB on the GPU · ~85 tok/s | 1.6 s | 2.8 s | 5.0 s | 9.3 s | 27.9 s | not tested yet |
| gpt-oss-20b (MoE, 3.6B active)13 GB on the GPU · ~80 tok/s | 3.5 s | 4.8 s | 7.1 s | 11.6 s | 31.2 s | 85 %, 1 wrong (Groq run) |
| Qwen 27B (dense)16.5 GB on the GPU · ~20 tok/s | 4.9 s | 9.8 s | 19.1 s | 36.7 s | 113.7 s | 97 %, 0 wrong (Groq run) |
| Measured today: Qwen3 4B on the CPUshared server, 8 threads | 6.4 s | 13.7 s | 10.8 s | 12.6 s | 94.9 s | 94 %, 1 wrong |
How the estimate is made. Writing speed is limited by memory bandwidth. A published Ollama benchmark of the full-size RTX PRO 4000 Blackwell (672 GB/s) measured 98.9 tokens/s for an 8B model and 57.8 tokens/s for a 14B model (Q4_K_M, June 2026); the GEX45's SFF card has 432 GB/s, so speeds are taken as 64 % of those. Time per text = reading the prompt and the text (650 + 1.35 × words tokens) + writing the facts (25 + 35 tokens per fact, one fact per 27 words as in our test set; gpt-oss adds about 150 reasoning tokens) + 0.3 s. Prompt-reading speeds are rough, but they are a small share of the time.
- Today's model, same quality, about 5× faster: about 1 s for a short note instead of 6.4 s. A 1,650-word text takes about 21 s because, unlike today's CPU run (which misses most facts in long texts), the estimate assumes every fact gets written out.
- Better models fit: Qwen 27B scored 97 % in our quality test (on Groq) and fits in 24 GB; on this card it would take about 5 s for a short note and about 2 minutes for a long one. Qwen3 14B and the mixture-of-experts models sit in between. The choice should be made by running section 1's quality test on the server itself.
- Long texts: splitting a long text into parts and extracting them in parallel on the GPU would bring a 1,650-word text to a few seconds with the smaller models.
- Throughput: the GPU can run several extractions at once, so a queue of imports clears several times faster than the per-text times suggest.
- To replace the estimate: once the server is rented, run the same quality test and size benchmark there (Ollama
--verbosefor speeds) and publish the measured table here.
4 · How we measured, and the limits
- Server: the shared company server; the local model is capped at 8 CPU threads and 10 GB memory so other services are not slowed. Timings vary with other load on the server.
- All test texts are synthetic. Real company text never went to a cloud model; production timings use only counts and times from the database, never the text.
- The quality judge was also a Claude model, so the reference's 100 % may be flattered.
- The live search numbers use a small data set (568 facts); the 6,068-fact numbers are from the earlier engine test without the permission layer.
- A third Groq model name (qwen3-32b) was not available on Groq on 2 Oct (HTTP 404), so it has no timing.
- On the largest text (about 1,650 words) Groq gpt-oss-120b failed all three tries (two HTTP 400 answers that did not fit the required format, one HTTP 429 free-tier rate limit) and gpt-oss-20b failed once (429). In the live system the test account then falls back to the local model automatically; a paid tier or splitting long texts would avoid this.
- Known gap: the job table records start and finish inside one database transaction, so per-job model time is not stored; the production figures above are measured from the gap between consecutive jobs. A fix is noted for later.
5 · What we chose
- Company data: Qwen3 4B Instruct on our own CPU (Apache-2.0 licence, 94 % found, 1 wrong, refuses injection). Slow but private and free.
- Synthetic test accounts: Groq gpt-oss-120b first (99 %, 0 wrong, about a second), local model as automatic fallback.
- Next: a GPU server (Hetzner GEX45, section 6) keeps data local at close to cloud speed and allows a stronger model; switching is a configuration change, not a code change.