Invoice Recognition Benchmark Comparison
CRS local tests · 200 invoices · 15 vendors, plus a 25-scan independent check
End-to-end structured recognition
| Pipeline | Vendor | Total | Invoice # | Date | Job | Overall |
|---|---|---|---|---|---|---|
| Gemma v0baseline | 96.0 | 94.0 | 74.0 | 97.0 | 46* | 83.4 |
| Gemma v1resolver + Tulsa rules | 95.5 | 93.5 | 88.5 | 96.5 | 41.5 | 85.4 |
| Gemma v2current proven stack | 97.0 | 93.5 | 91.5 | 96.5 | 44.2 | 86.8 |
| Gemma 12B A/B | 92.5 | 86.5 | 84.5 | 95.0 | 63.3 | 85.5 |
| Gemma E4B A/B | 93.5 | 72.5 | 86.0 | 97.5 | 24.5 | 77.6 |
| Xberg replacementfull flattened text | 99.5 | 76.5 | 80.5 | 92.0 | 49.7 | 81.3 |
| Xberg hybridtargeted evidence repairs | 100 | 94.5 | 95.0 | 96.5 | 44.2 | 88.4 |
Percent correct. Strict total shown; corrected total is 96.0% for v2 and 97.0% for the hybrid when valid pre-tax subtotal matches are accepted. *Original v0 journal reported job as 40/46%.
Raw field evidence and speed
| Extractor | Invoice # | Amount | Vendor | All three | Seconds/document |
|---|---|---|---|---|---|
| Tesseract | 82.0 | 88.5 | 91.0 | 64.5 | — |
| Xbergdigital PDFs | 93.0 | 95.0 | 87.0 | 85.0 | 0.013 |
| Suryalocal GPU VLM OCR | 95.5 | 97.5 | 91.0 | 84.0 | 15.27 |
Independent image-only scan check
| Extractor | Invoice # | Amount | Vendor | All three | Verdict |
|---|---|---|---|---|---|
| Tesseract | 22/25 | 25/25 | 25/25 | 22/25 | Works |
| Xberg default | 0/25 | 0/25 | 0/25 | 0/25 | No OCR |
Decision: retain Tesseract for scans, use Xberg as a fast corroboration layer on digital PDFs, and reserve Surya for difficult layout-heavy fallbacks. The targeted Xberg hybrid is the strongest measured end-to-end pipeline.