If you’ve hoped a chatbot could take bookkeeping off your plate, a new benchmark suggests it isn’t ready to do the job alone.
Digits, a company that sells AI-powered accounting software, re-ran tests it first published in June and says general-purpose AI models “did not get any better at accounting” in the months since, despite a wave of new releases. The results were published Oct. 7.
How the Test Worked
Digits asked AI models to categorize 2,000 real transactions from four small businesses. The answers were graded against categories reviewed by US accountants applying standard accounting rules (GAAP). In all, the team tested 84 setups using models from OpenAI, Anthropic, Google and others.
What It Found
- On their own, the best models were right about 80% of the time. OpenAI’s GPT-6 Luna led at 79.5%, slightly below June’s best score of 80.7%.
- That’s roughly what a newcomer does. Outsourced human accountants seeing the books for the first time scored 79.1%.
- Context helped, at a cost. When models could look up a company’s past records, accuracy rose. The best agent, Anthropic’s Claude Fable 5.1, hit 88.9%, but still missed about one in nine transactions and took 74 seconds per transaction.
- Some models just didn’t answer. One left 20.7% of transactions uncategorized.
Digits says its own system, which is trained on each business’s ledger, scored 97.8% and was far faster.
The Catch
This is a vendor testing its own product against competitors, and its own team ran the tests. The methodology is published in a white paper, but the results haven’t been independently verified. Treat the comparison with Digits’ product with caution.
The broader point is still useful: a general chatbot that doesn’t know your business makes the kind of mistakes a stranger would.
What It Means for Small Business Owners
- Don’t let a general chatbot categorize your books unsupervised. A 1-in-5 error rate adds up fast at tax time.
- If you use AI bookkeeping tools, review flagged and unusual transactions yourself, or have your accountant do it.
- Ask vendors how their tools learn from your own records, and how you can correct mistakes.
More on Contoh
- Three in Four Small Business Owners Now Use AI – Here’s What They Use It For
- Cash Flow 101: Simple Habits That Keep a Small Business Afloat
- The Small Business Guide to AI Tools: Where to Start, What to Avoid