Evaluation criteria
How We Evaluate Bookkeeping Automation Workflows
The tests we run before a tool earns a place in a guide — and how you can run the same ones yourself.
Why this page exists
A feature list tells you what a tool claims. A workflow test tells you what it does.
There is no shortage of tools claiming to automate bookkeeping. The harder question is whether a tool holds up when the workflow gets messy, because that is when automation either helps or creates new problems. A clean demo proves very little; a duplicate invoice at month end proves a lot.
This page explains the criteria used to assess automation workflows on this site, so you can see how the guides are built and apply the same tests to any tool you are considering. We weigh integration quality, audit trails, exception handling, control risk, maintenance cost, volume behavior, reporting, business-stage fit, and data handling — in that order of attention, because that is roughly the order in which weak workflows fall apart.
Nothing here ranks tools against each other. The point is fit: whether a tool's behavior matches the way your books actually move, and whether it keeps holding up once the team, the vendor list, and the transaction count change.
What the assessment covers
- What syncs, how often, and what happens when a sync fails mid-month.
- Who can see and export the audit trail, and who can quietly edit it.
- How exceptions — duplicates, partial matches, refunds, renamed vendors — get resolved.
- Whether automation strengthens or erodes the controls you already rely on.
- Setup, training, and upkeep — the cost that decides whether automation sticks.
The order we work in
Nine checks, run in sequence
Each check assumes the one before it passed. If integration drops fields, the audit trail is already untrustworthy — so we do not score further down the list until the earlier question is answered.
Integration quality comes first
What actually syncs — transactions, attachments, coding, vendor records — and how often.
Audit trail and who can see it
Who entered a transaction, who approved it, what changed — and whether that record can leave the platform.
Exception handling under real conditions
Duplicate invoices, partial matches, refunds, vendors that rename themselves — and where the exception lands.
Control risk in automation
Whether separation of duties survives auto-approval thresholds and hands-off coding.
Setup, maintenance, volume, reporting, and stage fit
The slower-moving criteria — the ones that decide whether the automation is still running in a year.
Integration quality comes first
A tool that does not sync cleanly with your accounting system is not automating anything. It is adding a step. The first thing we look at is what actually syncs: transactions, attachments, coding, vendor records, and how often.
A sync that drops fields or creates duplicates means someone has to reconcile the two systems, which defeats the purpose. Reliable integration with a widely used platform like QuickBooks Online matters more than a long feature list, because the sync is what carries the data. We also watch what happens when the connection fails: does the tool queue the work and retry, or does it silently stop and leave a gap nobody notices until close?
Where a tool integrates with more than one system, we check that the two connections agree. Coding applied in one place should arrive in the other, and attachments should follow the transaction rather than living in a separate folder that a reviewer has to hunt down.
Audit trail and who can see it
An audit trail should show who did what and when: who entered a transaction, who approved it, what changed. It should also be exportable, because the record needs to be shareable with an accountant, an auditor, or a business partner.
Tools that keep the trail locked inside the platform create a dependency. Tools that let you pull it out give you control. For a business with more than one person touching the books, the audit trail is a control, not a nice-to-have.
We also check retention. If the trail is readable for six months and then archived somewhere awkward, the record is technically present but practically useless during a review that happens next year.
Where tools separate
Exception handling under real conditions
The happy path is easy. The exceptions are where tools differentiate. Does the tool handle a duplicate invoice, a partial match, a refund, a vendor that changes its name? Can you split a transaction across categories? Can you resolve an exception in place, or does it sit in a queue?
We look at how the tool behaves when the input is messy, because real businesses produce messy input. A tool that only works on clean data is not a tool, it is a demo. The useful question is not whether the queue exists, but how long a real exception sits in it before someone notices.
Control risk in automation
Automation can weaken controls if it is set up carelessly. Auto-approving everything under a high threshold removes the review that used to catch errors. Letting the system code without spot-checking means mistakes compound. Skipping the audit trail leaves no record if a payment is questioned.
We assess whether the tool makes it easier to maintain separation of duties, verify changes, and keep a record. A tool that makes controls harder to maintain is a risk, regardless of how much time it saves. That trade-off deserves to be written down before the tool goes live, not discovered during an audit.
The practical test is simple. Ask who approves a payment above your review threshold, then ask what the tool does if that person is on holiday. If the honest answer is that the threshold gets raised, the control is already gone.
Cost, capacity, and growth
Two costs decide whether automation survives its first year
The setup and maintenance cost
Automation has a cost beyond the subscription. Someone has to configure it, train the team, and maintain it as vendors and accounts change. We look at how much of that is one-time versus ongoing, and whether the tool needs constant attention to stay useful.
A tool that takes a week to set up and runs itself is a different proposition from one that needs monthly tuning. Neither is automatically wrong — a high-volume business with unusual coding may reasonably accept more upkeep — but the maintenance cost is often what determines whether the automation sticks.
We note who inside the business owns that maintenance. If the answer is "the person who set it up," and that person is a contractor, the workflow has a single point of failure worth planning around.
How tools handle volume
A workflow that works for fifty transactions a month may not work for five hundred. We look at how the tool behaves as volume grows: whether capture keeps up, whether approvals scale, whether the reconciliation stays manageable.
Volume is also a proxy for complexity, because more transactions mean more exceptions. A tool that is fine at low volume but breaks at scale is a tool you will outgrow, and it is worth knowing that before you commit rather than during your busiest month.
When a vendor will not share how pricing or processing changes with volume, we treat that as a gap in the assessment rather than a neutral detail. It usually means the answer depends on negotiation, and it should be confirmed in writing.
Reporting and visibility
Reports should answer the questions a business actually has. We check whether the tool produces them from current data, not from last week's export.
-
What did we spend
Whether spend reporting reads straight from coded transactions, or needs a manual pass to become trustworthy.
-
What do we owe
Whether payables and open vendor balances reflect entries still sitting in an approval queue.
-
Where do we stand against budget
Whether reports can be filtered by entity or class, so a multi-entity business is not reading a blended total.
-
Can we hand it to someone else
Whether a report can be shared with an accountant or lender without screenshots and reformatting.
A tool that produces reports nobody reads is not adding value. A tool that produces the report you need, when you need it, is.
Fit for the business stage
A solopreneur, a service business, and a multi-entity startup have different needs. We assess tools against the stage they are built for, not against an absolute standard. Judging a tool outside its intended stage produces a misleading result, so we note the stage as part of the assessment.
Security and data handling
Financial data is sensitive, and the tool needs to protect it. We look at access controls, whether the platform supports role-based permissions, and how data is stored.
We also look at what happens when someone leaves the company, because access needs to be revoked cleanly. Security is not a feature you check once. It is a property of how the tool is set up and maintained, and it is part of the assessment.
What we do not score
We do not rank tools by popularity or by how many features they list. We do not treat a tool as better because it is newer, or worse because it is established. We do not score on price, because price depends on volume, seats, and what you already use. And we do not assume a tool is right for you because it is right for someone else.
The assessment is about fit, and fit depends on the workflow you are running. Two businesses in the same industry can reasonably reach opposite conclusions about the same tool, and both can be correct. What we can do is show the tests clearly enough that you can reach your own answer with evidence behind it.
When a tool changes its behavior — a new pricing tier, a changed sync, a different exception flow — we revisit the assessment rather than leaving an old conclusion in place. If we cannot confirm how something behaves now, we say so instead of carrying forward a stale verdict.
How to apply these criteria yourself
Take the criteria and test them against your own workflow. Run a month of data through a trial and see how the tool handles your exceptions, not the demo's.
The answers tell you more than any feature comparison, because they describe how the tool behaves when things go wrong.
How do I test exception handling before I commit? Open Close
Feed the trial a month you already know was messy. Include a duplicate invoice, a refund, and a vendor whose name changed mid-year, then watch where each one lands. If the tool resolves them in place and leaves a visible record, that is a good sign. If they disappear into a queue with no owner and no notification, you have learned something the feature list would never tell you.
What should I ask a vendor about the audit trail? Open Close
Ask whether the trail exports as a file you keep, or only displays inside the platform. Ask how long it is retained and what a reviewer sees a year later. Then ask who inside the tool can edit an entry after approval. The audit trail review matters most for businesses where more than one person touches the books, because that is where the record stops being paperwork and starts being a control.
How do I check that the integration really holds? Open Close
Reconcile a single week by hand and compare it against what the tool produced. Check whether attachments followed the transaction, whether coding arrived where you expected, and whether vendor records stayed matched. Then deliberately break the connection — disconnect it and reconnect — and see whether the tool catches up or leaves a silent gap. Integration assessment is less about the feature list and more about what survives a failure.
Where does control risk usually show up first? Open Close
Usually at the approval threshold. Automation makes it easy to raise a limit so fewer things need review, and the change looks like a small configuration tweak. In practice it removes the step that used to catch errors. Control risk in automation tends to appear as convenience first and as a finding later, which is why we ask who approves what above the threshold and what happens when that person is unavailable.
Why do you not score on price? Open Close
Because the number on a pricing page is not the number you pay. Volume tiers, seat counts, existing subscriptions, and the staff time spent maintaining the setup all change the real cost. A cheap tool that needs monthly attention can cost more than an expensive one that runs quietly. We describe what drives cost in each case and leave the arithmetic to you, since only you know your volumes and rates.
Run the same tests on the tool you are considering
Start with a month of your own messy data, check whether the audit trail exports, and ask the vendor what happens when a sync fails. If the answers are unclear, that is worth knowing before you sign. If you want a second opinion on a specific workflow, our office is open Monday to Friday, 9:00 AM – 6:00 PM, and you can reach us at [email protected] or +1-212-550-6882.