A good first AI agent pilot for a small business takes six to eight weeks, covers one repetitive low-risk task, runs on a separate account with narrow permissions, starts with a person approving every action, and is judged against numbers you measured before it began. At the end you decide to keep it, change it or stop it, based on time saved, errors caught and the effort of supervision, not on how impressive the demo looked.
This guide turns that into a concrete plan: what a pilot should prove, a step-by-step setup, a week-by-week timeline, a worked example from a distributor, the measures that matter, the mistakes that sink most pilots, questions to put to vendors, and the points where you need outside expertise.
What should a pilot prove, and what can it not prove?
A pilot answers a narrow question: on this specific task, with our data and our people, does an agent save meaningful time without introducing unacceptable errors, and can we supervise it at a reasonable cost? It is not a test of whether “AI works” in general, and it should not try to transform the company.
It should also tell you how the agent behaves when things go wrong: missing data, unusual inputs, changes in a supplier’s format. Those moments reveal more than smooth runs do. A pilot that only records successes has not tested much.
What a pilot cannot prove is how the agent will do on different tasks, at much larger volumes, or after the provider updates its models. Treat a successful pilot as permission to continue carefully, not as a guarantee.
Choosing the task is half the work. The criteria are covered in detail in our guide on which tasks to give an AI agent first: repetitive, rule-based, reachable data, easy to check, easy to undo. For a pilot, add one more: the person who does the task today must be willing to take part, because their knowledge is what makes the evaluation honest.
Finally, set the budget in advance, including subscription fees, any setup help and the hours staff will spend. Knowing the ceiling stops a pilot from quietly turning into an open-ended project.
Setting up the pilot, step by step
Each step takes hours, not weeks. Skipping one usually costs more time later.
- Write down the task and its owner. One paragraph describing the task, the expected result and the named person responsible for the pilot.
- Measure the baseline. For two weeks before the pilot, record how long the task takes, how many items are processed, and how many errors occur. Without this, any claim of improvement is a guess.
- Choose the tool. Prefer a product that offers narrow permissions, approval steps and readable logs. Check data terms with whoever handles legal or IT matters.
- Create a separate account with minimal access. Read-only where possible, specific folders or mailboxes, drafts instead of sends, spending limits. Never share an administrator password.
- Write the brief. Goal, sources, what not to do, how to handle unclear cases, what the report must contain.
- Test on copies. Run the agent on last month’s data or a copy of the files first, and compare its output with what your staff actually did.
- Go live with full approval. For the first weeks, every action that changes anything waits for a person.
- Review weekly and adjust. Use a short routine, as described in our guide on supervising AI agents with checkpoints and logs, and record every change to the brief or permissions.
Write each decision down. At the end of the pilot, that record is what lets you explain the outcome to colleagues or investors.
A week-by-week timeline
Adjust the length to your task, but keep the phases.
| Weeks | Phase | What happens | Exit condition |
|---|---|---|---|
| Before start | Baseline | Measure time, volume and errors of the manual task | Two weeks of reliable numbers |
| 1 | Setup and dry run | Create account, set permissions, write brief, test on copied data | Agent output matches staff output on test data |
| 2–3 | Live with full approval | Agent works on real items; a person approves every change | Approval rejections are rare and understood |
| 4–5 | Selective approval | Approvals only on sensitive actions; weekly sample review | Sample error rate at or below baseline |
| 6–7 | Steady state | Normal supervision routine; track time and cost | Stable results for two consecutive weeks |
| 8 | Decision | Compare with baseline; decide keep, change or stop | Written decision with reasons |
If a phase does not meet its exit condition, extend it or step back a phase. Moving forward on schedule when the evidence says otherwise is how pilots end in incidents.
A worked example: order entry at a distributor
A building materials distributor in Dong Nai receives around a few dozen orders a day by email from regular contractors, often as photos of handwritten lists or short messages. Two staff members type them into the sales system. Errors in quantities or product codes cause wrong deliveries several times a month.
The pilot task was narrow: read incoming order emails to one shared address, extract customer, items and quantities, match items to the product list, and create draft orders in a staging sheet. Staff would review each draft and enter it into the real system. The agent had read access to that mailbox and write access to one sheet only.
The baseline showed that typing took most of a morning, with a handful of errors a month. In the first two weeks, the agent got the structure right but often mismatched product names that contractors abbreviated in local ways. The team added a short list of common abbreviations to the brief, and match quality improved noticeably.
Handwritten photos remained a weak spot. The team changed the brief so the agent flagged any order from a photo with low readability for a person to enter manually, instead of guessing. That cut errors at the cost of handling some orders by hand.
At week eight, typing time had dropped substantially, errors reaching customers were lower than the baseline, and review took a few minutes per batch. The company kept the agent, still with staff confirming every order, and decided against letting it write directly into the sales system until it had a longer track record.
Which numbers tell you the pilot worked?
Total time per item, including review. Compare with the baseline. Drafting time alone overstates the gain.
Errors that reached customers or the real system. This is the number that matters most. It should be no worse than before.
Errors caught at review. High numbers early are normal; they should fall as instructions improve.
Share of items handled without human correction. Shows where the agent is reliable and where it needs help.
Items flagged as unclear. A healthy agent flags genuinely ambiguous cases rather than guessing.
Cost per item. Subscription, usage and supervision time divided by volume.
Staff opinion. Ask the people involved whether the work is easier, harder or just different. Their view often predicts long-term success.
Incidents. Any action outside the brief, any data exposure, any customer complaint linked to the agent.
Mistakes that sink most pilots
Picking the hardest process first. Complex, exception-heavy tasks fail in confusing ways and kill confidence early.
No baseline. Without before-numbers, success is a matter of opinion and often of enthusiasm.
Connecting the owner’s personal accounts. Fast to set up, impossible to contain, and mixes the agent’s actions with a person’s.
Relaxing approvals on a schedule rather than on evidence. “It is week four, so we stop approving” ignores what the logs say.
Leaving staff out. People who do the task know its exceptions. Excluding them produces a worse agent and worried colleagues.
Changing many things at once. If you change the brief, the tool and the task in the same week, you cannot tell what helped.
Ignoring supervision cost. An agent that saves two hours but needs two hours of checking has saved nothing.
Declaring victory too early. Two good weeks are encouraging, not conclusive. Watch for drift after the pilot ends.
Questions to ask an AI agent vendor
What permissions does the product need, and can they be narrowed? Look for options to limit to specific folders, mailboxes or actions.
Which actions can require approval? You should be able to choose, not just accept the vendor’s defaults.
What is logged, and can we export the logs? Logs should show sources read, actions taken and approvals, in a form you can keep.
How is our data used and where is it stored? Ask whether data is used for training and how long it is retained. Get the answer in writing.
How does the product handle untrusted content? Emails and web pages can contain hidden instructions. Ask what protections exist.
How do we revoke access instantly? There should be a clear, documented way to stop the agent and disconnect it.
What happens when the underlying model is updated? Ask whether you are notified and whether behaviour can change without warning.
What support is available during the pilot? Small businesses benefit from a named contact who answers practical questions quickly.
When do you need outside expertise?
If the agent will touch personal data of customers or employees, ask a lawyer how current rules on personal data protection apply, including consent, storage location and what happens if data leaks. Rules in Vietnam on these topics have been developing, so rely on current advice rather than on old articles.
If the agent will connect to core systems such as accounting, sales or banking platforms, involve an IT professional to set up accounts, permissions and logging correctly. Misconfigured access is the most common way a small pilot becomes a large incident.
If the agent will affect tax documents, invoices or financial reporting, have your accountant review the process. A tidy-looking spreadsheet can still violate a requirement the agent knows nothing about.
And if anyone promises that an agent will replace staff or pay for itself within a fixed time, ask for evidence from businesses like yours, and calculate the numbers yourself. Honest vendors talk about pilots, limits and supervision; pressure to commit quickly is a reason to slow down.
After the pilot: growing carefully
A successful pilot creates pressure to expand quickly: more tasks, more departments, more autonomy. Resist doing all three at once. The safest order is usually to keep the same task and widen its scope a little, for example from one mailbox to two, before adding a new kind of task.
Each new task deserves its own short pilot, even if shorter than the first. The lessons transfer, but the exceptions do not. An agent that handles orders well may struggle with complaints, where tone and judgement matter more than extraction.
Keep the paperwork light but real. Every agent in use should have a one-line entry in a shared list: what it does, who owns it, what it can access, when it was last reviewed. When a staff member leaves or a tool changes, that list is what tells you what to check.
Finally, watch the quiet costs. As agents multiply, so do subscriptions, permissions and review duties. A quarterly look at all agents together, asking which ones still earn their place, prevents a collection of half-supervised tools from building up in the background.
Frequently asked questions
How long should an AI agent pilot last?
Six to eight weeks suits most simple tasks, plus two weeks of baseline measurement beforehand. Shorter pilots rarely see enough unusual cases; longer ones risk losing focus.
How much does a pilot cost?
It depends on the tool’s pricing, the volume of work and how much setup help you need. The largest hidden cost is usually staff time for review, so include it when you budget.
Do we need a technical person on the team?
For ready-made tools and simple tasks, a careful non-technical owner can run a pilot. For connections to core systems or custom setups, involve an IT professional, at least for permissions and logging.
What if the pilot fails?
A clear “no” is a useful result: you learned the task or the tool is not ready, at limited cost. Record why, so a future attempt starts from that knowledge.
Should we tell customers we are using an AI agent?
If the agent communicates with customers or handles their data, being open is wise and may be required by platform rules or law. Ask a lawyer about disclosure duties that apply to your business.
Can we run two pilots at the same time?
Only if they involve different teams and tasks, and each has its own owner and baseline. For a first experience, one pilot at a time gives clearer lessons.
