The short version
Building an agent used to be expert work. Now the software does it, and every capable person on your team can build one without asking anyone. Kim's position is not that they shouldn't. It's that backing your people means equipping them with a process, not just handing over the capability and walking away.
Where you point it matters more than what it can do
There's an appealing idea that you can point an agent at your CRM or ERP and let it tidy the data, fix the workflows and do the background work you'd normally pay a developer for. Kim calls that one of the riskiest things you could do. It's your business data and your IP being handed over.
- Prompt injection, where a hidden instruction sits inside your own data
- The wrong browser extension quietly getting connected
- Guardrails set too loose for the access granted, and data you can't get back
- The system your whole firm runs on grinding to a halt behind the scenes
"Move fast and break things" is a fine motto for building a landing page. It isn't one for a client's payroll.
The reconciliation that ties out and is still wrong
An accountant hands a payroll reconciliation to an agent. Wages, PAYG and super all tie out. It looks done. A senior accountant looks at the same result and sees what the agent had no way of knowing: the client took on four casuals last quarter, or there was a back-payment after a Fair Work review.
The numbers can tie out perfectly and still be wrong. They can also fail to tie out for a reason that's entirely correct. The agent can match figures. It can't know the client hired three people in March, and "the AI did it" isn't a defence when your name is on the work.
An AI workflow is a new process, so it needs a sign-off
Firms have spent years building standard operating procedures, and the whole point of an SOP is to guarantee a result. When someone builds an AI workflow to do that same job, they're building a brand-new process to reach the result the SOP was written to guarantee. So the person who owns that result has to sign off the process, the same way they own the pen on the procedure.
- What happens when this breaks, and how would we even know?
- What happens when the data going in is wrong?
- Design workflows together, so the people who own the result shape how it's built
- Build a coach for juniors rather than a replacement, and keep a senior auditing the output
- Don't let AI mark its own homework. A second agent checking the first still needs a human drawing the rails
It makes no difference whether the agent came from an expensive outside expert or a clever team member in-house. Same decision-making rules.
How AI Collab thinks about it
The line that sticks is borrowed from James Wedmore: don't build whipped cream on garbage. Can you trust the data underneath it, can you trust the process, and are you spending more time reworking the thing than you'd spend doing the job yourself? That's the governance conversation, and it's what the AI for Teams Toolkit and our governance work are built to settle.
The episode closes on the risk Kim cares about most: value accruing to individuals rather than to the business. Ep18 picks that thread up.
Frequently asked questions
Should we let our team build their own AI agents?
Why can an AI reconciliation tie out perfectly and still be wrong?
Who should sign off an AI workflow in a professional services firm?
Want to turn this into a practical AI plan?
AI Collab helps established Australian businesses move from interest to implementation: strategy, team capability, governance and custom builds that fit the way the business actually works.