In partnership with
Hi, there!
Welcome to the 15th edition of Work in Beta.
In this edition, we tackle the question we get asked more than any other right now - what is an AI agent? - and give you a 3-part test that tells you, in 30 seconds, whether what you're looking at is actually one.
Also, if you are looking to build your individual or organisational system with AI, scroll down to the bottom of the newsletter to know more and connect with us.
We're planning a Claude Code workshop soon. Hands-on, practical, built for people who want to actually work differently with AI, not just watch someone else do it. Drop your details here to get on the waitlist.
Let's dive in!
IF YOU ONLY HAVE 2 MINUTES
Image Credits: Nano Banana Pro / Work in Beta
THE ‘HOW TO’ PLAYBOOK
What Is an AI Agent? The Decision Line That Separates the Real Ones
We get asked this more than any other question right now: what is an AI agent?
In every client conversation, in every workshop we run, it comes up. People don't know what an agent actually is and they shouldn't feel stupid for not knowing. The market has made it nearly impossible to answer.
A Zapier flow that saves emails to Drive gets called an agent. A Slack-to-Gmail connector gets called an agent. Vendors are branding chatbots as agents and charging premium prices for it. Gartner estimates only about 130 of the thousands of vendors selling "AI agents" are actually building them (June 2025). The rest are doing what the industry now calls "agent-washing" - slapping the word on chatbots, automations, and workflows to ride the hype.
This isn't a vocabulary problem. When you can't tell an automation from an agent, you can't evaluate what you're paying for. You can't set the right expectations for what you build. You can't decide what you actually need.
Here's the one question that cuts through the noise and the 3-part test underneath it.
The Decision Line
The difference between an automation and an agent comes down to one question:
Who decides what happens next - the person who built it, or the system, at runtime?
Automation: every decision is made at build time. The person who wired it up decided: "when X happens, do Y." The system just executes the plan.
Agent: decisions are made at runtime. You give it a goal. It reads the situation, picks what to do, picks which tools to use, picks when to stop.
This is a similar distinction Anthropic draws in its Building Effective Agents guide: workflows orchestrate through predefined paths; agents dynamically direct their own process. The market uses the same word for both. They're not the same thing.
Everything in the test below flows from this one distinction.
The 3-Part Agent Test
Point this at anything someone calls an "agent." Run three checks. Miss any one → it's an automation in an agent costume.
Test 1: Goal, not recipe
Automation: you gave it the recipe. Step 1, step 2, step 3.
Agent: you gave it the goal. It figures out the steps.
Quick check: could you write down exactly what it will do, in order, before it runs? If yes, it's an automation.
Test 2: Tool choice
Automation: the tools it uses are hardwired. An email connector always opens email. A calendar connector always opens calendar.
Agent: it picks which tools to use, and in what order, based on what the goal needs.
Quick check: if you gave it a different request, would it reach for different tools on its own?
Test 3: Runtime judgment
Automation: breaks on unexpected input. Designed for one path, returns an error when the path doesn't apply.
Agent: improvises. Not perfectly, it can be wrong, but it attempts to reason through a new situation.
Quick check: throw it something it wasn't built for. Does it try, or does it refuse?
The scoring:
All 3 yes → agent
Mixed → hybrid (this is where most real products sit today)
All 3 no → automation
Neither is better than the other. They're different tools for different jobs. The point of the test isn't to rank them, it's to know which one you're looking at.
Run the Test on Things You Already Use
Thing | Goal, not recipe? | Tool choice? | Runtime judgment? | Verdict |
|---|---|---|---|---|
Zapier Zap (new email → save to Drive) | ❌ | ❌ | ❌ | Automation |
Slack ↔ Gmail connector | ❌ | ❌ | ❌ | Automation |
Outlook rules (even with conditions) | ❌ | ❌ | ❌ | Automation |
Customer service chatbot with scripted replies | ❌ | ❌ | ❌ | Chatbot - not even an automation |
Zapier flow that calls ChatGPT as one step | ❌ | ❌ | ❌ | Automation with an AI step (still not an agent) |
ChatGPT plain chat, no tools connected | ✅ | ❌ | ✅ | Hybrid; chatbot with a smart brain |
ChatGPT with browsing + memory + tools | ✅ | ✅ | ✅ | Agent |
Claude Project with tools connected | ✅ | ✅ | ✅ | Agent |
Notice the pattern in what fails. Three common near-misses show up everywhere: things that talk but don't act (chatbots), things that act but don't decide (automations), and things that use AI inside a fixed path (automations with an LLM step tucked in the middle). All three get called agents in the market. None of them are.
An agent needs all three parts - goal, tool choice, runtime judgment - working together. Miss any one and you have something useful, but it's not an agent.
The honest takeaway: you probably already use 1-2 actual agents. ChatGPT with tools turned on. Claude Projects with connectors. You also probably pay for things marketed as agents that are really automations with better branding. Now you can tell.
The Mistakes We See People Make
Mistake 1: Calling it an agent because it has AI in it. An AI-powered spell checker isn't an agent. A Zapier step that calls ChatGPT isn't an agent. AI alone isn't the test - runtime decision-making is. "There's an LLM in there somewhere" doesn't qualify.
Mistake 2: Believing the vendor. If the vendor can't tell you what their system decides at runtime, assume it's an automation. Ask the question directly: "What does it decide, in the moment, that we haven't pre-programmed?" The answer tells you more than any demo.
Mistake 3: Building an agent when an automation would've done the job. Agents are slower, costlier, and more failure-prone than automations. If the path is knowable - same trigger, same action, same output - use an automation. Save the agent for tasks where the next step actually depends on what the last step revealed.
Mistake 4: Expecting agent creativity with automation reliability. Automations are predictable. That's the feature. Agents improvise. That's also the feature. If you want both at once, you'll be frustrated with both.
Mistake 5: Waiting for "the perfect agent" instead of starting with what you have. ChatGPT, Claude, and Gemini are already agents by this test. The tools you're already paying for can do more than you think. The constraint isn't capability, it's knowing how to use them.
Do This Week
Tomorrow: Pick 3 tools you use regularly. Run each one through the 3-part test. How many are actually agents? You'll be surprised how many "agents" are really just automations with better marketing.
This week: Find one task where you're using an automation but actually need agent judgment, something where the right next step depends on context. Run it in ChatGPT or Claude for a week instead. See what changes.
Next week: Pick one "agent" product you're being pitched, or already paying for. Run the 3-part test. If it fails, ask the vendor what their system decides at runtime and watch how they answer.
Final Thought
Most of what's being sold as an AI agent is a workflow wearing an agent costume. That's not a conspiracy, it's a market doing what markets do when a word gets valuable faster than the definition does.
Agents aren't better than automations. They're different. One does what you tell it. The other figures out what to do.
Knowing which one you're looking at is the whole game.
WORK WITH US
The Other 95%
Knowing how to prompt well is roughly 5% of what it means to actually work with AI. The other 95% - context architecture, workflow compression, thinking behaviors, tool orchestration - is where your workday actually changes. Not "I got a better first draft." More like "I built a full client proposal in one sitting that used to take my team three days."
That's what we work on with professionals and teams through Work in Beta.
For individuals: We teach you how to work on your actual workflows and rebuild them around what's possible now.
For organizations: If your AI strategy is "let people figure it out," it's not a strategy. We help teams redesign how they actually work together with AI.
If you're curious what the other 95% looks like, reach out to us here.