Evaluating AI agent startups: where workflow automation is defensible
The defensibility of an AI agent startup rests less on how smart its model is and more on four things: workflow data, depth of integration, evaluation infrastructure, and a track record of reliable operation. Foundation models improve on a cycle of months, so a product differentiated only by what it can do today is exposed to vendor updates and new entrants.
This article narrows the five-axis framework for AI startup competitive advantage to the agent category and sets out what investors and acquirers should check in technical due diligence.
What counts as an agent, and how it differs from chatbots and workflow automation
“Agent” is applied to products that behave very differently. The first step in any evaluation is to place the target.
| Type | How it works | How it fails | Likely source of defensibility |
|---|---|---|---|
| Chatbot | Answers a human’s question with information | Wrong answers (hallucination) | Reference data, answer quality |
| Workflow automation | Executes a predefined sequence mechanically | Stops on unexpected input | Number of connectors, ease of adoption |
| AI agent | An LLM picks the next action toward a goal and calls external tools across several steps | A bad decision midway propagates downstream | Integration depth, evaluation infrastructure, track record |
Because agents decide for themselves, they fail in more complicated ways. Even with high per-step accuracy, success multiplies down over a ten-step task. The question is therefore not “does it work in the demo?” but “does it finish the job in production?”
In practice, many products marketed as agents are closer to fixed-sequence workflows. That is not a flaw; the more fixed the sequence, the more reliable it tends to be. What matters is how much is left to the LLM’s judgment and how much is bounded by deterministic logic, not the label.
Candidate sources of defensibility: four axes
Axis 1: Workflow data
Where an agent runs, data accumulates: logs, past cases, exception records, and the edits people make to its output. The most valuable of these is how humans corrected the agent’s output. That data cannot be scraped from the public web; it exists only inside customers’ operations.
Check whether the data is merely stored or actually used for evaluation, prompt improvement, or fine-tuning. If it is not used, it is not an advantage. Also confirm that the contract secures the right to use it.
Axis 2: Integration depth
The more a product is wired into a customer’s core systems, permissions, and approval flows, the higher the cost of switching. But “can call the API” is not the same as “is part of the business process.” Three things indicate depth:
- Write access: read-only, or able to update and execute
- Embedding in approval flows: agent actions sit inside the customer’s internal controls
- Ownership of exceptions: handoff to humans for unexpected cases is part of the customer’s operating procedure
Integration is a strong barrier, but as noted in the earlier article it is also a form of path dependence. When the paradigm shifts, customers tolerate the incumbent only for so long.
Axis 3: Evaluation infrastructure
Agent output varies run to run. Without a way to confirm that a change did not degrade quality, a team can neither improve the product nor swap foundation models safely. Teams with evaluation datasets, automated grading, and release gates can test and adopt a new model quickly. In other words, they can turn model progress into a tailwind.
The article on LLM evals as a technical DD axis covers the details. For agents, the distinguishing factor is whether intermediate steps (tool selection, arguments, retries) are evaluated, not just the final result.
Axis 4: Operating track record
Agents meet unexpected inputs and unreliable external systems only in production. The reliability earned there cannot be copied quickly. Look for incident records and recurrence-prevention history, uptime and success rates by customer over time, and security and audit track record. In regulated domains such as finance, healthcare, and manufacturing, the period of uneventful operation is itself a form of trust.
What disappears and what remains as foundation models improve
The core of the evaluation is to ask how the product’s value changes when the next-generation model ships.
| Type of value | Effect of model progress | Reason |
|---|---|---|
| Prompt craft and output formatting | Tends to disappear | Smarter models make it unnecessary or a built-in feature |
| Complex orchestration (multi-branch control) | Tends to thin out | As models plan and use tools better, outer control simplifies |
| Generic features (summarization, extraction) | Tends to disappear | Vendors can ship them as standard |
| Workflow and correction data | Remains | Generated only inside customers’ operations |
| System integration and permission design | Remains | A smarter model does not settle connection and accountability |
| Evaluation infrastructure | Remains, even strengthens | Lets the team absorb model progress safely |
| Operating track record | Remains | Can only be earned over time |
A simple test for investors: if the underlying model’s capability doubled, would this product’s competitiveness go up or down? If up, the product rides the improvement. If down, much of its value may lie in compensating for model shortcomings.
Metrics to check
Task completion rate
The share of tasks the agent finishes without human help. Watch the denominator. Numbers look good easily if the sample is limited to easy tasks or failures are excluded. Check:
- Whether there is a breakdown by task type (difficulty)
- Whether results vary by customer
- Whether the trend is improving, and whether any customer is getting worse
- Whether “complete” matches business success for the customer
Human intervention rate
The share of tasks where a person approved, corrected, or took over. A low rate is not always good. If approvals have become a formality and nobody actually reads the output, a low number means risk is invisible. Conversely, a high rate that is falling, with interventions fed back into improvement, is a healthy growth trajectory.
Behavior on error
This is where products differ most, and it is invisible in a demo. Check whether failures are detected (no silent wrong answers), whether the agent stops, hands off, or rolls back, and whether anything prevents the same failure from recurring. For agents with side effects (payments, outbound email, data updates), a confirmation step before irreversible actions is a key signal.
Case study: two types in the same domain
Consider two agent startups automating invoice processing for finance teams. These are archetypes, not specific companies.
Type A: the generic agent. It uses several LLMs to read invoices and propose journal entries. Early revenue grew fast, but it does not tailor to each customer’s chart of accounts or approval flow and has no evaluation dataset. Behavior shifts whenever the underlying model updates, and customers report that something that worked last month no longer does. When the accounting software vendor added a similar feature as standard, the differentiation vanished.
Type B: the workflow specialist. At onboarding it ingests each customer’s account mapping and approval rules, and it stores accountants’ corrections as data. It owns an evaluation dataset and re-tests on past cases before switching to any new model. It writes to the accounting system only after the approval flow. As models improved, its accuracy rose and its human intervention rate fell.
The difference is not model quality. Type B had a mechanism to absorb model progress (evaluation infrastructure) and assets rooted in customers’ operations (correction data, integration). In an investment decision, ask which type the company is moving toward, and whether the founding team can articulate it.
Summary: questions for the decision
When evaluating an AI agent startup, check in order:
- Is it a chatbot, workflow automation, or an agent? How much is left to the LLM’s judgment?
- Is workflow data, especially correction data, accumulating and used for improvement?
- How deeply is it embedded in customers’ systems and approval flows?
- Is there evaluation infrastructure that lets the team adopt new models safely?
- Can the team explain completion rate, intervention rate, and error behavior by type and over time?
- If model capability rises, does competitiveness go up or down?
As what changes in technical due diligence in the AI era lays out, evaluating LLM-based products takes more than an extension of traditional DD. In the agent category, evaluation infrastructure and operating track record sit at the center of defensibility. For deal-side evaluation see TiedPro for investors, and for founders designing agent products see TiedPro for startups.