The gross margin structure of LLM products: what pricing and inference cost reveal about durability
When evaluating an LLM product, attention tends to gravitate toward feature quality and user growth. What actually determines the business’s durability is its gross margin structure. The conclusion up front: LLM inference cost is the product of three variables — token volume, model choice, and call count — and when this is paired with pricing that isn’t usage-based (flat-rate or seat-based plans), it produces a counterintuitive outcome: margin gets worse as usage grows. Investment due diligence on an LLM product needs to go beyond features and growth rate and confirm three things: how usage volume relates to cost, whether the pricing model is aligned with that cost structure, and whether the team has actually implemented technical levers to protect margin.
What determines the cost of an LLM product
At bottom, LLM inference cost is the product of three variables.
Token volume is the number of input and output tokens (subword units, not characters) processed per call. Longer context — conversation history, documents injected via RAG, system prompts — inflates input tokens, and more detailed responses inflate output tokens. Most API providers price output tokens well above the input token rate (often several times higher as a rough rule of thumb, though the exact ratio varies by provider, model, and pricing revision), so a design choice that produces longer answers has a direct, disproportionate cost impact.
Model choice is the variable with the largest multiplier on cost. The gap in per-token price between a frontier model and a lightweight model is typically several-fold to an order of magnitude. As architecture selection for LLM integration is decided across accuracy, confidentiality, cost, and speed, a design that defaults to the top-tier model everywhere starts the margin structure off in a hole, regardless of how well everything else is built.
Call count depends on per-user usage frequency and product design — and this is where the least visible cost hides. Query generation for RAG retrieval, self-verification passes on outputs, and multi-step agentic reasoning routinely turn a single user-facing request into three to ten model calls behind the scenes. When what looks like “one feature” is actually “several API calls” on the cost side, the only reliable way to estimate cost is from implementation logs, not from the user-facing interaction count.
The product of these three variables — inference cost per user per month — is the foundation of an LLM product’s unit economics. A team that can’t answer this number on the spot likely hasn’t validated whether its pricing is sound.
Pricing patterns and their fit with cost structure
LLM product pricing falls into three broad patterns, each with a different relationship to the underlying cost structure.
| Pricing model | Fit with cost | Risk |
|---|---|---|
| Usage-based (tied to token volume / call count) | Good — revenue tracks cost, so margin stays stable | Unpredictable pricing for customers; can drive churn among large accounts |
| Flat-rate (unlimited use) | Poor — cost scales without bound for heavy users | ”Unlimited” framing attracts disproportionately heavy users — adverse selection |
| Seat-based (tied to headcount, usage-agnostic) | Moderate — unpredictable if per-seat usage varies widely | A small share of heavy users can drag the average cost per seat far above plan |
Usage-based pricing is theoretically sound because revenue and cost move together, but it collides with the customer expectation — a SaaS-era norm — for simple, predictable pricing. Most products adopt flat-rate or seat-based plans precisely because of this customer-experience pressure.
The problem is that choosing flat-rate or seat-based pricing obligates you to monitor the usage distribution continuously. Pricing set against the average user’s usage will be blown past by the top few percent of heavy users, whose usage far exceeds that average. Ignoring this distribution when setting price is the direct cause of the “margin worsens as usage grows” pattern discussed below.
Technical levers that improve gross margin
Even after a cost-pricing mismatch is discovered, several technical levers exist to recover margin after the fact.
Caching reuses responses for identical or near-identical inputs, cutting the number of model calls outright. Prompt caching — processing the reusable portion of a long context or system prompt at a steep discount — is offered by most API providers, and the benefit scales with how much repeated context a product sends.
Model routing dynamically assigns the model to the task’s difficulty: a lightweight model for simple classification or summarization, a frontier model for complex reasoning. This can substantially reduce average cost per call. Moving from “always call the top-tier model” to “route to the right model per task” is one of the highest-leverage margin levers relative to its implementation cost.
Prompt optimization gets the same output quality with fewer tokens: trimming verbose system prompts, tightening output formats, dropping unnecessary confirmation steps. It looks unglamorous, but at millions of calls per month, saving even a few hundred tokens per call compounds into a meaningful cost reduction.
Distillation trains a smaller model on a larger model’s outputs as teacher data, substituting the smaller model for the larger one on a narrower task. It’s often paired with fine-tuning, which — as an architecture choice that trades upfront investment for room to optimize both accuracy and cost — pays off when a specific task is called frequently and the required accuracy range is narrow.
These levers aren’t independent; combining them compounds the benefit. Mature LLM products tend to layer caching and model routing together, then route the highest-frequency patterns to a distilled model on top of that.
How to spot the “margin worsens with usage” pattern
This is a classic trap for growth-stage LLM products: growth metrics — users, call volume — look healthy while gross margin quietly deteriorates. Four things to check.
First, look at the usage distribution. Compare the median to the top decile rather than relying on the average. If the top 10% of users consume five times the median or more, they are likely the ones driving up cost.
Second, check whether pricing is coupled to cost at all. If a flat-rate or seat-based plan has no usage ceiling — no rate limits, no feature gating — there is, by construction, no ceiling on cost either.
Third, track gross margin by cohort over time. Follow each new-user cohort’s margin at three, six, and twelve months post-signup. In products where deeper product usage is itself the success metric, margin by cohort tends to trend downward the longer a user sticks around — good for retention, risky for cost.
Fourth, check whether growth and cost management sit in separate parts of the org with no shared accountability. When a growth team is incentivized purely on usage volume and cost oversight is an afterthought, this structure tends to compound uncorrected as the company scales.
If several of these signals show up together, a high growth rate does not by itself indicate a healthy business — it may instead mean the company needs more capital the faster it grows.
Metrics to confirm during investment due diligence
For due diligence on an LLM product, look for the following as concrete numbers, not qualitative impressions.
Monthly inference cost per user. Comparing this against the per-user contract price gives you an inference-cost contribution margin — not the same thing as full gross margin, since it excludes other cost-of-revenue items such as non-LLM infrastructure and support. Still, if this narrower figure already falls well short of typical SaaS gross-margin benchmarks (commonly cited in the 70–80% range), that’s a strong signal a price change or cost-reduction plan is needed before any other COGS are even added in.
Usage distribution (median, top decile, maximum). Check how far the heaviest users’ usage sits past the break-even point.
Whether a cost-reduction roadmap exists. Confirm whether levers like caching, model routing, and distillation are already implemented or merely planned. Teams with no active lever risk a rapid margin collapse as they scale.
Model vendor dependence and cost volatility risk. Confirm the product’s resilience to a single provider’s pricing changes — whether it can realistically switch models.
How the contract handles usage. Even under flat-rate or seat-based pricing, check for fair-use policies or usage-tiered price adjustment clauses. A company with contractual room to absorb cost has a financial buffer even if its technical margin work is behind schedule.
Whether the management team can answer these on the spot is itself a signal of technical maturity worth weighing alongside a startup’s competitive moat. “We know our margin is thin but haven’t started cost reduction yet” is an honest, evaluable answer. “We haven’t discussed margin internally” suggests the team may be prioritizing growth without having validated the business’s durability.
Conclusion
An LLM product’s gross margin structure is set by the interaction of three cost variables — token volume, model choice, call count — and the pricing model layered on top: usage-based, flat-rate, or seat-based. Get this combination wrong, and even a product with strong growth metrics can end up in a structure where margin shrinks the more it’s used.
Investment due diligence on an LLM product should go past features and growth rate to confirm, in concrete numbers, cost per user, the usage distribution, and whether margin-protecting technical levers are actually implemented. Pairing this with the additional considerations that apply to technical due diligence in the AI era helps close the gaps that a feature-only review would miss.
If you’re an investor or acquirer looking to validate an LLM product’s unit economics as part of technical due diligence, reach out to Tied through our investor page.