InData Labs shares research on the real cost of one conversation in a live AI coaching product. Most of the bill comes not from the user’s words but from repeated instructions, retrieved context, and background jobs. The article also offers practical advice on what to focus on when measuring AI costs, from tracking real token usage early to ordering prompts so caching can cut expenses.
Key takeaways:
- LLMs have no memory between messages, so every turn re-sends roughly 10, 000 tokens of instructions and context before the user’s question is even counted.
- Invisible background jobs (summaries, profile updates, follow-up questions) account for about a quarter of total cost and are a flat fee per session, which hits short-interaction products hardest.
- Putting fixed instructions before changing content makes prompt caching four times more effective, with no new infrastructure, though the change needs quality testing.
- Paper estimates of model output were 2–5x too high; measure real token usage early instead of guessing.
We measured every message in a live AI coaching assistant.
The words the user types turned out to be less than 1% of the bill.
“How much will this cost per user? ” is one of the first questions anyone asks about an AI product, and one of the last you can answer honestly. Until real people are using the thing, you are guessing.
To understand where those costs come from, it helps to know what the product actually does.
The product is an AI coaching platform in the consumer well-being space. Users hold text conversations with specialized AI coaches, each with its own knowledge base, personality, and approach. The system retains context across sessions, gradually builds a profile of each user, and generates a personalized follow-up plan after every conversation. “The product is live, with real users, running in production”.
The interface shown here is a simplified example created for this article. We’ve kept the coach’s identity and branding private under our project NDA. In the real product, each experience is shaped around a real, credentialed coach and their body of research and guidance. This particular example is inspired by the work of a researcher focused on bullying and peer victimization.
Coach
Parenting & family wellbeing
My daughter is new in her children’s choir. Older kids tease her because she doesn’t sing the hard pieces well yet. What can I do to help her?
9:41 AM
Your daughter is entering a group where skill and age are mixed together — exactly the kind of setting where teasing can hide behind “high standards.”
The first step is telling normal correction apart from bullying. One sharp comment is worth addressing gently. A repeated pattern of mocking or excluding needs adult structure.
Try saying this:
Then talk with the director, calmly: she’s new, the repertoire is demanding, and she’s noticed older singers mocking others. Ask for clear rehearsal norms — corrections come from the director, not from peers.
What has she told you so far?
9:42 AM
The user writes ten words. The bill is for 40, 000 tokens.
AI providers charge by the token—roughly three-quarters of a word. In a typical session with our assistant, a user’s message is around 14 tokens. About ten words. The coach’s reply is around 88 tokens, a short paragraph. Three exchanges, and the whole visible conversation is a few hundred words.
The bill for that session covers about 40, 000 tokens.
Everything the user actually wrote and read accounts for well under one percent of what we pay for. The rest is invisible.
Why: The model forgets you between every message
“Here is the thing most people don’t know about how these systems work. Every time you send a message to a language model, it arrives with no memory of what came before. Not a vague memory — none at all. The conversation you experience as continuous is an illusion, maintained by re-sending everything, every single time,” Palina concludes.
Imagine a brilliant consultant with total amnesia. Before you can ask today’s question, you have to hand him the company handbook, his own job description, your file, and a transcript of every word you’ve exchanged so far. Then he answers in two sentences. Then he forgets, and you do it again.
That is what each message in our system carries:
- The instructions—the coaching methodology, safety rules, formatting, and the specific coach’s personality. About 5, 900 tokens, sent again with every turn.
- The context—relevant excerpts from past conversations and from the coach’s knowledge base, pulled in fresh each time. About 4, 200 tokens.
- The conversation so far—which grows with every exchange.
Before the user’s own question is counted, each turn already carries roughly 10, 000 tokens of overhead.
And the work nobody sees
When a session ends, four background jobs run: writing a summary, updating the user’s health profile, updating what the system knows about their family, and preparing follow-up questions for next time. The user never sees any of it.
That housekeeping is about a quarter of the total cost—and it’s a flat fee. A two-minute session pays the same background overhead as a long one. If your product is built around many short interactions, this is the number that will surprise you.
What it adds up to
Because the instructions repeat and the history grows, cost rises steadily with conversation length:
| Conversation length | Cost per session | Per month (3 sessions/week) |
|---|---|---|
| 5 exchanges | $0.16 | $2.11 |
| 10 exchanges | $0.30 | $3.91 |
| 20 exchanges | $0.60 | $7.76 |
| 30 exchanges | $0.92 | $11.94 |
Prices reflect model rates as of April–May 2026.
Every additional back-and-forth adds a cent and a half. Comfortable with a thousand users. A line item worth arguing about at a hundred thousand.
The cheapest fix we found was moving a paragraph
Providers offer a discount for repetition. If a request begins with exactly the same text as the last one, that repeated opening is billed at half price. It’s automatic and free, a feature called prompt caching, and one of the most overlooked levers for conversational AI cost savings.
The catch is that the discount only covers the unchanged beginning of a request. The moment something new appears, the discount stops there and doesn’t resume.
In our system, the user’s question sat near the top of the prompt, just above the largest, most repetitive block of instructions in the whole thing. So the discount applied to the first 1, 300 tokens and then switched off exactly where it would have mattered most.
No new infrastructure, no new vendor, no clever engineering. Reordering a document. The one real caveat: these models are genuinely sensitive to the order of their instructions, so a change like this needs to be quality-checked before it ships.
We were also bad at guessing
“Before measuring anything, we tried to predict these numbers on paper. For each of those background jobs, we asked ourselves how much text it would produce. We guessed too high every single time—by two to five times, ” Palina adds.
Take the session recap the user reads at the end. We expected around 400 tokens, several paragraphs. The system actually writes 74 tokens: a short paragraph, about fifty words.
After each session, the app creates a short recap of what stood out, along with a few practical next steps to help turn the conversation into action.
Session summary
Supporting Her Through Choir Teasing
Key insight
Things to try
- Tell her that being new and still learning the music is completely okay.
- Explain the difference between kind correction and mocking that crosses the line.
- Talk calmly with the director about her being new to the demanding repertoire.
- Ask the director to keep corrections coming from adults, not from other singers.
- Keep noticing what she shares about choir to gauge if things improve.
The reason is simple. When you write an instruction asking for a “detailed summary, ” you picture what you would write if someone asked you for one—a page, maybe more. The model doesn’t work that way. It’s much more concise than the instruction makes it sound. Any cost estimate built on that instinct comes out wrong.
Three things worth taking with you
If there’s one takeaway from tracing conversational AI cost down to the token, it’s that the expensive parts of an AI system are rarely the ones people assume.
Measure early, and measure real usage. The common shortcut of estimating tokens as characters divided by four is close enough for a napkin and misleading at scale. Real numbers from stored data took us an afternoon and changed several decisions.
Your instructions cost more than your conversations. Because the model has no memory, every rule you write is paid for again on every message. Prompt length is a running cost, not a one-time design choice.
Check the order of your prompt before launch. Fixed content first, changing content after. It costs nothing on day one and requires testing to retrofit later.
FAQ
-
Based on real production data from a live AI coaching assistant, a typical session costs around 11 cents—but the conversational AI cost breaks down very differently than most people expect. The user’s actual words account for less than 1% of the bill; the rest comes from LLM token costs tied to system instructions, retrieved context, and background processing that happens after the conversation ends.
This is why any serious estimate of how much conversational AI costs actually needs to look past visible chat length and measure real token usage instead—a lesson that applies to large language model development generally, not just chat products.
-
Because instructions and context repeat with every turn, cost per token LLM pricing means cost rises steadily as a conversation gets longer—not linearly with what the user typed, but with everything the system re-sends behind the scenes. In one measured system, a 5-exchange session cost about $0. 16, while a 30-exchange session cost $0. 92.
Understanding this relationship is central to conversational AI cost savings, since even small reductions in repeated overhead compound significantly at scale—a difference that shows up clearly in cost per lead reduction conversational AI calculations once you’re running thousands of sessions a month.
-
The short answer: don’t rely on paper estimates. Teams evaluating the cost to build an AI assistant often guess background-job output sizes are 2–5x too high, because it’s tempting to imagine what a person would write rather than what a model actually generates.
The more reliable approach to custom AI assistant development cost planning is to build a small working prototype, log real token counts, and calculate actual conversation AI software cost from that data—even a rough snapshot beats a theoretical model built on assumptions. This is also where working with an experienced AI consultant or AI consulting partner helps, since they’ve already seen where these estimates typically go wrong.
-
For a live, memory-enabled coaching assistant, each chat costs a few cents to under a dollar depending on conversation length, with typical sessions running 0. 16–0. 92. This kind of real benchmark is far more useful than generic AI assistant cost figures often quoted online, since actual pricing depends heavily on how much context, memory, and background processing a given product requires per session.
-
AI is inexpensive at the level people usually imagine—the words in a chat—but expensive at the level they don’t see: repeated instructions, retrieved context, and invisible background jobs.
Answering how much does it cost to develop an AI assistant honestly means looking at the full token bill, not just the visible conversation. In practice, the cost of AI assistant operation stays manageable at low usage volumes but becomes a serious line item—and a legitimate target for conversion AI cost optimization — once a product scales to tens or hundreds of thousands of users. This is one reason teams increasingly turn to specialized generative AI consulting or AI software development partners rather than building and optimizing cost architecture entirely in-house.
