What an In-App AI Assistant Can and Cannot Answer
Every business system now has a chat box in the corner. Almost none of them do what users assume they do on first sight. Here is precisely what ours answers, what it cannot see, why that boundary exists — and how to tell, in one question, what any vendor's assistant really is.
Put a chat box in a business system and the first question every user types is some version of "what were sales last month?". It is the obvious thing to ask. It is also, in most products including ours, the one category of question the assistant cannot answer — and the gap between what users assume and what is actually there is where most disappointment with these features comes from.
So this article does the thing product pages usually avoid: it draws the line precisely, and explains why the line is where it is.
Two completely different products wear the same interface
A chat box in the corner of an application can be one of two fundamentally different things, and they look identical until you ask the right question.
A guidance assistant
- Answers "how do I do X in this system?"
- Matches your phrasing to a known task and routes you to the page
- Knows the product; knows nothing about your records
- Cheap, predictable, and cannot state a wrong figure
- Replaces reading documentation and asking a colleague
A data agent
- Answers "what were sales in Nakuru last month?"
- Translates a question into a query against your database
- Must respect every permission the asking user has
- Expensive, harder to make correct, and can confidently be wrong
- Replaces building a report — when it works
Ours is the first kind, and we will say so before you buy rather than after. The in-app assistant matches what you typed against a library of known tasks and answers with guidance and a link to the right screen. It does not read your records.
How it actually works
Under the friendly surface the mechanism is deliberately simple, and knowing it tells you how to get good answers out of it.
-
Your question is normalised and expanded
Casing and punctuation are stripped, and common synonyms are expanded — so "buy", "purchase" and "procure" reach the same place. This is why phrasing matters less than people expect.
-
It is scored against known tasks
Each known task carries keywords, and the best-scoring match wins. Where the score is zero, you get a helpful fallback rather than a guess — a deliberate choice, because a confident wrong answer is worse than an admission.
-
The page you are on nudges the match
Asking "how do I approve this?" from the procurement screen and from the leave screen are different questions, and where you are asking from is used as a hint.
-
The answer may be phrased by a language model
Where an AI provider is configured, the matched answer can be expressed in natural language rather than as canned text. The provider phrases; it does not decide. If no provider is available it falls back to the written answer, which is why the assistant never simply breaks.
-
The question is logged
Including whether anything matched. Unmatched questions are the most valuable output of the whole feature — they are a list, written by your own staff, of what the product or its documentation fails to explain.
The most valuable thing an assistant produces is not its answers. It is the log of the questions it could not answer — a list, written by your own users, of everything your system fails to explain.
Why it does not read your data
This is a design decision, not a gap waiting to be filled next quarter, and there are three reasons worth stating.
The first is permissions. An assistant that answers questions about your records has to enforce, perfectly, every permission the asking user holds — including the ones that separate own payslips from all payslips, and sensitive fields from ordinary ones. An assistant that leaks a salary because a question was phrased unusually is not a feature with a bug; it is a data-protection incident with a chat interface.
The second is correctness. "Sales last month" has to resolve to a definition — invoiced or paid, gross or net of returns, by document date or by payment date. Your finance team already has answers to those questions and they are encoded in your reports. An assistant that guesses differently produces a number that disagrees with the report, and then nobody trusts either.
The third is that the honest alternative already exists. A report definition gives you the same answer, defined once, repeatable, shareable and schedulable. Where a number matters enough to ask about monthly, it deserves to be a report rather than a conversation — see building your own reports.
The question to ask any vendor
One question separates the two products
"Ask your assistant what our total sales were last month, in front of me, on a real dataset."
What you will hear
"It can help you find the sales report."
How to read it
It is a guidance assistant. Perfectly respectable — that is what ours is. Just do not buy it expecting analytics.
"Now ask it the same question as a user who cannot see sales."
What you will hear
A pause, or an offer to arrange a follow-up call.
How to read it
The data agent exists but permission enforcement has not been proven. This is the question that matters, and it is rarely asked.
"Does its number match what the sales report says, to the shilling?"
What you will hear
"It uses the same underlying data."
How to read it
Same data, possibly different definition. Ask them to run both side by side. Two authoritative numbers that disagree is worse than one number.
"Show me the log of questions it failed to answer."
What you will hear
Often genuine surprise that you asked.
How to read it
Whether they capture unanswered questions tells you whether the feature is being improved or was shipped as a demo.
Getting value from the one you have
A guidance assistant is genuinely useful in one specific circumstance, which happens to be the most expensive circumstance in software adoption: a new user, mid-task, who does not know what the feature is called. That person's alternatives are to interrupt a colleague, guess, or stop. An assistant that recognises "how do I send this back to the person who raised it" and points at the right screen has saved a real interruption.
- Point new staff at it during induction, explicitly as a "how do I" tool — the framing prevents the disappointment.
- Read the unmatched-question log monthly. It is the cheapest usability research you will ever get.
- When a question recurs and is unmatched, that is a documentation or training gap, and now you know exactly how your staff phrase it.
- Do not put it in front of customers expecting it to answer questions about their orders. It is an internal guidance tool.
What we do and do not do
What AWRA OpsHub does today
- Task guidance across the product — how to do a thing, and a link to the screen where you do it.
- Synonym-aware matching, so everyday phrasing reaches the right task without exact wording.
- Page-aware hints, using where you are asking from to disambiguate.
- Optional natural-language phrasing through a configured AI provider, with fallback to the written answer when none is available.
- Query logging, including which questions matched nothing — visible to you.
- A separate public assistant on the website, answering product questions for visitors from published content.
What it does not do
- It does not read your business data. No sales totals, no stock levels, no customer balances, no payroll figures. Not a limitation of the current version — a boundary.
- It cannot perform actions. It will not raise a purchase order, approve a request or edit a record on your behalf.
- It is not a support agent. It does not open tickets or escalate to a human within the same conversation.
- It does not learn from your corrections. Improvement happens when the task library is extended, informed by the unmatched-question log — a deliberate process, not silent drift.
If a vendor tells you their assistant does all four of those, ask for a live demonstration against a real dataset with a restricted user account. That single request separates working products from convincing ones.
Our take
Judge an assistant by what it refuses to answer. Ours answers "how do I" and routes you to the page; for "what is the number", build the report once and schedule it. Then read the unmatched-question log every month — that log, not the chat box, is where the actual return on this feature lives.
See the AWRA assistants
An in-app guidance assistant that routes you to the right screen, a public assistant for website visitors, and a log of the questions neither could answer.
Explore the assistantsFrequently asked questions
Can the assistant tell me our sales figure for last month?
No. It answers questions about how to use the system and routes you to the right screen — it does not query your records. That boundary is deliberate: an assistant answering data questions must enforce every permission the asking user holds and must agree exactly with your reports on definitions like invoiced versus paid. For recurring numbers, a saved report definition is the better instrument anyway, because it is defined once, repeatable and schedulable.
Does our data get sent to an AI provider?
What can be sent is the phrasing of the answer — where a provider is configured, the matched guidance is expressed in natural language. Your business records are not part of that exchange, because the assistant does not read them in the first place. If your policy requires that nothing at all leaves your environment, the assistant still works with no provider configured; you get the written answers rather than conversational phrasing.
What happens when it does not understand a question?
It says so and offers a fallback with suggested questions, rather than guessing at the closest match. That is a deliberate choice — a confident wrong answer in a business system costs more than an admission. Every unmatched question is logged, and that log is the most useful output of the feature: it is a list, in your staff's own words, of what your system or its training fails to explain.
Can we add our own answers to it?
The task library is configuration rather than code, and the unmatched-question log is what should drive additions to it. If you have organization-specific procedures you want it to answer — your own approval thresholds, your own naming for things — raise that as a requirement; it is a reasonable ask, but treat it as scoped work rather than assuming a self-service editor exists.
Is the assistant on your website the same thing?
Related but separate. The public assistant answers product and company questions for website visitors from published content — documentation, blog posts, feature pages — while the in-app assistant guides users through tasks inside the product. Neither reads customer or business data. Keeping them separate is why the public one can be open to anyone without any access question arising.
