zero gravitySOLUTIONS
← All insightsAI & AUTOMATION / EVALUATION GUIDE

Decide when a knowledge assistant should not answer

QUICK ANSWER

A knowledge assistant should answer only the claims supported by current authorized evidence. Evaluate answerability separately from answer quality, distinguish missing context from missing knowledge, and measure both unsupported answers and unnecessary declines before choosing an abstention policy.

Inside this article 5 sections

A knowledge assistant should not answer every question merely because it retrieved a related paragraph. It should also avoid declining questions that its evidence can answer. The useful design balances unsupported claims against unnecessary refusal, with the consequence of each error made explicit.

In retrieval-augmented generation, or RAG, answerability belongs to the combination of question, authorized evidence, applicable policy, and requested action. A question can be answerable for one user and unanswerable for another because their permitted evidence differs.

Label the evidence state before grading the prose

Use a small rubric: fully answerable, partially answerable, needs user clarification, conflicting evidence, and insufficient evidence. These states should lead to different responses. A missing product version may call for one clarifying question; an absent policy should not trigger a question the user cannot reasonably answer.

For a synthetic export-policy collection, “How long are annual-plan exports available?” may be fully answerable. “How long is my export available?” may need the plan or export date. “Will the product support an unpublished feature next quarter?” may be outside the available evidence even if the roadmap page contains similar language.

Have reviewers label answerability using the exact evidence snapshot available to the assistant. If reviewers consult an unrestricted source that the user cannot access, their label may unfairly demand an answer the application should not provide. Preserve the applicable access scope with the case.

Give each evidence state a useful response

ANSWERABILITY MAP / EVIDENCE TO RESPONSE
Complete supportAnswer the question with the relevant conditions and source references.
Partial supportAnswer the supported part and identify the specific missing part.
Missing user contextAsk the smallest clarifying question that changes the answer.
Unresolved conflictExpose the disagreement and route it to the source owner.
No sufficient evidenceDecline the unsupported claim and offer an appropriate next step.
Abstention is not one generic refusal sentence. The response should explain the actionable evidence gap without revealing restricted information.

A partial answer should not imply that the omitted part is unimportant. If the missing fact determines whether the user may take an action, say so clearly. For example, knowing the ordinary export deadline is insufficient to approve an exception whose eligibility is unknown.

Do not reveal the existence or title of inaccessible documents when explaining a decline. “I do not have sufficient accessible evidence for that decision” can be more appropriate than naming the confidential source that would answer it. The explanation itself must respect the same permission boundary.

Measure answers and declines together

Build an evaluation set containing answerable and unanswerable cases, then report how often the assistant answers each group. Within answered cases, measure supported correctness. Within declined cases, measure how many could have been answered from the supplied evidence.

Suppose an illustrative set contains 100 questions: 70 answerable and 30 unanswerable. A policy answers 60 of the answerable cases and incorrectly answers 6 unanswerable ones. It declines the remaining 34. Its coverage is 66%, while unsupported answers make up 6/66, or about 9.1%, of the answered cases.

The same policy unnecessarily declines 10/70, or about 14.3%, of answerable cases. Those figures expose two different costs. A headline saying “34 safe declines” would hide the ten avoidable failures to help; a headline saying “66 answers delivered” would hide six unsupported answers.

Use consequence-aware weighting only after showing the raw counts. A false answer about a harmless definition and a false answer authorizing a costly action should not automatically carry the same decision weight. The business owner should define the relevant consequences; avoid inventing financial penalties just to make a score look rigorous.

This Python fixture uses six synthetic judgments with separate labels for answerability, whether the assistant answered, and whether that answer was supported and correct. Each ratio keeps its own denominator visible.

cases = [
    (True, True, True), (True, True, True),
    (True, False, False), (False, True, False),
    (False, False, False), (False, False, False)
]
answered = [row for row in cases if row[1]]
answerable = [row for row in cases if row[0]]
coverage = len(answered) / len(cases)
supported_correctness = sum(row[2] for row in answered) / len(answered) if answered else None
unnecessary_declines = sum(not row[1] for row in answerable) / len(answerable) if answerable else None
assert coverage == 0.5
assert supported_correctness == 2 / 3
assert unnecessary_declines == 1 / 3

The fixture exposes both the unsupported answer and the answerable case that was declined. With zero answered cases, supported correctness is undefined rather than perfect. Keep case counts and slice results beside these ratios; six synthetic judgments are useful for testing the calculation, not estimating production quality.

Do not confuse retrieval score with answerability

A high similarity score means the query and passage are related under a particular retrieval method. It does not establish that the passage contains the answer. Hybrid ranking scores and reranker outputs also have system-specific meanings. A universal score threshold copied from another corpus is unlikely to define your product's evidence boundary.

Use observable checks where possible: required source type, current version, matching product scope, presence of the decisive condition, and citation support for the proposed claim. A model-based answerability judge can help with semantic cases, but evaluate it against reviewed examples and inspect disagreements.

Choose thresholds on development or validation data and reserve independent cases for the release decision. Include near-miss questions whose wording resembles an answerable question but changes a critical condition. These cases test whether the system recognizes missing evidence rather than simply matching familiar topics.

Allow bounded recovery before declining

The first retrieval may fail even when the corpus contains the answer. A limited query rewrite, alternative search path, or targeted source expansion can recover it. Record the additional latency and cost, and stop after a defined budget. Repeatedly asking the same model whether it is now confident does not create new evidence.

Track why the final response abstained: missing source, unclear user context, conflict, access restriction, stale content, or exhausted retrieval. These reasons point to different improvements. A source-onboarding problem should not be fixed by lowering the answerability threshold.

The citation verification guide checks claim support, and conflicting document guidance handles incompatible sources. Our AI integration work can connect those checks to a practical selective-answering policy.

FROM INSIGHT TO ACTIONAI-native implementation

Build an assistant that knows its evidence boundary.

We help create answerability evaluations and useful fallback behavior around the sources, permissions, and decisions your users depend on.

Explore AI implementationDiscuss your AI workflow

Start with your goals.
We’ll work through the next step together.