zero gravitySOLUTIONS
← All insightsAI & AUTOMATION / RETRIEVAL EXPERIMENT

Test hybrid search on product documentation questions

QUICK ANSWER

Hybrid search is useful when product questions mix exact identifiers with conceptual language. Evaluate keyword, vector, and fused results on the same versioned corpus, preserving product and permission filters and measuring complete supporting evidence rather than trusting a higher combined score.

Inside this article 5 sections

Product documentation contains two kinds of language that search must handle well. Customers ask conceptual questions such as “Why does my export stop halfway?” while engineers search for exact error codes, endpoint names, and configuration keys. Neither a keyword-only nor a vector-only strategy should win by assumption.

Hybrid search combines independently ranked lexical and semantic candidates. It can improve coverage when those methods find different useful passages, but it can also add duplicates or promote the wrong product version. The engineering task is to test whether the combined evidence answers your questions more reliably within the same operating budget.

Build a small corpus that exposes the difference

Create a synthetic documentation set with an export troubleshooting page, an API reference for EXPORT_409, a retired endpoint guide, and a current limits page. Give each document a product identifier, version, source status, and stable section IDs. Include one irrelevant page that happens to contain the word export frequently.

Write questions in three groups. Exact questions name codes or parameters. Conceptual questions describe symptoms without those terms. Mixed questions combine an identifier with a condition, such as whether EXPORT_409 can occur after a retry. For each question, identify the required passage and any exception needed for a correct answer.

Include unanswerable questions and deliberately confusing versions. A system that always retrieves something can look productive while supplying evidence from a retired API. Preserve the same corpus and relevance judgments across all candidate retrieval configurations.

Inspect each ranking before combining it

Run lexical search and vector search separately first. Lexical retrieval can preserve exact codes when tokenization and analyzers handle them correctly. Inspect how underscores, hyphens, case, and version suffixes are indexed. A field dedicated to identifiers may be more reliable than hoping a general prose analyzer preserves every code.

Vector retrieval can connect symptom descriptions to documentation written in different words. It can also retrieve a semantically similar but technically incompatible feature. Keep product, version, tenant, and access constraints as explicit filters. Similarity is not authority to cross those boundaries.

ILLUSTRATIVE RANK FUSION / ONE QUESTION
Reciprocal rank fusion with constant k = 60
PassageLexical rankVector rankCombined contribution
A · Error-code definition131/61 + 1/63 ≈ 0.032266
B · Retry conditions211/62 + 1/61 ≈ 0.032522
C · General export overviewAbsent20 + 1/62 ≈ 0.016129
These invented rankings put passage B first because it ranks well in both lists. The fused number is a ranking score, not a probability that the passage answers the question.

Reciprocal rank fusion avoids adding raw lexical and vector scores that may have unrelated scales. The example adds one contribution per list using one-based ranks; an absent item contributes zero. The constant controls how strongly rank positions differ. Treat its value and candidate-list lengths as configuration to evaluate, not a universal optimum.

Deduplicate by a stable passage identity before assembling context. If the two searches return different representations of the same source section, canonicalize them. Otherwise the generator may receive repeated evidence while a necessary exception falls outside the context budget.

This Python example implements reciprocal-rank fusion over stable, already authorized chunk IDs. It combines rank positions rather than adding a vector similarity to a lexical score. The synthetic result shows why a chunk appearing near the top of both lists can outrank a single-list result.

def fuse(rankings, k=60):
    if k <= 0:
        raise ValueError("k must be positive")
    scores = {}
    for ranking in rankings:
        seen = set()
        for rank, chunk_id in enumerate(ranking, start=1):
            if chunk_id in seen:
                continue
            seen.add(chunk_id)
            scores[chunk_id] = scores.get(chunk_id, 0) + 1 / (k + rank)
    return sorted(scores, key=lambda key: (-scores[key], key))

lexical = ["error-code", "export-guide"]
semantic = ["export-guide", "limits"]
assert fuse([lexical, semantic])[0] == "export-guide"
assert fuse([]) == []

The constant 60 is an illustrative smoothing choice, not an optimized setting for this corpus. Filter unauthorized and retired evidence consistently before fusion, then evaluate the final context under a fixed size budget. Ranking agreement improves a candidate’s position; it does not prove that its passage answers the question.

Compare the evidence under a matched budget

Keep the source snapshot, chunking, filters, question set, and final context budget fixed. Compare lexical-only, vector-only, and hybrid retrieval. If one method supplies ten times as much text, the experiment mixes retrieval quality with evidence volume.

Measure whether the top context contains the complete required evidence, whether the first useful passage appears early, and how much irrelevant or duplicated text is included. Break results out by exact, conceptual, and mixed questions. An overall average can hide a regression on error-code questions while showing improvement on paraphrases.

Review false positives with their metadata. A retired guide may be highly relevant semantically but invalid for the current question. Fix the version-filter or authority model before increasing the weight of one retriever. Ranking cannot compensate reliably for missing eligibility constraints.

If you add a reranker, treat it as a separate experimental stage. It can only reorder or select candidates it receives; it cannot recover a required passage excluded earlier. Record candidate recall before reranking and final evidence quality afterward so the failure location remains visible.

Test the answer without hiding retrieval failures

Give the same answer generator the evidence produced by each retrieval method. Grade factual claims and citation support separately from style. A strong model may answer from general knowledge even when retrieval failed, so require the expected evidence for product-specific claims. Otherwise the experiment rewards an unsupported answer that happens to be correct.

Use adversarial pairs such as two similarly named settings with opposite defaults. Ask questions that require preserving a negation or a version condition. These expose cases where the search result looks topically excellent but is insufficient for the actual decision.

Track latency and resource use for both retrieval branches and any reranking. Parallel execution does not make the work free, and the slower branch can dominate the response if the system waits for both. Define whether a partial branch failure permits a degraded answer or requires a retry.

Ship the smallest supported improvement

Retain the query set and ranking traces with the chosen configuration. Reevaluate when documentation structure, embeddings, analyzers, or product versions change. Add real failure questions after review, while keeping a separate holdout for future comparisons.

The chunk-boundary guide explains complete evidence units, and citation verification checks whether answers use them faithfully. Our AI integration services can connect retrieval experiments to a production documentation assistant.

FROM INSIGHT TO ACTIONAI-native implementation

Build retrieval around the questions your product receives.

We help compare lexical and vector search, define relevance judgments, and connect the selected retrieval path to grounded answers.

Explore AI implementationDiscuss your AI workflow

Start with your goals.
We’ll work through the next step together.