Semantic Search Is Not Truth

02 · Marketing

People do not want a vector database. They want the product to understand what they meant.

The best marketing story for semantic search is not “we use embeddings.” Customers do not care how many dimensions a vector has. They care that the search box stops punishing them for not knowing the product’s internal vocabulary.

A strong semantic experience makes four promises:

• Ask naturally. Search with a question, scenario, symptom, misspelling, goal, or remembered phrase instead of learning the database’s preferred nouns first.
• Discover connected ideas. Move from one result into prerequisites, related concepts, comparisons, evidence, commands, policies, or next steps.
• Understand why the result appeared. Show the exact match, semantic relationship, source identity, freshness, exclusions, and confidence boundary rather than presenting a mysterious ranking.
• Keep uncertainty honest. Clarify or abstain when the system cannot support a strong answer.

That last promise is part of the sizzle, not an apology. A search experience earns trust when it knows the difference between “I found something related” and “I found the right evidence for this decision.”

For a learning product, the benefit is momentum. A student can ask the question in their own words and still reach the canonical concept. For a business product, the benefit is reduced friction: fewer dead ends, fewer support escalations, and faster discovery of the right policy or record. For a research product, the benefit is breadth without silent hindsight. For a document product, the benefit is structure-aware understanding without turning every similar paragraph into permission to rewrite the file.

Find what the user means. Prove why it belongs.

That is a stronger product position than “AI-powered search.” It combines the delight of being understood with the credibility of an answer that can show its sources, limits, and authority.

03 · Business and executive case

Search quality is product quality—and wrong confidence is an operating cost.

Search sits closer to revenue and risk than many teams realize. It shapes which product a customer discovers, which policy an employee follows, which document a reviewer trusts, which support answer is reused, which learning path a student takes, and which evidence reaches an agent before it proposes an action.

That means the business case has two sides. Better retrieval creates value. Uncontrolled retrieval creates liability.

A useful executive scorecard therefore cannot stop at “the demo found the answer.” It should track retrieval coverage, ordering, citation correctness, permission enforcement, stale-source exclusion, supported abstention, duplicate suppression, latency, cost, and replay stability.

The economic advantage comes from allocating intelligence carefully. Exact and lexical retrieval can handle identifiers, commands, names, and known phrases cheaply. Local embeddings can recover conceptual intent without routine provider cost or disclosure. Graph expansion can add reviewed relationships. A larger hosted model can be reserved for the cases where synthesis justifies its cost. Deterministic admission remains stable across all of those choices.

This also reduces vendor lock-in. A team can replace its embedding model, vector index, reranker, or language model without automatically changing what counts as authorized evidence or official truth. The intelligent ranking layer can evolve while the authority contract remains reviewable.

04 · The conceptual model

Relevance, truth, authority, and action are four different things.

Search systems become difficult to govern when they collapse these responsibilities into one score. OverLift separates them.

A semantic score belongs only to the first row. It is evidence about relevance. It is not a probability that the passage is true, not a guarantee that it is the best source, and not permission to use it.

The distinction is easier to see with an example. Suppose a user asks, “Can this customer receive an emergency credit?” A vector search may find a handbook passage discussing emergency credits. That passage may be highly relevant. It may also be expired, written for another region, superseded by a contract, hidden from the current employee, or embedded with prompt-injection text telling an agent to ignore policy. Relevance found the neighborhood. It did not decide the case.

The same logic applies to educational search. “What is the Azure service that lets me run SQL without managing the server?” may retrieve Azure SQL Database, SQL Managed Instance, and SQL Server on Azure Virtual Machines. All are semantically related. The right answer depends on the exact requirement. A useful system compares candidates, explains the differences, and asks for clarification when the query does not contain enough evidence to choose.

OverLift therefore treats a search result as a typed candidate with identity and evidence, not as a floating sentence:

{
  "candidateId": "term:azure-sql-database",
  "sourceId": "catalog:azure-glossary@2026-08-21",
  "retrieval": {
    "exactRank": null,
    "lexicalRank": 4,
    "semanticRank": 1,
    "graphPath": ["managed-database", "azure-sql-database"]
  },
  "authority": {
    "rights": "public",
    "freshness": "current",
    "sourceClass": "reviewed-canonical",
    "admitted": true
  }
}

The candidate can be reranked. The source identity, rights, and admission verdict cannot be invented by the reranker.

05 · Architecture

A trustworthy semantic pipeline is a sequence of boundaries—not one clever model call.

The architecture begins before embeddings and continues after ranking.

Human language
    ↓
Typed query plan
    ↓
Exact and lexical candidates
    ↓
Local semantic candidates
    ↓
Reviewed graph expansion
    ↓
Rights / freshness / source admission
    ↓
Deterministic fusion and stable tie-breaks
    ↓
Q-Lens comparison, clarification, or abstention
    ↓
Evidence-bound answer or bounded next action
    ↓
Citation, receipt, replay, and evaluation

Typed query planning

The raw query is not the whole request. A query planner can make the user’s intent explicit: subject, operation, time, entities, measures, evidence needs, source boundaries, permissions, desired answer shape, and whether the user is asking for information or proposing an action.

That plan should preserve ambiguity rather than erase it. “Latest,” “best,” “safe,” “eligible,” and “before” all require context. A planner may create several interpretations and ask for clarification instead of locking onto the first plausible one.

Candidate generation

Different retrieval systems catch different failures:

• Exact retrieval protects identifiers, names, commands, error codes, quoted language, and canonical terms.
• Lexical retrieval rewards overlapping words, rare terms, field weights, and phrase structure.
• Vector retrieval recovers conceptual similarity when vocabulary differs.
• Reviewed graph expansion follows explicit relationships such as prerequisites, alternatives, misconceptions, source lineage, and product dependencies.
• Learned reranking can improve ordering inside a bounded candidate set.

None of these methods should own eligibility by itself. They generate evidence about usefulness.

Admission before final rank

Candidate admission applies the facts that ranking models cannot safely infer: source rights, tenant, document lifecycle, effective date, product version, canonical status, integrity hash, conflict state, and purpose limitation. A result may be relevant and still be excluded.

Deterministic fusion

Once candidates are admitted, the product can combine signals through a stable formula or rule set. The exact weights are product-specific. What matters is that candidate identity, signal versions, missing-signal behavior, tie-breaks, and exclusion reasons are reproducible.

Answer and action boundary

Search may produce an answer, a comparison, a clarification question, an abstention, or a proposed next step. A proposed action still crosses the deterministic authority layer described in the companion article. Retrieval does not grant execution rights.

06 · Engineering

Start with the exact baseline. Add semantics where they win.

A common failure mode is replacing a comprehensible search system with a vector index before the team has a stable corpus, identifiers, relevance judgments, or failure taxonomy. That makes the demo more magical and the product harder to debug.

The stronger sequence is incremental.

1. Give every source a stable identity. Document, revision, page, chunk, term, graph edge, and record IDs must survive reranking and replay.
2. Build an exact and lexical baseline. Known identifiers and canonical terms should work even when semantic assets are missing.
3. Define chunking as a product decision. Preserve headings, tables, page anchors, section paths, timestamps, and relationships rather than cutting text by arbitrary token count alone.
4. Pin the embedding model and preprocessing. Tokenization, normalization, dimensions, quantization, pooling, and model hash belong in the corpus identity.
5. Keep raw scores in their own domains. A BM25 score, cosine similarity, graph weight, and semantic reranker score are not naturally comparable.
6. Fuse admitted ranks or calibrated signals deterministically. Missing signals and ties need explicit policy.
7. Canonicalize before deduplication. Merge exact duplicates, near duplicates, aliases, and overlapping passages without erasing meaningful disagreement.
8. Version caches and indexes. Corpus, model, graph, policy, and permissions changes must invalidate the right artifacts.
9. Record the path. Preserve which retrievers ran, which candidates appeared, which were excluded, how the final order was produced, and which citations entered the answer.

A deterministic fusion record might look like this:

{
  "schema": "overlift.rank-receipt@1",
  "queryHash": "...",
  "corpusVersion": "2026-08-21.1",
  "model": "all-MiniLM-L6-v2@751bff3-int8",
  "retrievers": ["exact", "bm25", "vector", "reviewed-graph"],
  "admissionPolicy": "search-evidence@7",
  "fusionPolicy": "deterministic-rank-fusion@4",
  "topCandidates": [
    { "id": "term:recall-at-10", "rank": 1, "reasonBits": 19 },
    { "id": "term:recall-at-5", "rank": 2, "reasonBits": 17 }
  ],
  "verdict": "ANSWER_WITH_CITATIONS"
}

This is not logging every hidden model activation. It is preserving the system-level evidence needed to explain and reproduce the result.

Stable tie-breaks matter

If two candidates have equal final scores, the system still needs one deterministic order. Stable canonical IDs, source priority, or explicit rank precedence prevent output drift across browsers, caches, and replay. Tiny rules become important when they influence citations and downstream actions.

Semantic failure should not erase search

If the model, Worker, GPU path, or vector pack is unavailable, exact and lexical search should remain useful. The product may label semantic readiness, degrade to a qualified baseline, or abstain from features that require meaning. It should not pretend that the full system ran.

07 · The OverLift Method

Reshape the search landscape. Do not rewrite the ground.

OverLift treats semantic intelligence as a way to change the shape of exploration without changing the underlying facts.

The Semantic Bridge is the interface between exact application state and meaning. It projects facts into typed semantic objects—intent, entities, relationships, evidence needs, uncertainty, source boundaries, tools, and possible next actions—without allowing that projection to replace the exact source.

Q-Lens then treats search as bounded pathfinding. One path may favor exact terminology. Another may favor a scenario match. Another may expand through a reviewed prerequisite graph. Another may conclude that the evidence conflicts. The system compares those paths under one authority boundary and can choose clarification or abstention as deliberately as an answer.

This is also where the localized-Higgs-gradient idea becomes practical. The system can make canonical identity “heavier,” synonyms “lighter,” current policy more influential, stale commentary less influential, and forbidden sources impossible to admit. The landscape changes. The ground truth does not.

Similarity proposes. Reviewed evidence, rights, time, policy, and deterministic authority decide.

08 · Browser-based agentic AI

The browser can understand private intent without sending the whole question somewhere else.

Semantic search is one of the strongest uses of a browser-local AI runtime. A compact embedding model can run in a Worker. Vectors and exact indexes can stay in IndexedDB, origin-private files, or immutable content-addressed packs. A focused WebAssembly kernel can perform stable top-k retrieval, filtering, graph traversal, or fusion. WebGPU may accelerate qualified inference. The interface can show readiness, evidence, citations, and fallback in the same place where the user asks the question.

That architecture creates useful privacy choices:

• The raw query can remain inside the selected browser origin.
• Private documents and vectors can remain local or inside a company-controlled deployment.
• Only a minimum necessary passage can leave the boundary when a hosted model is genuinely useful.
• A provider outage does not have to erase exact search, local vectors, graph discovery, or deterministic answers.
• Model and corpus assets can be versioned and hash-pinned for replay.

Local execution is not automatically private or secure. Browser extensions, injected scripts, weak origin controls, unsafe imports, copied content, analytics, and deliberate provider calls can still disclose data. The product must make egress, storage, permissions, and capability boundaries explicit.

Prompt injection also remains content. A retrieved passage may say “ignore the rules,” “call this tool,” or “reveal another document.” Those words can be quoted as evidence. They cannot create a capability, change admission policy, enlarge rights, or become official instruction.

09 · Product proof

The same search contract appears across very different OverLift products.

Agentic AI Glossary

The current hard-coded glossary contains 721 reviewed terms. Its semantic gold-master campaign covers 6,904 queries across questions, learning language, alternate wording, misspellings, prior failure language, and unsupported gibberish controls. The current qualified source report records a mean of about 3.21 ms, p95 of about 5.06 ms, and p99 of about 7.76 ms for that campaign in the packaging environment. Those numbers are evidence for one frozen corpus and runtime—not a universal browser promise.

AzureGlossary

Exact terminology, commands, scenarios, graph relationships, local MiniLM, deterministic fusion, and reviewed learning objectives work together. A novice can search by outcome or confusion, but canonical Azure concepts and command safety still remain explicit.

Veil

A user can ask natural-language questions about a chart. Local semantic routing helps find relevant houses, aspects, planets, retained readings, and “what changed” evidence. Exact chart positions remain separate from authored interpretation.

Saros

Research retrieval must preserve point-in-time evidence, source rights, filing dates, vintages, disagreement, exclusions, and what was knowable then. Semantic similarity helps discover evidence; it cannot silently replace temporal truth.

Vellucent

Page-aware document evidence, defined terms, KEEP cases, citations, policy, and local semantics support bounded grammar proposals. A related sentence cannot become permission to rewrite a number, obligation, quote, or official record.

OverLift Logistics

Agents may retrieve policy and narrative evidence while typed tools read exact operational state. RAG is not used as a substitute for an authoritative ERP query. Search can propose the investigation path; identity, policy, approval, and exact state control action.

10 · Technical treatise

What the scores mean—and what they do not

Cosine similarity

Dense retrieval commonly compares normalized embedding vectors. For query vector q and document vector d:

cosine(q, d) = dot(q, d) / (||q|| · ||d||)

A larger value indicates greater alignment in the model’s vector space. It does not establish factual correctness, authority, or calibrated probability. The score depends on the model, preprocessing, pooling, quantization, language, corpus, and query distribution.

Lexical scoring

BM25-style ranking rewards term matches while accounting for rarity and document length. It often remains an excellent baseline because exact names, codes, and rare phrases carry strong signal. Dense retrieval and lexical retrieval fail differently; that is why hybrid systems can outperform either one alone.

Reciprocal Rank Fusion

RRF combines ranked lists without pretending their raw scores share one scale:

RRF(d) = Σ 1 / (k + rank_r(d))

The constant k controls how strongly top ranks dominate. RRF is useful, but it is still a ranking method. Rights, freshness, source authority, and eligibility should be applied before or alongside the fusion contract—not inferred from rank.

Deterministic weighted fusion

Some products need more explicit control than rank-only fusion. A product may assign stable weights to exact identity, lexical evidence, semantic similarity, graph relations, source class, freshness, and reviewed quality. The formula should define missing signals, caps, normalization, ties, and the conditions under which a learned score is allowed to participate.

finalScore =
    exactIdentityBoost
  + lexicalWeight      · lexicalSignal
  + semanticWeight     · semanticSignal
  + graphWeight        · reviewedGraphSignal
  + sourceWeight       · admittedSourceQuality
  + freshnessWeight    · freshnessSignal
  - duplicatePenalty
  - uncertaintyPenalty

A candidate excluded for rights or lifecycle reasons should not be rescued by a large semantic score.

Approximate nearest neighbors

Large vector collections often use approximate nearest-neighbor indexes such as HNSW to trade some recall for speed and memory efficiency. That trade must be measured against an exhaustive or otherwise trusted reference on representative queries. A faster candidate generator does not earn authority merely by being fast.

Candidate depth and reranking

Rerankers can only improve candidates they receive. A powerful semantic reranker cannot recover a relevant document that the first-stage retrieval omitted. Candidate depth therefore affects quality, latency, memory, and cost. Evaluate the first-stage retriever and final ranker separately.

Chunk identity

Chunking changes the retrieval problem. A chunk should retain document ID, revision, page or section path, source offsets, effective time, rights, neighboring context, and canonical parent. Otherwise citations become fragile and deduplication becomes guesswork.

Graph expansion

Graphs can expose relationships embeddings may blur: prerequisite, successor, supersedes, contradicts, example-of, command-for, policy-governs, or same-entity-as. These edges should be reviewed, versioned, and explainable. A learned graph can propose edges, but promoted relationships need governance when they influence authoritative answers.

11 · Evaluation

Measure retrieval, ranking, citations, abstention, duplicates, latency, and authority separately.

“The search felt good” is useful playtest feedback. It is not a release gate.

A serious evaluation corpus should include exact identifiers, natural questions, misspellings, novice language, scenario descriptions, temporal questions, comparisons, multi-hop relationships, hard negatives, permission failures, stale sources, conflicting evidence, prompt injection, unsupported gibberish, and queries that should trigger clarification or abstention.

Freeze the corpus, chunking, source revisions, model, tokenizer, vector dimensions, graph, candidate depths, fusion policy, and relevance judgments for each run. Record ranks after every stage: exact/lexical retrieval, vector retrieval, graph expansion, admission, fusion, reranking, and final answer selection.

Recall@5 and Recall@10

Recall@k asks how much of the judged relevant evidence appears in the first k results:

Recall@k = relevant items retrieved in top k / all judged relevant items

Recall@5 reflects a tight interaction surface. Recall@10 gives the system more room. High recall is necessary when downstream reranking or answer synthesis needs at least one relevant passage, but recall alone does not reward good ordering and can hide noisy result sets.

Mean Reciprocal Rank

MRR emphasizes where the first relevant result appears:

RR(query) = 1 / rank of first relevant result
MRR       = mean RR across queries

It is useful when one strong answer near the top matters. It says little about the quality of the remaining results and can over-reward systems that find one relevant item while missing broader evidence.

nDCG@10

Normalized Discounted Cumulative Gain rewards graded relevance and discounts useful items that appear later:

DCG@k  = Σ (2^relevance_i - 1) / log2(i + 1)
nDCG@k = DCG@k / idealDCG@k

nDCG@10 is appropriate when several results have different degrees of usefulness. It depends on the quality and consistency of the relevance judgments.

Citation correctness

Retrieval quality and answer quality are different. Citation correctness asks whether the cited source actually supports the claim, whether the anchor points to the right passage, and whether the source is admitted for the current answer. It is useful to track at least:

• Citation precision: the share of citations that genuinely support the associated claim.
• Citation coverage: the share of externally checkable claims that have adequate support.
• Source correctness: the share of citations using the right version, authority, rights, and temporal scope.
• Anchor correctness: the share of citations pointing to the precise supporting passage rather than a loosely related document.

Abstention

A system should be rewarded for refusing unsupported certainty. Measure supported abstention on queries where evidence is missing, conflicting, unauthorized, stale, or genuinely ambiguous. Also measure false abstention, where the system refuses despite sufficient evidence.

Duplicate-result measurements

Duplicates waste scarce top-k positions and create the illusion of agreement. Measure exact duplicates, canonical duplicates, near-duplicate chunks, and conceptual duplicates that restate the same evidence. Keep legitimate disagreement separate; two sources should not be merged merely because they discuss the same topic.

Authority and operational metrics

Track permission failures blocked, stale-source exclusions, unsupported-action blocks, replay agreement, deterministic tie stability, model readiness, provider use, cache hits, memory, and latency percentiles. A relevance improvement that doubles p99 latency or bypasses rights is not automatically a product improvement.

Most importantly, inspect query families rather than one aggregate. A model may improve conversational questions while damaging identifiers, temporal queries, permissions, or hard negatives. The average can hide the failure that matters most.

12 · Failure behavior

A trustworthy search system has explicit behavior when meaning is unavailable, ambiguous, or unsafe.

13 · Honest limits

What semantic search does not magically solve

• Embeddings inherit the blind spots, language coverage, and biases of their training and fine-tuning.
• A stronger reranker cannot recover evidence omitted by first-stage retrieval.
• Approximate indexes can trade recall for speed and require representative measurement.
• Chunking can destroy context, citations, tables, reading order, and document meaning.
• Human relevance judgments can be incomplete, inconsistent, or too narrow for production traffic.
• Graph relationships can become stale or encode editorial bias.
• Local execution does not automatically guarantee privacy, security, or accessibility.
• Citations can be precise and still point to an incorrect upstream source.
• Abstention thresholds can be too timid or too aggressive.
• Deterministic fusion can faithfully reproduce a bad policy.
• No offline benchmark fully predicts every user, device, language, corpus change, or adversarial query.

The goal is not to turn search into an infallible oracle. It is to make relevance powerful, truth boundaries explicit, failures measurable, and consequential use governable.

14 · Implementation guide

A practical checklist for semantic search that deserves trust

1. Name the user task. Product discovery, learning, research, support, evidence retrieval, and action planning need different relevance judgments.
2. Define canonical source identity. Preserve document, revision, page, section, term, record, tenant, and temporal scope.
3. Build exact and lexical baselines first. Protect identifiers, commands, quoted language, and fallback behavior.
4. Create representative query families. Include natural language, synonyms, misspellings, scenarios, temporal questions, hard negatives, permissions, and abstention cases.
5. Review chunking. Retain structure and citation anchors; do not assume arbitrary token windows are neutral.
6. Pin model and preprocessing identity. Version tokenizer, normalization, dimensions, pooling, quantization, and hashes.
7. Keep retriever scores separate. Do not compare unrelated raw scales as if they were probabilities.
8. Admit evidence explicitly. Rights, freshness, lifecycle, source class, integrity, and purpose belong outside the embedding model.
9. Fuse deterministically. Define weights or rank fusion, missing signals, caps, penalties, and stable tie-breaks.
10. Deduplicate deliberately. Remove wasted repetition without erasing independent sources or disagreement.
11. Make clarification and abstention first-class. Measure both supported and false abstention.
12. Bind citations to exact anchors. Preserve source version and passage identity.
13. Separate answers from actions. Search can propose; typed authority decides whether anything changes.
14. Measure every stage. Recall@5, Recall@10, MRR, nDCG@10, citation correctness, duplicates, permissions, stale exclusions, latency, and replay.
15. Compare query families. Do not let one aggregate hide damage to identifiers, temporal search, or safety cases.
16. Test failure and recovery. Missing models, corrupt packs, Worker restarts, stale caches, provider outages, and path divergence are release cases.
17. Preserve a search receipt. Record query identity, corpus, models, retrievers, exclusions, fusion, final candidates, citations, and verdict.

16 · Why it matters

The future of search should not be a prettier way to sound certain.

Semantic systems are becoming cheaper, faster, more local, and easier to attach to every product. That is good. People should not have to speak like databases. They should be able to search with incomplete memory, ordinary language, examples, symptoms, and intent.

But understanding language is only the first half of the product. The second half is deciding which evidence deserves to shape the answer, which uncertainty must remain visible, which sources are excluded, which citations prove the claim, and where search must stop before authority begins.

That is why OverLift does not ask semantic search to become truth. It asks semantic search to become an excellent explorer inside a governed evidence system.

Embeddings recover meaning. Evidence and authority decide what deserves belief.

A trustworthy search product does more than find related text. It helps people discover what they meant, understand why the result belongs, see what the source establishes, and know when the system should ask, abstain, or refuse.

Explore OverLift Semantic Search Read Deterministic Authority Read Browser-Based Agentic AI