Search for AddStandardResilienceHandler and you probably want the documentation containing that API. Search for “stop calling an unhealthy service for a while” and you may want the same article, even though the useful passage talks about a circuit breaker.
Those queries ask different things of a search engine. One supplies the vocabulary. The other supplies the intent. A retrieval system needs to handle both before an LLM can write a useful answer from its results.
In this issue, we build a .NET 11 application that searches real .NET documentation through Azure AI Search, using Microsoft Foundry for embeddings. We compare keyword search, vector search, hybrid search, and hybrid search with semantic reranking. The interesting part is watching which passages move, understanding why, and deciding whether another retrieval stage earns its cost.
What You Are Building
The companion repository is a command-line retrieval application with an .slnx solution. It downloads twelve official articles covering caching, HTTP resilience, HTTP clients, logging, options, dependency injection, worker services, and channels. Every source is pinned to a Git revision and content hash.
Four modes search the same index:
- Keyword: BM25 ranks searchable text.
- Vector: a Foundry query embedding searches the vector field.
- Hybrid: Azure AI Search combines lexical and vector rankings.
- Semantic: a managed reranker reorders the hybrid candidates using their text.
The model and index run in Azure. The .NET process prepares documents, sends requests, and presents results. We stop at ranked passages so answer generation does not obscure what retrieval contributed.
System Structure
At ingestion, the application chunks the documentation, embeds each chunk through Foundry, and uploads its text and vector to Azure AI Search. At query time, text takes a lexical path and its embedding takes a vector path. This diagram shows the complete hybrid-plus-reranking path; the simpler modes use fewer stages.
Azure AI Search performs the retrieval algorithms. We send explicit REST requests so you can inspect what changes between modes. The Foundry client adapts the endpoint and dimension settings from issue #46, and the request-scoped authentication and error handling used in issue #50. Here, the calls generate embeddings rather than agent decisions.
Runtime Configuration First
You need the .NET 11 SDK, an Azure AI Search service on Basic or higher with semantic ranking available, and a Foundry deployment of text-embedding-3-small. Supply your actual deployment name. The endpoint is the resource root; the client adds /openai/v1/embeddings. Run the following commands from the companion repository root. The global.json file selects SDK 11.0.100 or a later patch; check dotnet --version there before restoring.
$env:HYBRID_Search__Endpoint = 'https://YOUR-SEARCH-SERVICE.search.windows.net'
$env:HYBRID_Search__IndexName = 'dotnet-hybrid-20261121'
$env:HYBRID_Foundry__Endpoint = 'https://YOUR-FOUNDRY-RESOURCE.openai.azure.com'
$env:HYBRID_Foundry__EmbeddingDeployment = 'YOUR-EMBEDDING-DEPLOYMENT'
$env:HYBRID_Foundry__Dimensions = '1536'
az login
dotnet restore HybridRetrieval.slnx --locked-mode
dotnet test HybridRetrieval.slnx -c Release --no-restoreAzure Identity supplies authentication by default; the README lists the required roles. Separate environment variables also support service API keys. Settings load from appsettings.json, an optional ignored appsettings.Local.json, and environment overrides. No chat deployment is required: Azure AI Search supplies the semantic ranking model.
For key authentication, set HYBRID_SEARCH_API_KEY to a Search admin key and HYBRID_FOUNDRY_API_KEY to the Foundry resource key in the same terminal. In that case, az login is unnecessary. The application creates its own index and semantic configuration during ingestion; no portal document import or knowledge base is required.
BM25: Start with the Words
Engineers often know the exact thing they need: an API name, exception type, configuration property, or product term. BM25, short for Best Matching 25, ranks text by its query terms. This is lexical relevance: it uses words and their frequencies, without generating embeddings or invoking a semantic model.
It considers how often a term appears in a document, how common that term is across the collection, and document length. Repeated occurrences help, but their contribution saturates. Rare terms contribute more than common ones. In this index, each chunk is a search document, so these statistics describe chunks rather than whole source articles.
The index sets k1 to 1.2 and b to 0.75. The first affects term-frequency saturation; the second affects length normalization. These are starting values, not tuned optima for this corpus.
"similarity": {
"@odata.type": "#Microsoft.Azure.Search.BM25Similarity",
"k1": 1.2,
"b": 0.75
}Title and content use the en.microsoft analyzer. Lexical search operates on analyzed terms, not literal substrings. Tokenization and stemming can change how identifiers and punctuation behave. Inspect the API-name queries before assuming an analyzer preserves technical terms as you expect.
Vectors: Search for the Meaning
A reader may describe a behavior without knowing its name. Embeddings represent both the query and document in a space where related meanings can be close, even when their wording differs.
Foundry embeds each title followed by chunk content. At query time it embeds the question. Azure AI Search retrieves nearby vectors using cosine distance and an HNSW index. HNSW provides approximate nearest-neighbor retrieval; the application does not scan vectors itself.
var payload = new JsonObject
{
["model"] = settings.Foundry.EmbeddingDeployment,
["input"] = new JsonArray(
inputs.Select(s => (JsonNode?)JsonValue.Create(s)).ToArray()),
["dimensions"] = settings.Foundry.Dimensions,
["encoding_format"] = "float"
};Documents and queries must use the same deployment and dimensions. The response parser maps batch vectors by the provider's index field instead of assuming response order matches input order.
Similarity can still retrieve an article about the general area while missing the particular method or error in the question. That is why the lexical baseline stays in the comparison.
Hybrid Retrieval: Combine Rankings
BM25 and vector scores have different scales. Adding their raw values would give the larger numerical scale an arbitrary advantage. Reciprocal rank fusion works with positions instead.
Each result list containing a document contributes a value of the form 1 / (c + rank). RRF sums those contributions. A document ranking well in both lists receives support from both methods. The fusion constant is separate from the vector-neighbor count and the number of displayed results.
Fusion happens at chunk level. Only after Azure has combined and optionally reranked the chunks does the application collapse repeated source articles. The same article appearing in both retrieval paths does not guarantee that the same chunk receives both contributions.
The application uses Azure AI Search's native implementation. Its request builder adds the text query for lexical and hybrid modes, and a real Foundry vector for vector-bearing modes:
if (mode != RetrievalMode.Vector)
payload["search"] = query;
if (mode != RetrievalMode.Keyword)
{
payload["vectorQueries"] = new JsonArray(new JsonObject
{
["kind"] = "vector",
["vector"] = JsonSerializer.SerializeToNode(vector),
["fields"] = "contentVector",
["k"] = settings.Retrieval.VectorCandidates
});
}When both are present, Azure runs the lexical and vector searches and fuses their rankings internally. We retain their returned order instead of applying a second homemade fusion step.
Reranking: Read the Candidates with the Query
The first retrieval stages find candidates efficiently. A reranker spends more work on a smaller set, evaluating the relationship between the question and each candidate's text.
Semantic mode adds these fields to the hybrid request. The named semantic configuration tells the service to read the title and content fields.
payload["queryType"] = "semantic";
payload["semanticConfiguration"] = settings.Search.SemanticConfiguration;
payload["semanticErrorHandling"] = "fail";
// Do not set semanticMaxWaitInMilliseconds with error handling set to fail.Azure's managed ranker considers up to fifty initial candidates and returns a separate @search.rerankerScore. It can move a useful passage above a superficially similar one. It cannot recover a passage absent from that candidate set.
The console's score field means BM25 relevance in keyword mode, a vector-search score in vector mode, and RRF score in hybrid modes. Semantic mode also prints reranker, which explains its final ordering. These numbers use different scales and are not probabilities of correctness. A smaller RRF number is not evidence of a worse result than a larger BM25 number.
Candidate retrieval determines what the reranker can see. Reranking determines how that selected material is ordered. A stronger final language model does not remove the need to inspect either stage.
Chunking Changes What You Can Find
The splitter starts with at most 2,200 characters per chunk and 250 characters of overlap. It prefers paragraph boundaries in the latter half of a chunk. It is character-based, not a tokenizer or Markdown parser, and can split code blocks. Code-aware splitting would be a useful extension.
The cleaner removes frontmatter, comments, and legacy bracket-style include/code directives. It does not fully render Microsoft documentation Markdown: colon-style code directives and xref markup can remain in the indexed text, and referenced snippets are not fetched. The console output exposes some of that noise. Some examples visible on Microsoft Learn are therefore absent from the indexed source, while other chunks contain directive text instead of the referenced example.
Every mode requests up to fifty returned chunks by default. The application then keeps the first result from each source and displays up to five distinct documents. Otherwise, overlapping chunks from one long article could fill the list. A keyword query can return fewer matches, and a candidate pool dominated by one article can produce fewer than five distinct sources.
public static SearchHit[] Sources(IEnumerable<SearchHit> hits, int k) => hits
.DistinctBy(x => x.SourceId, StringComparer.Ordinal)
.Take(k)
.ToArray();This makes the comparison about source discovery. Finding the right article does not prove the selected chunk contains the exact evidence an answer needs. Read the passages too.
Run the Comparison Against Your Services
Download the documentation and build your cloud index. The sources command verifies the pinned content without calling Azure. The ingest command performs real Foundry requests and Search uploads. Run ingestion once for a new index; comparisons and evaluations reuse it. Repeating ingestion repeats the embedding work, so it is not a prerequisite for every test run.
dotnet run --project src/HybridRetrieval -c Release -- sources
dotnet run --project src/HybridRetrieval -c Release -- ingest
dotnet run --project src/HybridRetrieval -c Release -- compare 'AddStandardResilienceHandler'
dotnet run --project src/HybridRetrieval -c Release -- compare 'Stop calling an unhealthy downstream service for a while'Inspect how the first query handles the API identifier. For the second, look for the useful circuit-breaker passage even though the question avoids that name. Then inspect the hybrid and reranked orders. These are questions to investigate in your output, not assumed benchmark wins.
The quoted text after compare is the search query. Replace it with your own engineering question to run that same text through all four modes. Evaluation uses a different input source: the twelve questions and graded relevance judgments in data/queries.json. Changing the repetition count repeats that file's questions rather than generating new ones.
Each comparison prints titles, URLs, passage previews, provider scores, and search latency, then saves complete returned chunks in JSON. One query embedding is shared across the three vector-based modes. BM25 itself needs no embedding call.
Measure Relevance and the Extra Work
The repository includes twelve authored queries covering API names, paraphrases, and mixed technical language. Labels identify relevant source documents and assign grades from one to three. They are starting judgments for this corpus, not independently reviewed ground truth or production traffic.
dotnet run --project src/HybridRetrieval -c Release -- evaluate 3The JSON and Markdown reports expose several aspects of the ranking:
- Recall@K: how many labeled relevant sources appear in the first K distinct sources?
- MRR, mean reciprocal rank: the average of one divided by the first relevant source's position, with zero when none is found within K.
- nDCG@K, normalized discounted cumulative gain: how closely does the ordering match the ideal ranking under the graded relevance labels?
- Latency: what client-observed time do search and query embedding require?
The evaluator rotates request order. It times search separately and estimates standalone latency by adding the shared embedding observation to vector-based modes. Measurements include networking, authentication, and retries. There is no explicit warmup, and a small run cannot establish a dependable production p95.
A p50 is the median observation; p95 describes the slower end of the sample. The comparison host always generates the shared embedding before running its modes, including keyword mode. The keyword timing measures that Search request alone. Its estimated standalone timing describes the lexical path without an embedding call, not the total wall-clock duration of the comparison command.
Three repetitions make 144 Search requests, including 36 semantic requests, plus 36 query-embedding calls. Ingestion is extra. Check your quotas and pricing before expanding the experiment.
Inspect each query category before trusting the average. A strategy can improve paraphrases while making identifiers harder to find. Check the labels too: an unlisted article may contain a valid answer. Correct that judgment before treating its retrieval as a failure.
What the Live Run Showed
The cloud run on 21 November 2026 completed all four modes against 134 chunks from the twelve documentation articles. It used the Foundry deployment text-embedding-3-small at 1,536 dimensions, fifty vector candidates, and five distinct sources per result list. The saved evaluation used evaluate 3: three passes through twelve queries, producing 144 Search requests. These are repeated observations of twelve questions, not 36 independent questions.
The JSON report and its Markdown summary in artifacts/ recorded the following averages and observed search-request p50 values. Recall and nDCG are measured at five distinct source documents.
These values come from the completed three-pass report recorded at 05:44 UTC. The earlier one-pass run produced the same aggregate relevance scores with different timings. The repeated run adds observations, not broader query coverage. A score of 1.000 means the metric reached its maximum under these labels; it does not mean generated answers are always correct.
| Mode | Recall@5 | nDCG@5 | Search p50 |
|---|---|---|---|
| Keyword | 0.917 | 0.875 | 18.3 ms |
| Vector | 1.000 | 0.899 | 21.0 ms |
| Hybrid | 1.000 | 0.907 | 27.9 ms |
| Hybrid + semantic | 1.000 | 0.993 | 84.7 ms |
Search time alone leaves out an important cost. After adding each query's shared embedding observation before computing the percentile, estimated standalone p50 was 199.5 ms for vector search, 205.1 ms for hybrid, and 261.0 ms for hybrid plus semantic ranking. Keyword search remained at 18.3 ms because it needs no query embedding. These are observations from this small run, not service latency guarantees.
The clearest vocabulary mismatch was “Detect invalid configuration before the application begins serving work.” Keyword search missed both labeled sources in its top five. Vector and hybrid search found both, but hybrid placed the first relevant source fourth. Semantic reranking moved options to first place and raised nDCG@5 from 0.450 to 0.932. Recall alone would have hidden that improvement in ordering.
The outage query told a similar story: “Stop sending requests to an unhealthy downstream service for a while and then try it again.” Keyword search placed the resilience article third; vector and hybrid placed it second; semantic reranking placed it first. All four had recall@5 of 1.000, while nDCG@5 moved from 0.500 for keyword to 0.631 for vector and hybrid, then 1.000 after reranking. This evaluation query is longer than the separate interactive comparison above, so their rankings should not be treated as the same observation.
The extra stage did not improve everything. All four modes put the relevant source first for the four API-name queries. For “Configure HTTP client logging and redact request headers,” keyword search achieved nDCG@5 of 1.000, while semantic ranking scored 0.983. Hybrid also had slightly higher average nDCG than vector search but lower MRR: 0.896 versus 0.917. The best overall ordering in this run came from semantic reranking, but neither fusion nor reranking was a universal improvement.
This is a small source-discovery experiment with authored relevance labels. With only twelve source articles and a five-source result list, high recall is easier to achieve than it would be over a large corpus. It does not establish passage-level answer quality or reliable production latency. It does demonstrate why I would inspect vocabulary mismatches and ranking changes before choosing a more expensive retrieval path.
Why This Architecture Works
Each stage has a distinct job. BM25 uses the supplied vocabulary. Embeddings bridge differences in wording. RRF combines the rankings. The semantic model reads the shortlisted text against the question.
Keeping the corpus, chunks, and queries consistent makes those contributions easier to examine. Hybrid plus reranking may be useful. A cheaper lexical query may already suffice for a particular class of requests. The application exposes the passages and measurements needed to make that choice.
Potential Enhancements
I would first resolve the documentation directives into readable text and referenced examples, then improve code-aware chunking, inspect analyzer behavior on identifiers, and add independently reviewed relevance judgments. Candidate-depth experiments would distinguish passages never retrieved from passages retrieved but ranked poorly.
A larger corpus would make latency and candidate selection more informative. Passage-level relevance would also deserve attention before connecting retrieval to answer generation. Finding an article and finding the exact evidence span are different measurements.
Final Notes
Better retrieval begins with understanding the failure. A missing term, a vocabulary mismatch, a poor candidate list, and a weak final ordering call for different changes.
Build the lexical baseline, add vectors, inspect the fusion, and measure what reranking contributes. The companion solution keeps those steps visible, using real cloud retrieval and the same documentation throughout.
Explore the companion repository at the GitHub repository.
See you in the next issue.
Stay curious.
