Edit

Share via

Hybrid search in Cosmos DB in Microsoft Fabric (preview)

  • Applies to: ✅ Cosmos DB in Fabric

Important

This feature is in preview.

Cosmos DB in Microsoft Fabric supports the hybrid search capability that combines Vector Search with Full Text Search scoring (BM25) using the Reciprocal Rank Fusion (RRF) function.

Hybrid search uses the strengths of both vector-based and traditional keyword-based search methods to deliver more relevant and accurate search results. Hybrid search is easy to do in Cosmos DB in Fabric due to the ability to store both metadata and vectors within the same document.

Hybrid search in Cosmos DB in Fabric integrates two distinct search methodologies:

  • Vector search: Utilizes machine learning models to understand the semantic meaning of queries and documents. This methodology allows for more nuanced and context-aware search results, especially useful for complex queries where traditional keyword search might fall short.

  • Full text search (BM25): A well-established algorithm that scores documents based on the presence and frequency of words and terms. BM25 is effective for straightforward keyword searches, providing a robust baseline for search relevance.

The results from vector search and full text search are then combined using the Reciprocal Rank Fusion (RRF) function. RRF is a rank aggregation method that merges the rankings from multiple search algorithms to produce a single, unified ranking. This method ensures that the final search results benefit from the strengths of both search approaches and offers multiple benefits.

  • Enhanced Relevance: Hybrid search delivers more relevant results for a wide range of queries by combining semantic understanding with keyword matching.

  • Improved Accuracy: The RRF function ensures that the most pertinent results from both search methods are prioritized.

  • Versatility: Suitable for various use cases including Retrieval Augmented Generation (RAG) to improve the responses generated by a large language model (LLM) grounded on your own data.

Review this sample indexing policy with both full text and vector indexes:

{
  "indexingMode": "consistent",
  "automatic": true,
  "includedPaths": [
    {
      "path": "/*"
    }
  ],
  "excludedPaths": [
    {
      "path": "/\"_etag\"/?"
    },
    {
      "path": "/vector/*"
    }
  ],
  "fullTextIndexes": [
    {
      "path": "/text"
    }
  ],
  "vectorIndexes": [
    {
      "path": "/vector",
      "type": "DiskANN"
    }
  ]
}

Important

Currently, vector policies and vector indexes are immutable after creation. To make changes, create a new collection. Other indexes still remain mutable.

Hybrid search queries

Hybrid search queries can be executed by using the RRF system function in an ORDER BY RANK clause that includes both a VECTORDISTANCE function and FULLTEXTSCORE. For example, a parameterized query to find the top k most relevant results would look like:

SELECT TOP @k
  *
FROM
  container c
ORDER BY
  RANK RRF(VECTORDISTANCE(c.vector, @queryVector), FULLTEXTSCORE(c.content, @searchTerm1, @searchTerm2, ...))

Suppose you have a document that has vector embeddings stored in each document in the property c.vector and text data contained in the property c.text. To get the 10 most relevant documents using Hybrid search, the query can be written as:

SELECT TOP 10
  * 
FROM
  container c
ORDER BY
  RANK RRF(VECTORDISTANCE(c.vector, [1,2,3]), FULLTEXTSCORE(c.text, "searchable", "text", "goes" ,"here"))

Weighted hybrid search queries

You can optionally specify an array of weights to affect how important each component score is in the RRF function. For example, if you have two component scores (VECTORDISTANCE and FULLTEXTSCORE) and want to weight the vector search twice as important as the BM25 scoring, you can add the array of numbers [2, 1] as the last argument to RRF as shown here:

SELECT TOP 10
  *
FROM
  container c
ORDER BY
  RANK RRF(VECTORDISTANCE(c.vector, [1,2,3]), FULLTEXTSCORE(c.text, "searchable", "text", "goes" ,"here"), [2, 1])

Related content