Semantic search

📘
Note
Semantic search is an experimental feature and may be subject to change.

Optimizely Graph supports matching and ranking of content beyond the standard lexical (literal keyword) matching using artificial intelligence (AI). The intent of the user query can be captured by extending it with context. The meaning of words can be part of that context. Language models have been created for that purpose. We can extend keyword matching with pre-trained language models. This approach to search is commonly called semantic search, but other names given to it are for example (but not necessarily synonyms) neural search, vector search, and dense neural retrieval. For more technical background, see the blog post, Do you know what I mean? Introducing Semantic Search in Optimizely Graph.

Semantic search is a solution for the so-called vocabulary mismatch problem. The keywords entered by a user like a site visitor may not match with the words used by the content marketer. For example, a site visitor may enter in the search box "non-alcoholic cold beverage" and expect content items about "cola" being returned. With standard search, we do not return content if the keyword used in the query do not appear in the content, but with semantic search we can return content because it "knows" about the context of the query.

One way to solve the vocabulary mismatch problem is by creating synonyms, which Optimizely Graph supports. This is very effective, but can be time-consuming. Semantic search will be a good way to automate the expansion of queries with synonyms and improve the ranking of results. Other important use-case that this technology supports is supporting conversational AI (chatbots) by feeding it with relevant results which solves hallucination (a technique called Retrieval Augmented Generation). We can also use the vector search technology that drives semantic search to cluster or de-duplicate content and detect anomalies.

This technology to do seach has proven to be working the best in combination with traditional keyword search. That is, the system can return most relevant content in the right order that users want. This is what Optimizely Graph supports and refers to as semantic search, but which is effectively a mixed approach between standard keyword search and pure vector search.

📘
Note
For existing accounts to use semantic search, you must reset your existing account. See the reset account functionality instructions.

How does Semantic Search work in Optimizely Graph?

Semantic search is supported on searchable string fields, and for the full-text search operators contains and match. It is recommended to set fields that have a lot of content (such as the MainBody in the Optimizely CMS) as searchable to unlock the full-text search capabilities. Optimizely Graph uses a pre-trained model for semantic search.

Semantic search can be enabled simply by changing the value of _ranking in the OrderBy to SEMANTIC. By default Optimizely Graph uses RELEVANCE that is standard keyword search with BM25 relevance ranking. You can combine SEMANTIC with other ranking criteria, for instance, use the semantic search capabilities but rank the results by field value instead.

Examples

Given the following query which the user wants content that are about action movie:

{
  Content(orderBy: { _ranking: SEMANTIC }, where: { MainBody: { contains: "action movie" } }) {
    total
    items {
      _score
      Name
      MainBody
      _fulltext
    }
  }
}

Optimizely Graph will correctly the most relevant content to the top. Note that both "action" and "movie" do not appear in the content.

{
  "Content": {
    "total": 4,
      "items": [
        {
          "_score": 0.013479907,
          "Name": "Standard Page 12",
          "MainBody": "Wild Wild West is a 1999 American steampunk Western film co-produced and directed by Barry Sonnenfeld and written by S. S. Wilson and Brent Maddock alongside Jeffrey Price and Peter S. Seaman, from a story penned by brothers Jim and John Thomas. Loosely adapted from The Wild Wild West, a 1960s television series created by Michael Garrison, it is the only production since the television film More Wild Wild West (1980) to feature the characters from the original series. The film stars Will Smith (who previously collaborated with Sonnenfeld on Men in Black two years earlier in 1997) and Kevin Kline as two U.S. Secret Service agents who work together to protect U.S. President Ulysses S. Grant (Kline, in a dual role) and the United States from all manner of dangerous threats during the American Old West.",
          "_fulltext": [
            "Wild Wild West is a 1999 American steampunk Western film co-produced and directed by Barry Sonnenfeld and written by S. S. Wilson and Brent Maddock alongside Jeffrey Price and Peter S. Seaman, from a story penned by brothers Jim and John Thomas. Loosely adapted from The Wild Wild West, a 1960s television series created by Michael Garrison, it is the only production since the television film More Wild Wild West (1980) to feature the characters from the original series. The film stars Will Smith (who previously collaborated with Sonnenfeld on Men in Black two years earlier in 1997) and Kevin Kline as two U.S. Secret Service agents who work together to protect U.S. President Ulysses S. Grant (Kline, in a dual role) and the United States from all manner of dangerous threats during the American Old West.",
            "Standard Page 12"
          ]
        },
        {
          "_score": 0.011364636,
          "Name": "Temporary Page Title",
          "MainBody": "The American frontier, also known as the Old West, popularly known as the Wild West, encompasses the geography, history, folklore, and culture associated with the forward wave of American expansion in mainland North America that began with European colonial settlements in the early 17th century and ended with the admission of the last few contiguous western territories as states in 1912. This era of massive migration and settlement was particularly encouraged by President Thomas Jefferson following the Louisiana Purchase, giving rise to the expansionist attitude known as \"Manifest Destiny\" and the historians' \"Frontier Thesis\". The legends, historical events and folklore of the American frontier have embedded themselves into United States culture so much so that the Old West, and the Western genre of media specifically, has become one of the defining periods of American national identity.",
          "_fulltext": [
            "The American frontier, also known as the Old West, popularly known as the Wild West, encompasses the geography, history, folklore, and culture associated with the forward wave of American expansion in mainland North America that began with European colonial settlements in the early 17th century and ended with the admission of the last few contiguous western territories as states in 1912. This era of massive migration and settlement was particularly encouraged by President Thomas Jefferson following the Louisiana Purchase, giving rise to the expansionist attitude known as \"Manifest Destiny\" and the historians' \"Frontier Thesis\". The legends, historical events and folklore of the American frontier have embedded themselves into United States culture so much so that the Old West, and the Western genre of media specifically, has become one of the defining periods of American national identity.",
            "Temporary Page Title"
          ]
        },
        {
          "_score": 0.011052586,
          "Name": "Wilder Westen",
          "MainBody": "Wilder Westen ist eine – geographisch und historisch grob eingegrenzte – umgangssprachliche Bezeichnung für die ungefähr westlich des Mississippi gelegenen Gebiete der heutigen Vereinigten Staaten. In der auch als „Pionierzeit“ bezeichneten Ära des 19. Jahrhunderts waren sie noch nicht als Bundesstaaten in die Union der Vereinigten Staaten aufgenommen. Im Verlauf der voranschreitenden Landnahme und Urbanisierung nahm die Besiedlung dieser Regionen vor allem durch Angloamerikaner – bzw. aus Europa stammende Immigranten – kontinuierlich zu, bis die Gebiete um 1890 in den organisierten Territorien der Vereinigten Staaten aufgingen. Symbolisch stehen die Öffnung der letzten Indianerterritorien im späteren US-Bundesstaat Oklahoma für die Besiedlung durch Kolonisten 1889–1895 durch eine Serie von Land Runs und das Massaker der United States Army an etwa 200 bis 300 Lakota am Wounded Knee Creek/South Dakota im Dezember 1890 für das Ende der Zeit des Wilden Westens. Mit diesen Ereignissen galten die Indianerkriege ebenso als abgeschlossen wie die Kolonisation der bis dahin von den Vereinigten Staaten beanspruchten Hoheitsgebiete (engl. territories) durch die aus Europa eingewanderten Siedler.",
          "_fulltext": [
            "Wilder Westen ist eine – geographisch und historisch grob eingegrenzte – umgangssprachliche Bezeichnung für die ungefähr westlich des Mississippi gelegenen Gebiete der heutigen Vereinigten Staaten. In der auch als „Pionierzeit“ bezeichneten Ära des 19. Jahrhunderts waren sie noch nicht als Bundesstaaten in die Union der Vereinigten Staaten aufgenommen. Im Verlauf der voranschreitenden Landnahme und Urbanisierung nahm die Besiedlung dieser Regionen vor allem durch Angloamerikaner – bzw. aus Europa stammende Immigranten – kontinuierlich zu, bis die Gebiete um 1890 in den organisierten Territorien der Vereinigten Staaten aufgingen. Symbolisch stehen die Öffnung der letzten Indianerterritorien im späteren US-Bundesstaat Oklahoma für die Besiedlung durch Kolonisten 1889–1895 durch eine Serie von Land Runs und das Massaker der United States Army an etwa 200 bis 300 Lakota am Wounded Knee Creek/South Dakota im Dezember 1890 für das Ende der Zeit des Wilden Westens. Mit diesen Ereignissen galten die Indianerkriege ebenso als abgeschlossen wie die Kolonisation der bis dahin von den Vereinigten Staaten beanspruchten Hoheitsgebiete (engl. territories) durch die aus Europa eingewanderten Siedler.",
            "Wilder Westen"
          ]
        },
        {
          "_score": 0.009704749,
          "Name": "Arnold Schwarzenegger",
          "MainBody": null,
          "_fulltext": [
            "Arnold Schwarzenegger"
          ]
        }
      ]
  }
}

Similarly, when a site visitor queries for californian governor:

{
  Content(
    locale: en,
    orderBy: { _ranking: SEMANTIC }
    where: { _fulltext: { match: "california governor" } }
  ) {
    total
    items {
      _score
      Name
      _fulltext
    }
  }
}

The most relevant and expected content is returned to the top:

{
  "Content": {
    "total": 3,
    "items": [
      {
        "_score": 0.057718195,
        "Name": "Arnold Schwarzenegger",
        "_fulltext": [
          "Arnold Schwarzenegger"
        ]
      },
      {
        "_score": 0.054158323,
        "Name": "Temporary Page Title",
        "_fulltext": [
          "The American frontier, also known as the Old West, popularly known as the Wild West, encompasses the geography, history, folklore, and culture associated with the forward wave of American expansion in mainland North America that began with European colonial settlements in the early 17th century and ended with the admission of the last few contiguous western territories as states in 1912. This era of massive migration and settlement was particularly encouraged by President Thomas Jefferson following the Louisiana Purchase, giving rise to the expansionist attitude known as \"Manifest Destiny\" and the historians' \"Frontier Thesis\". The legends, historical events and folklore of the American frontier have embedded themselves into United States culture so much so that the Old West, and the Western genre of media specifically, has become one of the defining periods of American national identity.",
          "Temporary Page Title"
        ]
      },
      {
        "_score": 0.05409809,
        "Name": "Alan Turing",
        "_fulltext": [
          "1913-06-22T17:17:56Z",
          "Alan Turing",
          "true",
          "1",
          "2",
          "3",
          "4",
          "5"
        ]
      }
    ]
  }
}

We can combine different ranking criteria, where we rank primarily rank by date in ascending order, and then with using the relevance score computed with semantic search as a tie-breaker.

{
      BiographyPage(
        locale: en
        orderBy: { Die: ASC, _ranking: SEMANTIC }
        where: { _fulltext: { match: "scientist" } }
      ) {
        total
        items {
          Name
          Die
          _fulltext
        }
      }
}