Semantic search
Optimizely Graph supports using AI with semantic search, also called neural search or vector search.
Note
Semantic search is an experimental feature and may be subject to change.
Optimizely Graph supports matching and ranking of content beyond the standard lexical (literal keyword) matching using artificial intelligence (AI). The intent of the user query can be captured by extending it with context. The meaning of words can be part of that context. Language models are created for that purpose. You can extend keyword matching with pre-trained language models. This approach to search is commonly called semantic search. It can also be called neural search, vector search, and dense or sparse neural retrieval. For more technical background, see the blog post Do you know what I mean? Introducing Semantic Search in Optimizely Graph.
Semantic search is a solution for the so-called vocabulary mismatch problem. The keywords a user enters, like a site visitor, may not match the words the content marketer uses. For example, a site visitor may search for "non-alcoholic cold beverage" and expect content items about "cola" to be returned. A standard search would not return content if the keyword used in the query does not display in the content, but with a semantic search, you can return content because it "knows" about the context of the query.
One way to solve the vocabulary mismatch problem is by creating synonyms, which Optimizely Graph supports. This is effective but can be time-consuming. Semantic search is a good way to automate expanding queries with synonyms and improve results ranking.
Another important use case that semantic searching supports is helping conversational AI (chatbots) by feeding it with relevant results that solve hallucinations (a technique called Retrieval Augmented Generation (RAG)). See the blog post, How to Create AI-driven Chatbots with Optimizely Graph, for how you could implement RAG with Optimizely Graph. You can also use the vector search technology that drives semantic search to cluster or de-duplicate content and detect anomalies.
This methodology to do search has proven to work best in combination with traditional keyword search. The system can return the most relevant content in the right order that users want. Optimizely Graph supports and refers to this as semantic search, but it is effectively a mixed approach between standard keyword search and pure vector search.
How does Semantic Search work in Optimizely Graph?
Semantic search is supported on searchable string fields, and for the full-text search operators, contains
and match
. You must set fields with a lot of content (such as the MainBody
in the Optimizely CMS) as searchable to unlock the full-text search capabilities. Optimizely Graph uses a pre-trained model for semantic search.
To enable semantic search, change the value of _ranking
in the orderby to SEMANTIC
. By default Optimizely Graph uses RELEVANCE
, which is a standard keyword search with BM25 relevance ranking. You can combine SEMANTIC
with other ranking criteria, for instance, use the semantic search capabilities but rank the results by field value instead.
How to configure the weight of semantic search
Optimizely Graph computes the relevance score by adding matching results with semantic search to the score. You can configure the weight of semantic search by using the _semanticWeight
in the orderBy
. By default, the weight is set to 0.2
. You could decrease or increase it by carefully testing it with your use case, queries, and data.
Example:
{
Content(
orderBy: { _ranking: SEMANTIC, _semanticWeight: 0.5 }
where: { _fulltext: { match: "greatest president" } }
) {
total
items {
_score
_fulltext
}
}
}
Supported languages
The models for semantic search are trained on English and multilingual datasets. Hence, Optimizely Graph supports the languages with the following locale values
Language | 2-letter Locale |
---|---|
Arabic | ar |
Bulgarian | bg |
Catalan | ca |
CJK (special) | cjk |
German | de |
Greek | el |
English | en |
Spanish | es |
Farsi | fa |
Finnish | fi |
French | fr |
Galician | gl |
Hindi | hi |
Hungarian | hu |
Armenian | hy |
Indonesian | id |
Italian | it |
Japanese | ja |
Korean | ko |
Kurdish | ku |
Latvian | lv |
Dutch | nl |
Norwegian | no |
Polish | pl |
Romanian | ro |
Russian | ru |
Swedish | sv |
Thai | th |
Turkish | tr |
Ukrainian | uk |
Chinese | zh |
Examples
For example, using the following query, the user wants content that is about action movie
{
Content(orderBy: { _ranking: SEMANTIC }, where: { MainBody: { contains: "action movie" } }) {
total
items {
_score
Name
MainBody
_fulltext
}
}
}
Optimizely Graph will return the most relevant content to the top.
Note
Both "action" and "movie" do not display in the content.
{
"Content": {
"total": 4,
"items": [
{
"_score": 0.013479907,
"Name": "Standard Page 12",
"MainBody": "Wild Wild West is a 1999 American steampunk Western film co-produced and directed by Barry Sonnenfeld and written by S. S. Wilson and Brent Maddock alongside Jeffrey Price and Peter S. Seaman, from a story penned by brothers Jim and John Thomas. Loosely adapted from The Wild Wild West, a 1960s television series created by Michael Garrison, it is the only production since the television film More Wild Wild West (1980) to feature the characters from the original series. The film stars Will Smith (who previously collaborated with Sonnenfeld on Men in Black two years earlier in 1997) and Kevin Kline as two U.S. Secret Service agents who work together to protect U.S. President Ulysses S. Grant (Kline, in a dual role) and the United States from all manner of dangerous threats during the American Old West.",
"_fulltext": [
"Wild Wild West is a 1999 American steampunk Western film co-produced and directed by Barry Sonnenfeld and written by S. S. Wilson and Brent Maddock alongside Jeffrey Price and Peter S. Seaman, from a story penned by brothers Jim and John Thomas. Loosely adapted from The Wild Wild West, a 1960s television series created by Michael Garrison, it is the only production since the television film More Wild Wild West (1980) to feature the characters from the original series. The film stars Will Smith (who previously collaborated with Sonnenfeld on Men in Black two years earlier in 1997) and Kevin Kline as two U.S. Secret Service agents who work together to protect U.S. President Ulysses S. Grant (Kline, in a dual role) and the United States from all manner of dangerous threats during the American Old West.",
"Standard Page 12"
]
},
{
"_score": 0.011364636,
"Name": "Temporary Page Title",
"MainBody": "The American frontier, also known as the Old West, popularly known as the Wild West, encompasses the geography, history, folklore, and culture associated with the forward wave of American expansion in mainland North America that began with European colonial settlements in the early 17th century and ended with the admission of the last few contiguous western territories as states in 1912. This era of massive migration and settlement was particularly encouraged by President Thomas Jefferson following the Louisiana Purchase, giving rise to the expansionist attitude known as \"Manifest Destiny\" and the historians' \"Frontier Thesis\". The legends, historical events and folklore of the American frontier have embedded themselves into United States culture so much so that the Old West, and the Western genre of media specifically, has become one of the defining periods of American national identity.",
"_fulltext": [
"The American frontier, also known as the Old West, popularly known as the Wild West, encompasses the geography, history, folklore, and culture associated with the forward wave of American expansion in mainland North America that began with European colonial settlements in the early 17th century and ended with the admission of the last few contiguous western territories as states in 1912. This era of massive migration and settlement was particularly encouraged by President Thomas Jefferson following the Louisiana Purchase, giving rise to the expansionist attitude known as \"Manifest Destiny\" and the historians' \"Frontier Thesis\". The legends, historical events and folklore of the American frontier have embedded themselves into United States culture so much so that the Old West, and the Western genre of media specifically, has become one of the defining periods of American national identity.",
"Temporary Page Title"
]
},
{
"_score": 0.011052586,
"Name": "Wilder Westen",
"MainBody": "Wilder Westen ist eine – geographisch und historisch grob eingegrenzte – umgangssprachliche Bezeichnung für die ungefähr westlich des Mississippi gelegenen Gebiete der heutigen Vereinigten Staaten. In der auch als „Pionierzeit“ bezeichneten Ära des 19. Jahrhunderts waren sie noch nicht als Bundesstaaten in die Union der Vereinigten Staaten aufgenommen. Im Verlauf der voranschreitenden Landnahme und Urbanisierung nahm die Besiedlung dieser Regionen vor allem durch Angloamerikaner – bzw. aus Europa stammende Immigranten – kontinuierlich zu, bis die Gebiete um 1890 in den organisierten Territorien der Vereinigten Staaten aufgingen. Symbolisch stehen die Öffnung der letzten Indianerterritorien im späteren US-Bundesstaat Oklahoma für die Besiedlung durch Kolonisten 1889–1895 durch eine Serie von Land Runs und das Massaker der United States Army an etwa 200 bis 300 Lakota am Wounded Knee Creek/South Dakota im Dezember 1890 für das Ende der Zeit des Wilden Westens. Mit diesen Ereignissen galten die Indianerkriege ebenso als abgeschlossen wie die Kolonisation der bis dahin von den Vereinigten Staaten beanspruchten Hoheitsgebiete (engl. territories) durch die aus Europa eingewanderten Siedler.",
"_fulltext": [
"Wilder Westen ist eine – geographisch und historisch grob eingegrenzte – umgangssprachliche Bezeichnung für die ungefähr westlich des Mississippi gelegenen Gebiete der heutigen Vereinigten Staaten. In der auch als „Pionierzeit“ bezeichneten Ära des 19. Jahrhunderts waren sie noch nicht als Bundesstaaten in die Union der Vereinigten Staaten aufgenommen. Im Verlauf der voranschreitenden Landnahme und Urbanisierung nahm die Besiedlung dieser Regionen vor allem durch Angloamerikaner – bzw. aus Europa stammende Immigranten – kontinuierlich zu, bis die Gebiete um 1890 in den organisierten Territorien der Vereinigten Staaten aufgingen. Symbolisch stehen die Öffnung der letzten Indianerterritorien im späteren US-Bundesstaat Oklahoma für die Besiedlung durch Kolonisten 1889–1895 durch eine Serie von Land Runs und das Massaker der United States Army an etwa 200 bis 300 Lakota am Wounded Knee Creek/South Dakota im Dezember 1890 für das Ende der Zeit des Wilden Westens. Mit diesen Ereignissen galten die Indianerkriege ebenso als abgeschlossen wie die Kolonisation der bis dahin von den Vereinigten Staaten beanspruchten Hoheitsgebiete (engl. territories) durch die aus Europa eingewanderten Siedler.",
"Wilder Westen"
]
},
{
"_score": 0.009704749,
"Name": "Arnold Schwarzenegger",
"MainBody": null,
"_fulltext": [
"Arnold Schwarzenegger"
]
}
]
}
}
Similarly, when a site visitor queries for california governor
{
Content(
locale: en,
orderBy: { _ranking: SEMANTIC }
where: { _fulltext: { match: "california governor" } }
) {
total
items {
_score
Name
_fulltext
}
}
}
The most relevant and expected content is returned to the top
{
"Content": {
"total": 3,
"items": [
{
"_score": 0.057718195,
"Name": "Arnold Schwarzenegger",
"_fulltext": [
"Arnold Schwarzenegger"
]
},
{
"_score": 0.054158323,
"Name": "Temporary Page Title",
"_fulltext": [
"The American frontier, also known as the Old West, popularly known as the Wild West, encompasses the geography, history, folklore, and culture associated with the forward wave of American expansion in mainland North America that began with European colonial settlements in the early 17th century and ended with the admission of the last few contiguous western territories as states in 1912. This era of massive migration and settlement was particularly encouraged by President Thomas Jefferson following the Louisiana Purchase, giving rise to the expansionist attitude known as \"Manifest Destiny\" and the historians' \"Frontier Thesis\". The legends, historical events and folklore of the American frontier have embedded themselves into United States culture so much so that the Old West, and the Western genre of media specifically, has become one of the defining periods of American national identity.",
"Temporary Page Title"
]
},
{
"_score": 0.05409809,
"Name": "Alan Turing",
"_fulltext": [
"1913-06-22T17:17:56Z",
"Alan Turing",
"true",
"1",
"2",
"3",
"4",
"5"
]
}
]
}
}
You can combine different ranking criteria. For example, primarily rank by date in ascending order and then use the relevance score computed with semantic search as a tie-breaker
{
BiographyPage(
locale: en
orderBy: { Date: ASC, _ranking: SEMANTIC }
where: { _fulltext: { match: "scientist" } }
) {
total
items {
Name
Date
_fulltext
}
}
}
Updated about 2 months ago