Stop words

Stop words are the words in a stop list (or stop list or negative dictionary) that are filtered out (stopped) before or after processing of natural language data (text) because they are insignificant.

A use-case of stop words, besides stopping unimportant words from being processed, is stopping words that are considered noise otherwise from a business or societal perspective. This means that no matches are retrieved by the search engine given the queried stop words. Optimizely Graph does not use stop words by default, but you can configure them.

Stop words with full-text search

The following list of English words are often considered stop words:

a, an, and, are, as, at, be, but, by, for, if, in, into, is, it, no, not, of, on, or, such, that, the, their, then, there, these, they, this, to, was, will, with.

A stop word is usually a single word that is used as a filter to stop a token from being indexed.

For example, if you have this field value "the dog is at the park", and use this stop list, then the following tokens get indexed ["dog", "park"] and you can only match on these two tokens when doing a full-text search using the contains or like operators.

❗️

Warning

Stop words are supported for searchable string fields but not supported for normal string, number, date, and Boolean fields.

In the latter field types, stop words are not applied, and results are found when querying with stop words. Optimizely Graph supports only single-token stop words, and multi-word stop words are not applied.

Store custom stop words

Stop words are stored as a text file, each line being a single stop word.

🚧

Important

A line in the stop list cannot exceed 1,000 characters or bytes, and the maximum number of entries is 50,000.

Stop words are treated case-sensitively at index and query time. For example, the is different than The. This could be useful to fully index and query on The Guardian (newspaper) but ignore the in the guardian with full-text search.

The following is an example of a list of stop words. They are used in the query examples below.

the
Schwarzenegger
amy
Bob

You can store stop words using the REST endpoint configured in the GraphQL gateway. It requires authorization using your HMAC key and secret.

• PUT <GATEWAY_URL>/resources/stopwords with the following optional query string:
  • language_routing to store the custom stop words in the request body for a specific locale (default is standard, that is, no locale)

The body should contain stop words as previously described or can be empty if you do not want to configure any stop words (the default behavior). If you do not use a query parameter with this endpoint, then the custom stop list is applied to the NEUTRAL locale (index with no languages configured).

After storing stop words, they are automatically applied when synchronizing content and ignored when querying with Optimizely Graph.

❗️

Warning

You must store your stop words in Optimizely Graph before provisioning your account and synchronizing content. You cannot update stop lists after your account is provisioned. If you want to update your stop list, you must do the following:

1. Upload the updated stop list with the PUT endpoint as described above.
2. Reset account.
3. Synchronize content.

Query examples

For full-text search with the contains and like operators on searchable string fields, Optimizely only permits single-token stop words, and multi-word stop words will not be applied.

When Schwarzenegger is a stop word and occurs as Schwarzenegger (case-sensitive) in your content, the following query will not return any results.

{
  BiographyPage(where: { Name: { contains: "Schwarzenegger" } }) {
    items {
      Name
      Die
      Born
      Language {
        DisplayName
        Name
      }
      _score
    }
  }
}

However, if the name Amy Winehouse occurs in your content but amy (note the lowercase) is defined as a stop word, you still get a result returned with the following GraphQL query because the term Amy (note the uppercase) was never stopped from being indexed and will return the result.

{
  BiographyPage(where: { Name: { contains: "Amy" } }) {
    items {
      Name
      Die
      Born
      Language {
        DisplayName
        Name
      }
      _score
    }
  }
}

This query is equivalent in this form and will also return the result.

{
  BiographyPage(where: { Name: { like: "%Amy%" } }) {
    items {
      Name
      Die
      Born
      Language {
        DisplayName
        Name
      }
      _score
    }
  }
}

Both examples will return this result:

{
  "data": {
    "BiographyPage": {
      "items": [
        {
          "Name": "Amy Winehouse",
          "Die": "2011-07-23T00:00:00Z",
          "Born": "1983-11-14T00:00:00Z",
          "Language": {
            "DisplayName": "English",
            "Name": "en"
          },
          "_score": 1.6928279
        }
      ]
    }
  }
}

Stop words are processed case-sensitively at indexing time. Therefore, the following query will not return any results because it is a stop word.

{
  BiographyPage(where: { Name: { contains: "amy" } }) {
    items {
      Name
      Die
      Born
      Language {
        DisplayName
        Name
      }
      _score
    }
  }
}