FAQ: How does topic extraction work?

Topic extraction is the automated process of identifying the key subjects and concepts within a piece of content. By understanding what a text is about, we can power features like content recommendations and analytics, helping to surface the most relevant information to users.

This document provides a high-level overview of how this process works.

What is a "Topic"?

In our system, a "topic" is a specific entity - such as a person, place, organization, or concept - that can be uniquely identified. Think of topics as entries in an encyclopedia. Our system uses Wikipedia as its knowledge source, so a topic generally corresponds to a concept that has a Wikipedia page.

For example, in the sentence, "The new phone from Apple has a great camera," the topics identified would be "Apple Inc." and "Camera phone." Natural language can be ambiguous; the word "Apple" could refer to the fruit or the technology company. Our system is designed to understand the difference based on the context.

How are Topics Extracted?

Topics are extracted from the main body of your content using Natural Language Processing (NLP). While the underlying process is complex, it can be broken down into a few key steps:

Identifying Potential Topics: The system first scans the text to find words or phrases (like "Apple" or "New York") that could refer to known topics.
Generating Candidates: For each potential topic, the system identifies all its possible meanings. For instance, the term "Giants" could refer to a sports team, a mythological creature, or a type of star.
Disambiguation: The system then analyzes the surrounding words and the overall context of the content to determine the most probable meaning. If an article about "Giants" also mentions "baseball" and "San Francisco," it will correctly associate the term with the sports team, not the mythological being.

What Affects the Quality of Topic Extraction?

The accuracy and relevance of extracted topics depend on several factors:

Content Length: Longer, more detailed articles provide more context, which helps the system accurately identify and disambiguate topics. Topic extraction from very short texts can be less reliable due to the lack of contextual clues.
Frequency of Terms: The more frequently a concept is mentioned throughout a text, the more likely it is to be identified as a primary topic with a higher relevance score.

A Note on Topic Naming

Because our topic data is sourced from Wikipedia, the topic names directly reflect the titles of the corresponding Wikipedia articles. This can sometimes lead to scientific or formal terms being used instead of more common names. For example, an ingredient in a recipe might be identified by its Latin name if that is the title of its Wikipedia page.

If you have issues with how topics are named or believe a topic has been identified incorrectly, please open a support ticket with a link to the content and details about the issue. Our team can then investigate.