Mine Your Own Business: Market Structure Surveillance Through Text Mining

Use text mining and network analysis to capture and interpret an ocean of online data — listening in on consumers without asking a single question.
September 24, 2010
Print this page

The Idea

Use text mining and network analysis to capture and interpret an ocean of online data — listening in on consumers without asking a single question.

The Research

Marketing managers are often challenged by the difficulty and expense of collecting meaningful data about consumers using traditional methods such as surveys and focus groups. At the same time, today’s marketing managers confront an almost limitless bounty of data generated by consumers via online sources such as consumer forums, chat rooms, blogs, and product review sites. While these platforms provide large amounts of rich, qualitative information directly from consumers, the data is not easily quantified or analyzed.

Professor Oded Netzer worked with Ronen Feldman and Jacob Goldenberg of Hebrew University in Jerusalem to create a two-part approach to capture and analyze online data generated by consumers. First, they created a text mining tool that converts unstructured online data — including correcting abbreviations or misspellings of keywords like brand names and product attributes — into structured, quantifiable data. The second part of their method employs semantic network analysis and mapping techniques, derived in part from psychology theory that posits that the brain has an associative network that groups together and recalls items and concepts that are closely associated in memory. The method allowed the researchers to create visualizations of large-scale data by assessing how frequently keywords — such as brand names or product attributes — occurred together in the text, and treating the keywords and co-occurrences as nodes in the semantic network.

The researchers tested the accuracy of these techniques on an online consumer car forum. The analysis allowed the researchers to assess similarities between different cars in the discussion and the derived market structure. They found that their analysis correlated very closely to actual sales and results generated by traditional survey-based techniques. The analysis also revealed less obvious aspects of market structure. For example, the Cadillac brand was more closely associated with higher-end European brands than with other American brands, which may reflect General Motors’ efforts to reposition the Cadillac brand over the past few years. The tools allow for tracking such market position trends over time.

Used to analyze online consumer discussions about prescription drugs, the technique revealed that consumers frequently mentioned several side effects not commonly associated with those drugs in medical records. Firms and regulators could monitor such reports to take corrective action or — since a side effect could be due to an underlying medical condition — correct consumers’ perceptions about the origin of the side effect. The research holds great potential for gaining insight into market structures, competitive landscapes, and the top-of-mind association between products and features.

Practical Applications

Marketing managers, customer relationship managers, social media managers

You can use this research to collect data on customer preferences and competitors’ products by mining online consumer forums. You can also track the effectiveness of ad campaigns and related marketing efforts and monitor word-of-mouth as spread by opinion leaders.

Product developers

You can use this research to mine online data in real time, monitoring the course of existing trends and identifying new ones. You can also use this research to identify unmet needs of current consumers and develop new products. This research can be used to track the lifecycle of a product or competitor’s product from launch to new model and alter and add features to existing products based on consumer feedback.

Jacob Goldenberg

Jacob Goldenberg is a professor of Marketing at the School of Business Administration in the Hebrew University of Jerusalem, and a visiting professor in the Columbia business school and in IDC. He received his Ph.D. from the Hebrew University of Jerusalem in a joint program of the School of Business Administration and the Racach Institute of Physics. His research focuses on creativity, new...

View full profile

Oded Netzer

Professor Netzer's research centers on one of the major business challenges of the data-rich environment of the 21st century: developing quantitative methods that leverage data to gain a deeper understanding of customer behavior and guide firms' decisions. He focuses primarily on building statistical and econometric models to measure consumer preferences and understand how customer choices change over time, and across contexts. His research...

View full profile

Read the Research

Oded Netzer, Ronen Feldman, Jacob Goldenberg, Moshe Fresko

"Mine Your Own Business: Market Structure Surveillance Through Text Mining"


Download PDF View abstract/citation
View Research