September 24, 2010

Mine Your Own Business: Market Structure Surveillance Through Text Mining

Use text mining and network analysis to capture and interpret an ocean of online data — listening in on consumers without asking a single question.

The Idea

Use text mining and network analysis to capture and interpret an ocean of online data — listening in on consumers without asking a single question.

The Research

Marketing managers are often challenged by the difficulty and expense of collecting meaningful data about consumers using traditional methods such as surveys and focus groups. At the same time, today’s marketing managers confront an almost limitless bounty of data generated by consumers via online sources such as consumer forums, chat rooms, blogs, and product review sites. While these platforms provide large amounts of rich, qualitative information directly from consumers, the data is not easily quantified or analyzed.

Professor Oded Netzer worked with Ronen Feldman and Jacob Goldenberg of Hebrew University in Jerusalem to create a two-part approach to capture and analyze online data generated by consumers. First, they created a text mining tool that converts unstructured online data — including correcting abbreviations or misspellings of keywords like brand names and product attributes — into structured, quantifiable data. The second part of their method employs semantic network analysis and mapping techniques, derived in part from psychology theory that posits that the brain has an associative network that groups together and recalls items and concepts that are closely associated in memory. The method allowed the researchers to create visualizations of large-scale data by assessing how frequently keywords — such as brand names or product attributes — occurred together in the text, and treating the keywords and co-occurrences as nodes in the semantic network.

The researchers tested the accuracy of these techniques on an online consumer car forum. The analysis allowed the researchers to assess similarities between different cars in the discussion and the derived market structure. They found that their analysis correlated very closely to actual sales and results generated by traditional survey-based techniques. The analysis also revealed less obvious aspects of market structure. For example, the Cadillac brand was more closely associated with higher-end European brands than with other American brands, which may reflect General Motors’ efforts to reposition the Cadillac brand over the past few years. The tools allow for tracking such market position trends over time.

Used to analyze online consumer discussions about prescription drugs, the technique revealed that consumers frequently mentioned several side effects not commonly associated with those drugs in medical records. Firms and regulators could monitor such reports to take corrective action or — since a side effect could be due to an underlying medical condition — correct consumers’ perceptions about the origin of the side effect. The research holds great potential for gaining insight into market structures, competitive landscapes, and the top-of-mind association between products and features.

Practical Applications

Marketing managers, customer relationship managers, social media managers

You can use this research to collect data on customer preferences and competitors’ products by mining online consumer forums. You can also track the effectiveness of ad campaigns and related marketing efforts and monitor word-of-mouth as spread by opinion leaders.

Product developers

You can use this research to mine online data in real time, monitoring the course of existing trends and identifying new ones. You can also use this research to identify unmet needs of current consumers and develop new products. This research can be used to track the lifecycle of a product or competitor’s product from launch to new model and alter and add features to existing products based on consumer feedback.

Read the Research

Oded Netzer, Ronen Feldman, Jacob Goldenberg, Moshe Fresko

"Mine Your Own Business: Market Structure Surveillance Through Text Mining"

View abstract/citation  Download PDF   View Research  

Have You Read Columbia Business?

Columbia Business School’s alumni magazine connects alumni with each other and the School; celebrates alumni milestones and accomplishments; and chronicles the impact of Columbia Business School alumni, faculty members, and students on the global business landscape.

Read Columbia Business>