UK company FreshMinds Research recently ran a test by pulling social media commentary about Starbucks using several popular analytic tools offering automated sentiment analysis of the text gathered. They found flipping a coin to determine the sentiment of each individual comment would have been more accurate than what the tools reported.
FreshMinds analyzed over 19,000 online conversations with tools from Alterian, Biz360, Brandwatch, Nielsen, Radian6, Scoutlabs and Sysomos. All content was centered on Starbucks.
The good news is aggregate level reporting of sentiment (average overall) was between 60% and 80% in agreement with a manual coding by trained staff. Not bad. The bad news? Only about a third of individual comments were accurately coded.
Somehow, the randomization of automation errors resulted in an aggregate number of coding all conversations that wasn’t off by much. But, if you wanted to dig deeper into individual conversations either for more insight or to engage in the conversation, the likelihood of finding the right positive or negative comments is not very high at all.
Their report is an excellent overview of these seven tools and how they perform across geographies and content sources. And, as a side note, it’s a great marketing effort to get you and me to pull down their paper in exchange for contact information.
It’s not surprising to me that these tools are still so far off. It’s a micro-representation of a macro-level challenge facing most research firms, agencies, and marketers today: putting things into context from a people-centric approach. We have so much data today that making it both accurate and actionable requires a more concerted effort to put everything into context, mirroring the reality of human decision-making and behavior as much as possible.
I’m sure some combination of neural networks, complexity science, and/or agent-based simulation tools eventually will yield “smarter” sentiment analysis tools to speed up the process of sifting through thousands of lines of text-based data. Those pursuing that dream need not lose sight of the biggest mystery to solve: understanding the meaning of words within a human context.
The FreshMinds report is definitely worth the read. I’m curious what the makers of these tools would have to say about their report.
Thanks to Research (the magazine) for the heads up on the white paper release.