More terrible than terrific: do we have more negative words than positive ones in English?

One of the common approaches to the problem of sentiment analysis (a field under text mining & natural language processing (NLP), where programs try to detect opinion in natural language texts) is to build a dictionary of 'opinion' words. The words are classified as negative & positive. Given words from a sentence, a program can look up the dictionary to see if any of these words appear in dictionary, and then use the positive or negative category as an input in detecting sentiment for that sentence. (Of course, this is a simplified explanation of what actually happens.)

We work in this field and so, in one of our approaches, have built such a lexicon. Our's is a small list and hence not comprehensive, but sufficient for our purposes. Now, I noticed that I had a lot more words tagged as negative rather than as positive. Stated in numbers, there were 434 words marked positive, and 1348 marked negative. I had initially built a much smaller list by hand, and then expanded the lexicon automatically by (partially) using an approach (pdf) described by Italian researchers Andrea Esuli and Fabrizio Sebastiani.

They had also created SentiWordNet. This extends WordNet, which is a popular language resource used in natural lanuage processing and in essence, is a dictionary-thesaurus on steroids (the good kind :-)). WordNet contains over 150,000 words and arranges them 'conceptually', by grouping together synonyms that make up unique 'senses' (these groups are called 'synsets') (it may be obvious why I didn't the word 'sensually' to describe the arrangement). SentiWordNet augments this by attaching a positive and a negative score to each synset. (Here, I won't discuss why a synset can have both a positive & negative score.) Words like 'horrible' or bad have a high negative score, while awesome and pleasant are very positive.

Coming back to our question. Seeing the difference in my list, I wondered if this was a possibly valid observation, or if my lexicon was just poorly constructed, or a consequence of applying the expansion technique in part. So I counted the number of positive & negative synsets in SentiWordNet (again, not going into details here). I found 14134 negative synsets and 12720 positive ones. Perhaps not a significant difference, but still the negative side is a little greater in number (and I haven't actually counted words, only sense groups). So it could just be that I chose or generated more negative words.

This is all anecdotal and perhaps some fun for language geeks to talk about when they're stuck in a long queue and haven't brought a book along :-)

Going senti

Paul Greenberg is an analyst in the world of CRM tools, with a book on the topic, to his credit. In a recent ZDNet blog post, he added Cognizant's Social CRM ventures to his annual CRM watchlist, saying:

This is one of my most out-of-place and yet most appropriate picks for 2010.[...]The fact that they have a practice is well and good but they back the practice up with content and that makes a huge difference. They are actually developing a Social CRM framework for both technology and strategy and investing resources in products that can support them.

Further he adds:

For example, they have a sentiment analyzer that I saw not too long ago that is a solid entry into the tool belt. This is not a commercial product, but a tool for their work and their clients to use. Honestly, with not much more work, it could be a very salable sentiment analysis tool - competitive in most ways.

I quote these lines because my group is responsible for creating the Sentiment Analyzer quoted above. I've been working on this for the last couple of years and it is interesting to finally see some non-traditional applications of sentiment analysis come to the fore. (Usually, they have revolved around brand monitoring and reputation analysis.)

The jury is still out on what constitutes Social CRM and its benefits beyond simply adding another set of channels to existing CRM systems. People like Cognizant's A. Prem have been writing and talking about this fledgling area, but as someone involved in text analytics , I am interested in seeing how this evolves in the future.