What do users really want in an event summarization system?

The wide usage of social media means that users now have to keep up with a large number of incoming content, motivating the development of several stream monitoring tools, such as PalanteerTopsyTweet Archivist, etc. Such tools could be used to aid in sensemaking about real-life events by detecting and summarizing social media content about these events. Given the large amount of content being shared and the limited attention of users, what information should we provide to users about special events as they are detected in social media? 

In our analysis, we analyzed tweets related to four diverse events:

  1. Facebook IPO
  2. Obamacare
  3. Japan Earthquake
  4. BP Oil Spill

The figure below shows the temporal patterns of usage for words related to the Facebook launch price. By exploiting the content similarity between tweets written around the same time, we could discover various aspects (topics) of an event.

Facebook IPO Launch Price
These plots show frequency of usage over time for various words related to the Facebook IPO. We can see similarities and differences in the temporal profiles of the usage of each of these words.

The figure below shows how the volume of content related to various aspects (topics) of an event changes over time, as the event unfolds. Notice that some aspects have a longer lifespan of attention from tweeters, while others peak and die off quickly.

Topics through time
These two figures show how the topics within an event change over time. The figure on the left shows raw volumes, while the figure on the right shows underlying patterns used in our model. Notice how topics spike at different times and with different amounts of concentration over time.

We used our model to generate summaries and hired workers on Amazon Mechanical Turk to provide feedback. Please refer to this link for the summaries we showed to our workers. Which summary do you like best? This is what some of our respondents had to say:

  1. Number 3 has the most facts.
  2. Summary 2 is more straight forward information & not personal appeal pieces like live chats and other stuff with people who are unqualified to speak about the issue.
  3. None. All too partisan
  4. Summary 3 has most news with less personal commentary than the others.
  5. I believe that summary 1 and 2 had a large amount of personal opinion and not fact.
  6. I think summary 3 best summarize Facebook IPO because it shows a broad range of information related to the event.
  7. Summary 3 is more comprehensive and offers better overall summary.

Overall, we received feedback from users that they want summaries that are comprehensive, covering a broad range of information. Furthermore, they want summaries to be objective, factual, and non-partisanWhile we believe we have done well in giving users comprehensive and broad range information, we think that future work in summarization will reduce the gap between what researchers are doing and what users really want.

For more, see our full paper,  Automatic Summarization of Events from Social Media.
Freddy Chua, Living Analytics Research Centre, Singapore Management University
Sitaram Asur, Social Computing Research Group, Hewlett Packard Research Labs