Discussion Graphs: Better social media analysis through tools

Capturing and exploring the context of social media discussions is critical to understanding the relationships and information we extract from them. Knowing where information comes from helps us interpret it more correctly and knowing how our extracted results change as we condition on context provides insights into the underlying phenomena and can suggest further lines of investigation and action. For instance, the Livehoods project demonstrated how we can calculate a social distance between locations and use this to map out neighborhood boundaries. Extending this analysis by including contextual factors, such as gender and temporal information, we find that these boundaries can shift, sometimes significantly so. Consider the two maps below of neighborhood boundaries extracted from social data gathered on weekdays (Fig 1) and weekends (Fig 2), demonstrating clearly distinct mobility patterns, such as the appearance of a distinct “5th Ave shopping district” on weekends. Weekday_crop3

Figure 1. Neighborhood boundaries inferred from weekday behaviors

 Weekend_crop3

Figure 2. Neighborhood boundaries inferred from weekend behaviors

The social sciences have long taken the conditioning on demographic factors such as gender and socioeconomic status seriously. In fact, it’s commonly understood that incorrect conditioning can in some cases completely reverse empirical results (see for example, Simpson’s Paradox). So, why aren’t social media analyses more commonly conditioned on such demographic variables? We believe that much of the blame lies on the practical difficulties of implementing the necessary feature extractors, aggregators and analysis components. Moreover, without appropriate computational abstractions, much of this work must be adapted or re-created for each new data set and research question. To address this challenge, we are presenting a paper next week at ICWSM 2014, and releasing software that dramatically simplifies the implementation of co-occurrence analyses, a surprisingly common class of social media analyses.  At the core of our software are discussion graphs, a data model for representing and computing upon relationships extracted from social media. Discussion graphs capture both the structural features of relationships inferred from social media, as well as the context from which they are derived, in just a couple steps. First, feature extraction turns raw social media data into an initial discussion graph, where each node is an extracted feature-value, and hyper-edges connect all nodes that co-occurred together in a tweet.  Secondly, we project this graph to include only the relationships among target nodes, aggregating the remaining features as context annotating each relationship. Figure

Figure 3. Discussion graph framework

We implement a tool (cleverly named Discussion Graph Tool, or DGT) to easily build and manipulate discussion graphs. DGT enables sharing and re-use feature extractors (for example, to infer gender from names, or mood from language cues), extraction of relationships and information, and conditioning on context. With DGT, extracting relationships from social media, including the social distances that underlie the neighborhood boundary inference shown above, is a simple 4-5 line script. To learn more about discussion graphs and DGT, read our paper Discussion Graphs: Putting Social Media Analysis in Context, at ICWSM 2014, and look for our upcoming tool release at our project site, http://research.microsoft.com/dgt  This work is a collaboration between Emre Kıcıman, Scott Counts, Michael Gamon, Munmun De Choudhury and Bo Thiesson.

Towards Supporting Search over Trending Events with Social Media

Trending events are events that serve as novel or evolving sources of widespread online activity. Such events range from anticipated events to breaking news, and topics vary from politics to sporting events to celebrity gossip. Recently, search engines have started reflecting search activity around trending events back to users (e.g. Bing Popular Now or Google Hot Searches).

Real-time content published via social media can provide valuable information about time-sensitive topics, but the topics being discussed can change quite rapidly over time. In our analysis, we aimed to answer the following questions: For what types of trending events will real-time information be useful, and for how long will it continue to align with the information needs of users searching about these events?

Figure: Information Types
We surveyed 288 users about their experience with trending events over a week in August 2012. Among other things, they reported on the utility of various types of information when making sense of such events; here, we see how important real-time information is to the users surveyed.

In order to identify ways to better support users issuing such queries, we examined hundreds of trending events during the summer of 2012, using three sources of data: (1) qualitative survey data, (2) query logs from Bing, and (3) Twitter updates from the complete Twitter firehose.

Our findings revealed that:

  • Searchers who click Trending Queries links engage less and with different result content than users who search manually for the same topics. This may be due to a preference for real-time information that is perhaps not currently being satisfied.
  • Search query and social media activity follow similar temporal patterns, but social media activity tends to lead by 4.3 hours on average, providing enough time for a search engine to index and process relevant content.
  • User interest becomes more diverse during the peak of activity for a trending event, but a corresponding increase in overlap between content searched and shared highlights opportunities for supporting search with social media content.
Search vs. Social Media Delays - Histogram
Each data point in this histogram corresponds to a single trending event. The value represents the delay between patterns of query activity and social media activity (negative values indicate that social media precedes search). The dotted red line shows the mean h = -4.3 hours.

Many current search interfaces leveraging social media content tend to provide a reverse-chronologically ordered list of keyword-matched updates. Our finding that search activity often lags behind social media activity means that there may be time for more complex indexing and ranking computation to present more relevant “near-real-time” content in search results.

For more about our study and implications for supporting search over trending events, see our full paper, Towards Supporting Search over Trending Events with Social Media.
Sanjay Kairam, Stanford University
Meredith Ringel Morris, Microsoft Research
Jaime Teevan, Microsoft Research
Dan Liebling, Microsoft Research
Susan Dumais, Microsoft Research

Memes and Cultural Organisms

In biology, the fundamental building blocks of complex organisms like ourselves are replicating DNA segments called genes. A cultural theory, called memetics, states that there also exists fundamental building blocks in culture, and this blocks are known as memes.

We study memes from an websites where users can create, combine, evolve and extinct memes, called Quickmeme.com. Two examples of memes are:

Screenshot from 2013-05-17 15:15:43
Two examples of similar memes that user created in the website Quickmeme.com

The current studies about memes focus on social networks. They are interested in understanding how the social dynamics affect the spread of memes in the human minds.

We, instead, study the direct interactions between memes, seeking their fundamental characteristics, without looking at social networks. We analyzed hundreds of such memes, with tens of thousand variations created by users. Our results show that:

  • Just like genes, we are able to prove that memes competes one against the other.
  • Just like genes, we also find traces of collaboration between memes.
  • Collaboration do no end in simple pairs of memes. Memes clump up and literally form cultural organisms.

What does it mean? It means that it is possible that the complex culture we live in (songs, books, cathedrals and so on) is the result of dynamics that closely resemble the ones among genes in the primeval broth.

We are able to define some characteristics of memes. They can be prone to competition or collaboration. Or they can bring the collaboration to a next level and create a large cluster of collaborating memes: a cultural organism.

Using these characteristics, we are able to predict if the meme will be a successful meme or not. A successful meme is a meme that is preserved in the minds of the users of the websites, and they use it often.

memetree
The decision tree describing the odds of success of memes, given their characteristics.

In the picture, we report a visualization about it. You can read the probabilities of success and the characteristics of the memes. Memes are successful in 35.47% of the cases. But lower popularity peaks, high competing strategies and being in a meme organism raise this probability up to 80.3%.

For more, see our full paper, Competition and Success in the Meme Pool: a Case Study on Quickmeme.com. You can also check further information on my website: www.michelecoscia.com.

Michele Coscia, CID – Harvard University

CrowdE: Filtering Tweets for Direct Customer Engagements

Many consumer brands hire customer agents to engage customers on social media services such as Twitter; these agents solicit opinions, respond to questions and requests, and thank or apologize to customers when necessary.

The usual method of filtering relevant customer opinions using simple keywords is often insufficient, because coming up with the right keywords is not easy. For instance, a representative at Delta Airlines filtering for “delta” would also collect posts aout “alpha delta phi” and “Nile delta”. A more restrictive query, requiring both “delta” and “airline” would miss posts such as “I flew to Seattle on Delta.” Furthermore, even among posts that indeed refer to the brand, many are side comments with no brand-relevant opinion, and therefore are usually not worth an agent’s attention.

As a result, agents end up wasting tons of time reviewing irrelevant content.

What’s the solution?

Image of CrowdE Dashboard
The CrowdE Dashboard allows users to filter for brand-relevant tweets and mark them for follow-up actions.

We produced CrowdE, an intelligent filtering system that helps brand agents filter tweets. We designed a common reusable filter creation process, where we ask crowd workers to label tweets for a brand and then extract insights through machine learning. The resulting filtering system has a number of nice properties:

  • It can be customized for any particular consumer brand with minimal cost and design effort.
  • It supports filtering by relevance to the brand and by presence of brand-related opinion.
  • Filtering accuracy is on-par with expert-crafted filter rules for the given brand.

Using the CrowdE system, agents can filter the live Twitter stream at will, and mark relevant follow-up actions for each tweet. In user studies, both experienced and novice users preferred CrowdE to a traditional keyword-based filter. Users considered CrowdE-based filtering to be more efficientmore completeless difficult, and less tedious. CrowdE also gave users more confidence in their filtering. Users performed better, as well, correctly marking more follow-up actions in the same amount of time.

For more details, see our ICWSM 2013 paper, CrowdE: Filtering Tweets for Direct Customer Engagements.
Jilin Chen, IBM Almaden Research Center
Allen Cypher, IBM Almaden Research Center
Clemens Drews, IBM Almaden Research Center
Jeffrey Nichols, IBM Almaden Research Center

 

What do users really want in an event summarization system?

The wide usage of social media means that users now have to keep up with a large number of incoming content, motivating the development of several stream monitoring tools, such as PalanteerTopsyTweet Archivist, etc. Such tools could be used to aid in sensemaking about real-life events by detecting and summarizing social media content about these events. Given the large amount of content being shared and the limited attention of users, what information should we provide to users about special events as they are detected in social media? 

In our analysis, we analyzed tweets related to four diverse events:

  1. Facebook IPO
  2. Obamacare
  3. Japan Earthquake
  4. BP Oil Spill

The figure below shows the temporal patterns of usage for words related to the Facebook launch price. By exploiting the content similarity between tweets written around the same time, we could discover various aspects (topics) of an event.

Facebook IPO Launch Price
These plots show frequency of usage over time for various words related to the Facebook IPO. We can see similarities and differences in the temporal profiles of the usage of each of these words.

The figure below shows how the volume of content related to various aspects (topics) of an event changes over time, as the event unfolds. Notice that some aspects have a longer lifespan of attention from tweeters, while others peak and die off quickly.

Topics through time
These two figures show how the topics within an event change over time. The figure on the left shows raw volumes, while the figure on the right shows underlying patterns used in our model. Notice how topics spike at different times and with different amounts of concentration over time.

We used our model to generate summaries and hired workers on Amazon Mechanical Turk to provide feedback. Please refer to this link for the summaries we showed to our workers. Which summary do you like best? This is what some of our respondents had to say:

  1. Number 3 has the most facts.
  2. Summary 2 is more straight forward information & not personal appeal pieces like live chats and other stuff with people who are unqualified to speak about the issue.
  3. None. All too partisan
  4. Summary 3 has most news with less personal commentary than the others.
  5. I believe that summary 1 and 2 had a large amount of personal opinion and not fact.
  6. I think summary 3 best summarize Facebook IPO because it shows a broad range of information related to the event.
  7. Summary 3 is more comprehensive and offers better overall summary.

Overall, we received feedback from users that they want summaries that are comprehensive, covering a broad range of information. Furthermore, they want summaries to be objective, factual, and non-partisanWhile we believe we have done well in giving users comprehensive and broad range information, we think that future work in summarization will reduce the gap between what researchers are doing and what users really want.

For more, see our full paper,  Automatic Summarization of Events from Social Media.
Freddy Chua, Living Analytics Research Centre, Singapore Management University
Sitaram Asur, Social Computing Research Group, Hewlett Packard Research Labs

Self-Censorship on Facebook

Ever start writing a post or comment on Facebook, but ultimately decide against sharing it? You wouldn’t be alone. We found that 71% of Facebook users exhibited some form of “last-minute” self-censorship over 17 days.

More specifically, users refrained from sharing 33% of the posts, and 13% of the comments, that they began writing.

Overall self-censorship rates, broken down across product usage. Comments are represented in the left chart, and posts are on the right.
Overall self-censorship rates, broken down across product usage. Comments are represented in the left chart, and posts in the right. The aggregation of all comments and posts are represented with the “comments” and “posts” label, respectively.

While last minute self-censorship is generally prevalent, the frequency of self-censorship does seem to vary by the nature of the content (e.g., is it a post or a comment?) and the context surrounding it (e.g., is it a status update or an event post?). Indeed, status updates (34%) and posts within Facebook groups (38%) were censored far more frequently than posts on friends’ timelines (25%) or events (25%).

The frequency of self-censorship for comments did not vary as drastically as with posts, though comments on photos (15%) and on group posts (14%) were censored more than comments on timeline posts (12%) and status updates (11%).

The decision to self-censor, thus, seems to be partially driven by two simple principles:

  • People censor more when their audience is harder to define, and
  • People censor more when the relevance or topicality of a CMC “space” is narrower.

In other words, undirected content that might be read by anyone is censored frequently, but so is very specifically directed content. After all, knowing one’s audience is only one part of the battle. A known audience is a double-edged sword: Topics relevant to the group may be easier to share, but fewer thoughts, statements, or photos may be considered relevant to the group.

Overall, a user’s “perceived audience” does indeed seem to lie at the heart of the matter, but the effect is not always straightforward.

We also found that:

  • People with more boundaries to regulate self-censor more;
  • Males self-censor more posts than females;
  • People who exercise more control over their audience self-censor more content; and,
  • Users with more politically and age diverse friends self-censor less, in general.

For more, see our full paper, Self-Censorship on Facebook.

Sauvik Das, Carnegie Mellon University
Adam Kramer, Facebook

What do users’ comments tell us about designing better tutorial systems?

If you’ve spent any time using complex software, you’ve probably had help from web-based tutorials. Though they are community-created content, web tutorials serve as the de-facto source of help for many users of these applications.

We analyzed user comments posted on popular tutorials for Word, Photoshop, and Excel to understand:

  • How web tutorials are currently being used,
  • the social practices in their comment sections,

and,

  • what these findings imply for creating new and better tutorial systems.

Check out the graphic below for a high-level sketch of our findings. Afterward I’ll talk about two results that we found particularly interesting.

Our analysis revealed insights into how people use tutorials, and the social practices in their comment areas.
Our analysis revealed insights into how people use tutorials, and the social practices in their comment areas.

First, we found an unexpected use of web tutorials that we’ve termed “expert shadowing”. In this scenario the user attempts to recreate a complex end result by mimicking the actions of an expert. The user’s primary goal in this use appears to be recreational; the tutorial is allowing the user to experience using the software at a level of ability they couldn’t otherwise, and this is both challenging and rewarding. This is qualitatively different from reading a tutorial to learn a new skill, or applying its instructions to a current problem, and it suggests an opportunity for tutorial systems that are designed to explicitly create this type of rich experience for users.

Second, we identified social practices that produce valuable information that could help other users, or improve the tutorial content. For example, when users run into trouble following a tutorial, they sometimes post “help-me” stack traces describing where they got stuck and their actions leading up to the problem. This information could help subsequent users, but in current tutorial designs it’s typically stuck in a long list of comments at the bottom of the page, where it’s unlikely to be seen when it’s needed most. In the paper, we outline some ideas on how tutorial interfaces could embrace and support valuable social practices to enable tutorials that improve over time as users use and respond to them.

For full details see our ICWSM 2013 paper, Understanding the Roles and Uses of Web Tutorials.
Ben Lafreniere, University of Waterloo
Andrea Bunt, University of Manitoba
Matthew Lount, University of Manitoba
Michael Terry, University of Waterloo

How are Wikipedia’s breaking news collaborations different?

When breaking news events like natural disasters happen, where do you go for information? Once upon a time, professional journalists were responsible for gathering information and sharing it with an audience. These days, however, long and detailed Wikipedia articles about these events are often written within hours of the event itself by Wikipedia editors, the vast majority of whom are not journalists. How are these breaking news collaborations different from more traditional Wikipedia article collaborations? And how have these breaking news collaborations on Wikipedia changed over time?

We examined the collaboration networks of 114,153 unique users making contributions to 3,233 Wikipedia articles about natural disasters, conflicts, crimes, and industrial and transportation accidents. Whereas a social network like Facebook is a set of people and the friendship relationships between them, a collaboration network on Wikipedia is a set of articles and the people who have made an edit to them.

We find overall that more popular articles have more prolific editors. For articles with many editors, these contributors tend to edit many other articles; articles with few editors are edited by less active users. This pattern has intensified over time.

degree_correlation
For all the breaking news articles in a year, the x-axis is the number of editors these articles have (degree) and the y-axis is the number of other articles its editors contribute to (assortativity).

To answer the question of whether these breaking news article collaborations are different from traditional Wikipedia article collaborations, we look at how quickly editors organize themselves into well-connected collaborations. Imagine you’re throwing a party: you need lots of people to leave what they were doing before and come together in one place around the same time. We examined an analogous process of how long it took all these breaking news articles and the editors who contribute to them to form what network scientists call a “giant component” — the “party” where everyone is in the same place and can interact directly or indirectly with everyone else.

article_lcc
Articles about breaking news events are in red, articles about recent but non-breaking news events are in blue, and articles about historical events are in green. The x-axis is time and the y-axis is the number of articles in the giant component.

Our results show that 90% of breaking news articles are indirectly connected to other breaking news articles through a shared editor within 24 hours of their creation. Moreover, for Wikipedia articles about recent but non-breaking events (in blue) and articles about historical events (in green), these non-breaking articles take over a year to form a giant component. To return to the party analogy, it only take 24 hours of notice for the party to form for breaking news articles but it takes over a year of notice for the party to form for other types of Wikipedia articles. This suggests that while there may not be professional journalists editing Wikipedia articles about breaking news events, there are editors who specialize in editing these events and they are the glue that ties these articles together.

For more, see our full paper, Hot off the Wiki: Structures and Dynamics of Wikipedia’s Coverage of Breaking News Events.

Brian Keegan, Northeastern University
Darren Gergle, Northwestern University
Noshir Contractor, Northwestern University

The Remixing Dilemma: The Trade-Off Between Generativity and Originality

Proponents of remix culture often frame remixing in terms of rich ecosystems where creative works are novel and highly generative. However, examples like this can be difficult to find. Although there is a steady stream of media being shared freely on the web, only a tiny fraction of these projects are remixed even once. On top of this, many remixes are not very different from the works they are built upon. Why is some content more attractive to remixers? Why are some projects remixed in deeper and more transformative ways?

Remix Diagram

We investigate these questions using data from Scratch — a large online remixing community where young people build, share, and collaborate on interactive animations and video games. The community was built to support users of the Scratch programming environment, a desktop application similar to Flash created by the Lifelong Kindergarten Group at the MIT Media Lab.

In our analysis, we found support for several popular theories about what makes projects remixable or generative: (1) Remixed projects are neither overly complex (i.e., too intimidating) nor too simplistic (i.e., vague and undefined); (2) Projects by prominent creators are more generative; (3) Remixes are more likely to attract remixers than de novo projects.

We also studied the originality of remixes and ask when remixing is more or less transformative. For example, a highly generative projects producing near-identical copies of previous projects may be viewed as less transformative or original. For a series of reasons — including the fact that increased generativity might come by attracting less interested, skilled, or motivated individuals — we suggest that each of the factors associated with generativity will also be associated with less original forms of remixing. We call this trade-off the remixing dilemma.

We find strong evidence of a trade-off:

  1. Projects of moderate complexity are remixed more lightly than more complicated projects. [Qualified, as we do not find evidence of increased originality for the simplest projects, as our theory predicted]
  2. Projects by more prominent creators tend to be remixed in less transformative ways.
  3. Cumulative remixing tends to be associated with shallower and less transformative derivatives.
Two plots of estimated values for prototypical projects. Panel 1 (left) display predicted probabilities of being remixed. Panel 2 (right) display predicted edit distances. Both panels show predicted values for both remixes and de novo projects from 0 to 1,204 blocks (99th percentile).

These results raise difficult, but important challenges, especially for designers of social media systems. For example, many social media sites track and display user prominence with leaderboards or lists of aggregate views. This technique may increase generativity by emphasizing and highlighting creator prominence while possibly decreasing the originality of the remixes elicited. Our results suggest that supporting increased complexity, at least for most projects, may have fewer drawbacks.

For more, see our full paper, “The remixing dilemma: The trade-off between generativity and originality.” Published in American Behavioral Scientist. 57-5, Pp. 643—663. (Official Link, Pay-Walled ).

Benjamin Mako Hill, Massachusetts Institute of Technology
Andrés Monroy-Hernández, Microsoft Research

The Rise and Decline of an Open Collaboration System: How Wikipedia’s reaction to sudden popularity is causing its decline

Summary (TL;DR):

To deal with the massive influx of new editors between 2004 and 2007, Wikipedians built automated quality control tools and solidified their rules of governance. In our paper, we observe that these reasonable and effective strategies for maintaining the quality of the encyclopedia have come at the cost of decreased retention of desirable new editors.

The Rise and Decline of the English Wikipedia
The number of active editors (>=5 edits/month) is plotted over time for the English language Wikipedia.

The Story:

In 2006, the English Wikipedia faced an amazing opportunity; the open encyclopedia was growing exponentially both in new content and new contributors. With this success and growth, however, came a problem — anonymous vandalism.

In Wikipedia, content is contributed openly by Internet users, often anonymously. As the English Wikipedia gained in popularity, the potential for malicious activity grew, as well. Many feared that the vandals could overwhelm the good-faith editors tasked with keeping them at bay.

In response, Wikipedians constructed a complex immune system to fight vandalism, incorporating several strategies, including:

  • Robots to automatically catch egregious cases.
  • Semi-automated systems that combined human judgment with computational efficiency.
  • Interface improvements to streamline the process of reverting malicious edits.

Our Results:

In early 2007, the English Wikipedia’s exponential growth in active editors changed directions and entered a steady decline. In this paper, we show that this decline was primarily due to a substantial drop in the retention of new, good-faith editors. Since 2007, desirable newcomers are more likely to have their work rejected, often through semi-autonomous vandal fighting tools (like Huggle). Furthermore, new users are being pushed out of policy articulation. During Wikipedia’s exponential growth period, Wikipedia’s policies and guidelines of behavior were effectively locked down against changes by new editors, and newcomers today struggle to find out where to ask for help.

For more, see our full paper, The Rise and Decline of an Open Collaboration System: How Wikipedia’s reaction to sudden popularity is causing its decline.
Aaron Halfaker, University of Minnesota
Stuart Geiger, University of California, Berkeley
Jonathan Morgan, University of Washington
John Riedl, University of Minnesota