When we think about how to ask a question online, a few options come to mind:
- Sending an e-mail to a few people we already know that might be knowledgeable enough to answer the question.
- Using a community web site like Quora or Yahoo! Answers to post our question and hope that someone will find that question and post the answer.
- Posting the question to our Facebook or Twitter status feeds, so that any of our friends or followers who might have the answer can respond.
These approaches can all work well, but what if your question is time-sensitive and requires some knowledge that isn’t known to anyone in your social network?
In our CSCW paper, we propose a new approach to question answering in which the public stream of status updates from a service like Twitter is used to identify people that might have the knowledge to answer a question and questions are directed to those users via the social network platform. We believe there are certain classes of questions for which this approach provides some advantage over other approaches:
- Questions about an event that are best answered soon after the event: The real-time nature of Twitter allows a question asker to identify strangers experiencing relevant events based on keywords in status updates and send questions immediately. For example, an asker could determine the approximate wait time through an airport security checkpoint by identifying people who are at that airport and asking about their wait time.
- Questions for which there may be a diversity of opinion: Questions could be sent targeting knowledgeable users across a range of different biases as detected from their social media posts and extract answers across that range. For example, we might ask questions about a camera to both users that prefer Nikon and those that prefer Canon, or target a political question to people of different party affiliations.
In this note, we evaluate one aspect of the feasibility of this approach, by sending a bunch of questions in two domains and examining the response rate to our questions. We also incorporate a couple manipulations: asking questions with an incentive and without, and asking follow-on questions of people who chose to answer. The two domains that we looked at were airport security wait times and digital camera product reviews. If you want to take a look at the accounts, see @tsatracker, @tsatracking, and @productqa. Data collected for the airport security case can be found at http://tsatracker.org/.
For this study, most of our decisions were made to maximize the expected response rate from the people targeted with questions. We did this because we expected that response rates would be quite low, even in the most favorable conditions. Our choice of domains followed from this decision. Many people talk about airport security and electronics products, and we believed that most people wouldn’t object to answering questions on these topics.
I’ll focus on the airport security case in the rest of this blog post. In that case, we asked a question of the form you see below.
We used two forms of this question. The one you see above includes the motivating sentence “Info will be used to help other travelers.” The second form was identical, but did not include the motivating sentence. We also asked the two types of questions from different accounts, so that there would not be any contamination if a user inspected the profile and previous tweets of the account that sent them a question.
The result was interesting. We asked 574 total questions in the airport security domain, 424 with the motivating sentence and 150 without. The response rate was 42% overall, and there was no significant difference in response between the two accounts. We were surprised at the fairly high response rate, which we had expected to be closer to 20%, and that the motivating sentence had no effect on response rate.
Without having complete data about how many messages were marked as spam for both accounts, it is hard to draw any hard conclusions about the effect of the motivating sentence. My interpretation of the results is this: There are two groups of people, those who are willing to answer questions and those who aren’t. The motivating question does not move people from one group to another. People who are willing to answer a question don’t care about the motivating sentence, as we can see from the lack of difference in response rate. People who are not willing to answer a question do care about the motivating sentence, but it only in terms of whether they consider the question to be reasonable. Without the sentence, this group does not understand why they received the message and may mark it as spam. With the sentence, this group seems to feel that the message was reasonable and will not mark it as spam. Of course, they won’t respond to the question either.
A more complete reporting of the data and results from the camera product question asking can be found in the paper (which I’ll link once it appears on the ACM DL).
Ultimately, I think this initial study suggests many more questions than it answers. How will this type of question asking work in other domains? What is the quality of answers that are received via this approach? What applications might be enabled via this approach? What are the long-term implications for social network platforms if people, and especially businesses, start to employ this approach regularly to collect data or opinions? I’m sure there are many others, and feel free to help me brainstorm in the comments.
Our immediate next steps are two-fold:
- We are building a system to facilitate question asking that utilizes a variety of social media analytics to identify people who have the knowledge, might be willing, and are available to provide an answer.
- Planning a study to investigate the quality of answers collected using this approach, which will probably also investigate question asking in at least one new domain.
We expect that social media engagement research building off of this concept will be the foundation for a research program at IBM Research – Almaden for at least the next couple years.
* I would like to thank Jeon Kang, my co-author on this work, and the rest of the Social Engagement team at IBM Almaden for their help in conducting this research. For more information, feel free to contact me directly.