Communicating Context to the Crowd

context-matters

Crowdsourcing has traditionally consisted of short, independent microtasks that require no background. The advantage of this strategy is that work can be decoupled and assigned to independent workers. But this strategy struggles to support tasks that are increasingly complex such as writing or programming and are not independent of their context.

For instance, imagine that you ask a crowd worker to write a biography for a speaker you’ve invited to your workshop. After the work is completed you realize that the biography is written in an informal, personal tone. This is not technically wrong, it’s just not what you had in mind. You realize that you could have added a line to your task description asking for a formal/academic tone. However, there are countless nuances to a writing task that can’t all be predicted beforehand. This is what we are interested in: the context of a task meaning the collection of conditions and tacit information surrounding the task (e.g. the fact that the biography is needed for an academic workshop).

IF THIS INFORMATION CAN’T BE PRE-PACKAGED AND SENT ALONG WITH THE TASK, WHAT CAN WE DO ABOUT IT? 

context

OUR APPROACH IS TO ITERATE: do some work, communicate with the requester, and edit to fix errors. How can we support communication between the requester and crowd workers to maximize benefits while minimizing costs? If achieved, this goal would create the conditions for crowd work that is more complex and integrated than currently possible.

The main take away is to support this communication through structured microtasks. We have designed 5 different mechanisms for structured communication:

context3

We compare these methods in two studies, the first measuring the benefit of each mechanism, and the second measuring the costs to the requester (e.g. cognitive demand). We found that these mechanisms are most effective when writing is in the early phases. For text that is already high quality, the mechanisms become less effective and can even be counter-productive.

context4

We also found that the mechanisms had varying benefits depending on the quality of the initial text. Early on, when content quality is poor, the requester needs to communicate major issues. Therefore identifying the “main problem” was most effective at improving writing. Later, for average quality content, the different mechanisms have relatively similar added value.

Finally, we found that the cost of a mechanism for the requester is not always correlated with the value that it adds. For instance, for average quality paragraphs, commenting/editing was very costly but did not provide more value than simply highlighting.

 

For more, see our full paper, Communicating Context to the Crowd for Complex Writing Tasks.
Niloufar Salehi, Stanford University
Jaime Teevan, Microsoft Research
Shamsi Iqbal, Microsoft Research
Ece Kamar, Microsoft Research

Don’t Bother Me. I’m Socializing!

IT DOES BOTHER US when we see our friends checking their smartphones while having a conversation with us. Although people want to focus on a conversation, it is hard to ignore a series of notification alarms coming from their smartphones. It is reported that smartphone users receive an average of tens to hundreds of push notifications a day [1,2]. Despite its usefulness in immediate delivery of information, an untimely smartphone notification is considered a source of distraction and annoyance during social interactions.

deferred_notifications
(Left) Notifications interrupt an ongoing social interaction. (Right) Notifications are deferred to a breakpoint, in-between two activities, so that people are less interrupted by notifications.

 

TO ADDRESS THIS PROBLEM, we have proposed a novel notification management scheme, in which the smartphone defers notifications until an opportune moment during social interactions. A breakpoint [3] is a term originated from psychology that describes a unit of time in between two adjacent actions. The intuition is that there exist breakpoints in which notifications do not, if so minimally, interrupt a social interaction.

video_survey_screenshot
A screenshot of the video survey. Participants are asked to respond whether this moment is appropriate to receive a notification.

TO DISCOVER SUCH BREAKPOINTS, we devised a video survey in which participants watch a typical social interaction scenario and respond whether prompted moments in the video are appropriate moments to receive smartphone notifications. People responded that the following four types of breakpoints are appropriate breakpoints in a social interaction; (1) a long silence, (2) a user leaving the table, (3) others using smartphones, and (4) a user left alone.

SCAN_social_context
Types of social context detected by SCAN.

BASED ON THE INSIGHTS FROM THE VIDEO SURVEY, we designed and implemented a Social Context-Aware smartphone Notification system, SCAN, that defers smartphone notifications until a breakpoint. SCAN is a mobile application that detects social context using only built-in sensors. It also works collaboratively with the rest of the group members’ smartphones to sense collocated members, conversation, and others’ smartphone use. SCAN then classifies a breakpoint based on the social context and decides whether to deliver or defer notifications.

SCAN HAS BEEN EVALUATED on ten groups of friends in a controlled setting. SCAN detects four target breakpoint types with high accuracy (precision= 92.0%, recall= 82.5%). Most participants appreciated the value of deferred notifications and found the selected breakpoints appropriate. Overall, we demonstrated that breakpoint-based smartphone notification management is a promising approach to reducing interruptions during social interactions.

WE ARE CURRENTLY EXTENDING SCAN to apply it to various types of social interactions. We also aim to add personalized notification management and to address technical challenges such as system robustness and energy efficiency. Our ultimate goal is to release SCAN as an Android application in Google Play Store and help users to be less distracted by smartphone notifications during social interactions.

You can check out our CSCW 2017 paper to read about this work in more detail.  

“Don’t Bother Me. I’m Socializing!: A Breakpoint-Based Smartphone Notification System”. Proceedings of CSCW 2017. Chunjong Park, Junsung Lim, Juho Kim, Sung-Ju Lee, and Dongman Lee (KAIST)


[1]”An In-situ Study of Mobile Phone Notifications”. Proceedings of MobileHCI 2014. Martin Pielot, Karen Church, and Rodrigo de Oliveira.
[2] “Hooked on Smartphones: An Exploratory Study on Smartphone Overuse Among College Students”. Proceedings of CHI 2014. Uichin Lee, Joonwon Lee, Minsam Ko, Changhun Lee, Yuhwan Kim, Subin Yang, Koji Yatani, Gahgene Gweon, Kyong-Mee Chung, and Junehwa Song.
[3] “The perceptual organization of ongoing behavior”. Journal of Experimental Social Psychology 12, 5 (1976), 436–450. Darren Newtson and Gretchen Engquist.