Subcontracting Microwork

Mainstream crowdwork platforms treat microtasks as indivisible units; however, in our upcoming CHI 2017 paper, we propose that there is value in re-examining this assumption. We argue that crowdwork platforms can improve their value proposition for all stakeholders by supporting subcontracting within microtasks.

We define three models for microtask subcontracting: real-time assistance, task management, and task improvement:

  • Real-time assistance encompasses a model of subcontracting in which the primary worker engages one or more secondary workers to provide real-time advice, assistance, or support during a task
  • Task management subcontracting applies to situations in which a primary worker takes on a meta-work role for a complex task, delegating components to secondary workers and taking responsibility for integrating and/or approving the products of the secondary workers’ labor.
  • Task improvement subcontracting entails allowing a primary worker to edit task structure, including clarifying instructions, fixing user interface components, changing the task workflow, and adding, removing, or merging sub-tasks.

Subcontracting of microwork fundamentally alters many of the assumptions currently underlying crowd work platforms, such as economic incentive models and the efficacy of some prevailing workflows. However, subcontracting also legitimizes and codifies some existing informal practices that currently take place off-platform. In our paper, we identify five key issues crucial to creating a successful subcontracting structure, and reflect on design alternatives for each: incentive models, reputation models, transparency, quality control, and ethical considerations.

To learn more about worker motivations for engaging with subcontracting workflows, we conducted some experimental HITs on mTurk. In one, workers had the choice of whether to complete a complex, three-part task, or to choose to subcontract portions to other (hypothetical) workers (and give up some of the associated pay); we then asked these workers why they did or did not choose to subcontract each task component. Money, skills, and interests all factored into these decisions in complex ways.

Implementing and exploring the parameter space of the subcontracting concepts we propose is a key area for future research. Building platforms that support subcontracting workflows in an intentional manner will enable the crowdwork research community to evaluate the efficacy of these choices and further refine this concept. We particularly stress the importance of the ethical considerations component, as our intent in introducing and formalizing concepts related to subcontracting microwork is to facilitate more inclusive, satisfying, efficient, and high-quality work, rather than to facilitate extreme task decomposition strategies that may result in deskilling or untenable wages.

You can download our CHI 2017 paper to read about subcontracting in more detail.  (Fun fact — the idea for this paper began at the CrowdCamp Workshop at HCOMP 2015 in San Diego; Hooray for CrowdCamp!)

Subcontracting Microwork. Proceedings of CHI 2017. Meredith Ringel Morris (Microsoft Research), Jeffrey P. Bigham (Carnegie Mellon University), Robin Brewer (Northwestern University), Jonathan Bragg (University of Washington), Anand Kulkarni (UC Berkeley), Jessie Li (Carnegie Mellon University), and Saiph Savage (West Virginia University).

Accessible Crowdwork?

Crowdsourcing is an important, and growing, new class of digital labor that may well transform the future of work. Microtasking marketplaces, such as Amazon’s Mechanical Turk, are a key example of this new work form, which is still evolving and is largely unregulated. In addition to concern about lack of regulation regarding minimum wages for workers, it is unclear to what extent, if any, crowd labor marketplaces must comply with the Americans with Disabilities Act, which, among other provisions, requires employers to make “reasonable accommodations” for workers with disabilities. While several people have reported on the demographics of turkers with respect to factors like gender, age, country of origin, and socioeconomic status, such demographies have not reported on turkers’ disability status. Our research set out to understand whether people with disabilities are currently participating in crowd labor as workers, and what challenges they face in participating fully and equally in this new class of work.

Our methods included interviewing eight crowd workers with disabilities and nine professional job coaches who help match disabled workers with employment opportunities. Based on these interviews, we then developed an online survey which was taken by 486 American adults with disabilities, 12.4% of whom had tried crowd work.

Our findings established that people with disabilities are indeed participating in crowd work, and that many are eager to do so — benefits include a feeling of accomplishment, the ability to work from home and on a flexible schedule, and the ability to conceal one’s disability status if desired. However, crowd workers with disabilities faced several challenges, in three tiers of accessibility:

  • The first level of accessibility challenge was with the basic usability of crowdwork software, including both the crowdwork platform itself (e.g., AMT) and the many third-party sites that requesters’ tasks linked to. These sites had highly varying levels of compliance with issues such as compatibility with screen reader technology used by people who are blind, and these issues make completing microtasks frustrating, slow, or impossible for many people with disabilities.
  • The second level of accessibility challenge was in the way that microtasking workflows are structuredIdentifying tasks that are a match for a users’ abilities is quite difficult. For example, a hearing-impaired user might select a survey task, and discover halfway through the survey that they will be required to listen to audio for one of the survey questions. Time limits that are often embedded in tasks to weed out spammers or other types of malicious or lazy workers are quite problematic; workers with cognitive challenges such as dyslexia or motor impairments that require the use of alternative input devices might legitimately need a few extra seconds or minutes to complete a task.
  • The third level of accessibility challenge was in the fundamental accessibility of new job experiences; for example, learning about the availability of crowd work as an employment opportunity. Seven of the nine job coaches we interviewed had not been familiar with crowd work, for instance, suggesting a lack of dissemination of information about digital labor opportunities to the disability community. Further, those workers with disabilities who had tried crowd work were limited in their ability to grow in their crowd work careers by factors such as reputation systems in which they were penalized due to accessibility issues such as not completing tasks within the allotted time or not completing tasks that they discovered to be inaccessible partway through.

Our findings have several implications for the design of more accessible crowd work systems. For example, tasks could include metadata that indicates what abilities are required (sight, vision, speed, etc.) — such metadata could be manually added by task requesters or by other crowd workers, or could be automated through machine learning or recommender algorithms. Platforms could allow workers to sub-contract components of tasks to other workers, e.g., addressing our example of the hearing-impaired worker who encountered an audio-based component within a task. Optional self-identification of abilities or disabilities in a worker’s profile could help with better task suggestions or allow automatic time extensions or microbreak opportunities. Creating an online community specifically for crowd workers with disabilities could also provide important social and organizational opportunities, as well as build up a repository of knowledge about platforms and task requesters that are and are not accessible.

Currently, platform operators and task requesters do not appear to be legally compelled to provide accommodations such as those suggested above. However, we believe that it would benefit platform owners to voluntarily make their platforms and workflows more accessible, and to enforce compliance by third-party requesters — this would broaden their pool of eligible workers and help increase task completion rates, as well as enhancing that platform operators’ reputation.

Full access to participate in emerging forms of labor is important not only as an economic opportunity for people with disabilities, but as a social recognition of their full participation in all aspects of society; our research has highlighted important considerations for platform operators, job requesters, and policy makers to consider as a next step along the path to making this full access a reality.

You can read more in our CSCW 2015 paper: Zyskowski, K., Morris, M.R., Bigham, J.P., Gray, M.L., and Kane, S.K. Accessible Crowdwork? Understanding the Value in and Challenge of Microtask Employment for People with Disabilities. 

***

Meredith Ringel Morris is a Senior Researcher at Microsoft Research, where she designs, develops, and evaluates collaborative and social technologies.

Remote Shopping Advice: Crowdsourcing In-Store Purchase Decisions

Recent Pew reports, as well as our own survey, have found that consumers shopping in brick-and-mortar stores are increasingly using their mobile phones to contact others while they shop. The increasing capabilities of smartphones, combined with the emergence of powerful social platforms like social networking sites and crowd labor marketplaces, offer new opportunities for turning solitary in-store shopping into a rich social experience.We conducted a study to explore the potential of friendsourcing and paid crowdsourcing to enhance in-store shopping. Participants selected and tried on three outfits at a Seattle-area Eddie Bauer store; we created a single, composite image showing the three potential purchases side-by-side. Participants then posted the image to Facebook, asking their friends for feedback on which outfit to purchase; we also posted the image to Amazon’s Mechanical Turk service, and asked up to 20 U.S.-based Turkers to identify their favorite outfit, provide comments explaining their choice, and provide basic demographic information (gender, age).

Study participants posted composite photos showing their three purchase possibilities; these photos were the posted to Facebook and Mechanical Turk to crowdsource the shopping decision.
Study participants posted composite photos showing their three purchase possibilities; these photos were the posted to Facebook and Mechanical Turk to crowdsource the shopping decision.

Although none of our participants had used paid crowdsourcing before, and all were doubtful that it would be useful to them when we described what we planned to do at the start of the study session, the shopping feedback provided by paid crowd workers turned out to be surprisingly compelling to participants – more so than the friendsourced feedback from Facebook, in part because the crowd workers were more honest, explaining not only what looked good, but also what looked bad, and why! They also enjoyed the ability to see how opinions varied among different demographic groups (e.g., did male raters prefer a different outfit than female raters?).

Although Mechanical Turk had a speed advantage over Facebook, both sources generally provided multiple responses within a few minutes – fast enough that a shopper could get real-time decision-support information from the crowd while still in the store.

Our CSCW 2014 paper on “Remote Shopping Advice” describes our study in more detail, as well as how our findings can be applied toward designing next-generation social shopping experiences.

For more, see our full paper, Remote Shopping Advice: Enhancing In-Store Shopping with Social Technologies.

Meredith Ringel Morris, Microsoft Research
Kori Inkpen, Microsoft Research
Gina Venolia, Microsoft Research

Social Media Use by Mothers of Young Children

In our upcoming CSCW 2014 paper, we present the first formal study of how mothers of young children use social media, by analyzing surveys and social media feeds provided by several hundred mothers of infants and toddlers in the U.S.

Mothers overwhelmingly did not use Twitter for sharing information about their children, but nearly all of them used Facebook; for example, 96% reported having posted photos of their child on Facebook.

Our findings indicate several common trends in the way mothers use Facebook. Notably, the frequency of posting status updates falls by more than half after the birth of their child, and does not appear to rebound in the first few years of parenthood. However, the rate of photo-posting holds steady at pre-birth levels, meaning that photos comprise a relatively larger portion of posts than prior to the birth.

Contrary to popular belief (as exemplified by apps like unbaby.me that remove a perceived overabundance of baby photos from one’s Newsfeed), mothers do not appear to post exclusively about their offspring (about 15% of posts for first-time moms, and 11% for subsequent births). The first month post-birth contains the most baby-related postings, which then drop off.

After a baby is one month old, the percentage of a mother's posts that mention the child drop off.
After a baby is one month old, the percentage of a mother’s posts that mention the child drop off.

However, posts containing the child’s name receive far more likes and comments than other status updates; this likely gives them prominence in Facebook’s Newsfeed, likely reinforcing the image that mothers’ statuses are overly child-centric.

For more details about how new mothers use social media, including special groups such as those diagnosed with postpartum depression and those whose children have developmental delays, and discussion of how these findings can be used to design social networks and apps that support new moms’ needs, you can download our full paper,  Social Networking Site Use by Mothers of Young Children.
Meredith Ringel Morris, Microsoft Research

Crowd-Powered Replies to Public Twitter Questions

“Friendsourcing” information seeking by posting questions to a social networking site, like Twitter, is becoming increasingly common. However, unlike conventional search engines, friendsourced information seeking does not always return an answer. Examining over 85,000 information-seeking tweets from the public Twitter firehose, we found that only about a third received replies at all.

We created a Twitter agent that monitors the public feed for information-seeking questions, generates an answer via crowdsourced labor, and tweets the reply back to the original asker. This image shows an example:

Our crowd-powered Twitter agent serendipitiously offers advice about how to clean Sperrys (a type of shoe). The asker, pleased with the response, favorited the reply and retweeted it.
Our crowd-powered Twitter agent serendipitiously offers advice about how to clean Sperrys (a type of shoe). The asker, pleased with the response, favorited the reply and retweeted it.

Raters evaluating naturally occurring “friendsourced” answers to questions on Twitter and the answers generated via our crowdsourcing project rated them as being of equally high quality. Indeed, the people who received replies from our agent were generally quite pleased with the experience — over a third of them thanked our agent in some way, such as by favoriting the tweet or sending a reply message on Twitter.

Our work illustrates ways in which a “socially embedded search engine” can augment basic social network Q&A experiences.

For more, see our full paper, A Crowd-Powered Socially Embedded Search Engine. You can also view some sample Q&A exchanges our agent participated in at our project’s Twitter page.

Jin-Woo Jeong (Hanyang University), Meredith Ringel Morris, Jaime Teevan, & Dan Liebling (Microsoft Research)

CSCW 2014 Call for Papers

CSCW, the ACM’s conference on Computer Supported Cooperative Work and Social Computing, is an international and interdisciplinary conference focused on how technology intersects with social practices. The 2014 CSCW conference will be held in Baltimore, Maryland from February 15th– 18th, 2014. Paper submissions are due on May 31st, 2013.

We invite submissions that detail existing practices, inform the design or deployment of systems, or introduce novel systems, interaction techniques, or algorithms. The scope of CSCW includes, but is not limited to, social computing and social media, crowdsourcing, technologically-enabled or enhanced communication, CSCL and related educational technologies (e.g., MOOCs), multi-user input technologies (e.g., surface computing), collaboration, information sharing, and coordination. It includes socio-technical activities at work, in the home, in education, in healthcare, in the arts, for socializing, and for entertainment. New results or new ways of thinking about, studying, or supporting shared activities can be in these and related areas:

  • Social and crowd computing. Studies, theories, designs, mechanisms, systems, and/or infrastructures addressing social media, social networking, user-generated content, wikis, blogs, online gaming, crowdsourcing, collective intelligence, virtual worlds, collaborative information seeking, etc.
  • System design. Hardware, architectures, infrastructures, interaction design, technical foundations, algorithms, and/or toolkits that enable the building of new social and collaborative systems and experiences.
  • Theories and models. Critical analysis or organizing theory with clear relevance to the design or study of social and collaborative systems.
  • Empirical investigations. Findings, guidelines, and/or ethnographic studies relating to technologies, practices, or use of communication, collaboration, and social technologies.
  • Methodologies and tools. Novel methods or combinations of approaches and tools used in building systems or studying their use.
  • Domain-specific social and collaborative applications. Including for healthcare, transportation, gaming (for enjoyment or productivity), ICT4D, sustainability, education, accessibility, collective intelligence, global collaboration, or other domains.
  • Collaboration systems based on emerging technologies. Mobile and ubiquitous computing, game engines, virtual worlds, multi-touch technologies, novel display technologies, vision and gesture recognition systems, big data infrastructures, MOOCs, crowd labor markets, SNSes, sensor-based environments, etc.
  • Crossing boundaries. Studies, prototypes, or other investigations that explore interactions across disciplines, distance, languages, generations, and cultures, to help better understand how to transcend social, temporal, and/or spatial boundaries.

Papers should detail original research contributions. Papers must report new research results that represent a contribution to the field. They must provide sufficient details and support for their results and conclusions. They must cite relevant published research or experience, highlight novel aspects of the submission, and identify the most significant contributions. Papers are evaluated on the basis of originality, significance, quality of research, quality of writing, and contribution to conference program diversity.

Important Dates:

  • May 31st, 2013, 5 p.m. PDT: Submissions due
  • July 6th: First-round notification (Revise & Resubmit or Reject)
  • July 26th, 5 p.m. PDT: Revised submissions due
  • August 23rd: Final notifications (Accept or Reject)

For more, including formatting and submission instructions, see the complete CSCW 2014 CFP. For more information, you may contact the papers chairs at papers2014@cscw.acm.org.
Meredith Ringel Morris, Microsoft Research
Madhu Reddy, The Pennsylvania State University

 

CSCW 2013: Microblog Credibility Perceptions – Comparing the U.S. & China

Meredith Ringel Morris, Microsoft Research

From relatively harmless and perhaps even amusing celebrity death rumors to the more serious false reporting of dangers and damage during Hurricane Sandy, social media like Twitter often contain untrustworthy content — in addition to untruths, social media is rife with spam, surreptitious advertising, and imposter accounts. As microblogs emerge as an increasingly important news source both domestically and abroad (microblogs are particularly popular in China, where the most popular Weibo services have over 300 million users), ascertaining the credibility of tweets is an increasingly important skill, and one which our prior research has shown is underdeveloped. In our upcoming CSCW 2013 paper, we continue to investigate the mechanisms by which end-users judge the credibility of microblog content — several hundred people in the U.S. and China completed our online experiment in which we systematically manipulated several features of Twitter and Weibo posts to gauge their impact on crediblity perceptions.

Depicts sample English and Chinese microblog updates used in the experiment.

Continue reading

CSCW 2013: Collaborative Search Revisited

Meredith Ringel Morris, Microsoft Research

Collaborative and social search technologies have received a lot of buzz in the past few years, yet I would argue that none has really taken off — popular, mainstream search technologies remain optimized for the scenario of a person performing a solitary information-seeking task. However, you can probably easily come up with one (or several) recent occasions in which such solitary tools were inadequate or clumsy for your search task — perhaps you were out with friends trying to simultaneously use your smartphones to find a dinner spot, or you and a colleague were trying to coordinate a literature search for an upcoming journal article, or you were overwhelmed by the number of hotel suggestions a search engine returned for your upcoming trip to Hawaii and wanted to gather input from your Facebook friends about which would be the most promising. In my CSCW paper Collaborative Search Revisited, I present survey findings about current collaborative search practices, and reflect on how these have changed since my initial exploration of this topic, and on why research and commercial systems addressing this issue have not yet caught on.

Continue reading