Report: HCOMP 2016 Workshop on Mathematical Foundations of Human Computation

The HCOMP Workshop on Mathematical Foundations of Human Computation was held at HCOMP 2016 in Austin, Texas on November 3, 2016. The goal of the workshop was to bring together researchers across disciplines to discuss the future of research on the mathematical foundations of human computation, with a particular emphasis on the ways in which theorists can learn from the existing empirical literature on human computation and the ways in which applied and empirical work on human computation can benefit from mathematical foundations.

There has been great progress in human computation in the past decade. However, human computation is still not well understood from a foundational mathematical perspective.  Mathematical foundations are important to influence and shape the future of human computation. They provide frameworks for formalizing desirable properties of systems (e.g., correctness, optimality, scalability, privacy, fairness), predicting the impact of design decisions, designing systems with provable guarantees, and performing counterfactual analysis, which is notoriously hard to do empirically.

The workshop engaged researchers across different disciplines. The workshop featured an opening talk to set the stage for the day and four invited talks covering broad topics including a model of human conscious computation with applications to password generation, meta-unsupervised-learning methods, the design of a programming framework for humans in the loop, and how to increase productivity using (potentially crowdsourced) microtasks. The program also included six contributed talks and ample time reserved for general discussion. The slides for many of the talks are available on the workshop website.

The workshop concluded with an open problem session in which participants delivered short informal pitches of open problems related to the workshop theme. The main themes of the open problems that were proposed include:

  • The characterization of the important properties of computational models of human cognition. For example, how do we classify or quantify the hardness of problems in human computation?
  • Evaluation metrics for human computation problems. What properties do we want human computation to achieve?
  • The design of models that accurately incorporate human behavior. At a minimum, this requires diving into the literature from other disciplines, such as psychology and behavioral economics. Ideally it should be explored in an interdisciplinary manner.
  • How to develop theory and design algorithms that are robust to errors in modeling and experiments.

 

Workshop organizers:

Shuchi Chawla, University of Wisconsin – Madison
Chien-Ju Ho, Cornell University
Michael Kearns, University of Pennsylvania
Jenn Wortman Vaughan, Microsoft Research
Santosh Vempala, Georgia Tech

Humans, Machines and the Future of Work

ConferenceHumans, Machines, and the Future of Work

Automation, driven by technological progress, has been increasing inexorably for the past several decades. Two schools of economic thinking have been engaging in a debate about the potential effects of automation on jobs, employment and human activity: will new technology spawn mass unemployment, as the robots take jobs away from humans? Or will the jobs robots take over release or unveil – or even create – demand for new human jobs?

The debate has flared up again recently because of technological achievements that recently enabled a Google software program called AlphaGo to beat Go world champion Lee Sedol, a task considered even harder than beating the world’s chess champions.

Ultimately the question boils down to this: are today’s modern technological innovations like those of the past, which made obsolete the job of buggy maker, but created the job of automobile manufacturer? Or is there something about today that is markedly different? This is not a new concern. Dating back at least as far as the Luddites of early 19th-century Britain, new technologies caused fear about the inevitable changes they bring.

There is considerable evidence that this concern may be justified. Eric Brynjolfsson and Andrew McAfee of MIT recently wrote:

For several decades after World War II the economic statistics we care most about all rose together here in America as if they were tightly coupled. GDP grew, and so did productivity — our ability to get more output from each worker. At the same time, we created millions of jobs, and many of these were the kinds of jobs that allowed the average American worker, who didn’t (and still doesn’t) have a college degree, to enjoy a high and rising standard of living. But … productivity growth and employment growth started to become decoupled from each other.

As the decoupling data show, the U.S. economy has been performing quite poorly for the bottom 90 percent of Americans for the past 40 years. Technology is driving productivity improvements, which grow the economy, but most people are not seeing any benefit from this growth. While the U.S. economy is still creating jobs, it is not creating enough of them. The labor force participation rate, which measures the active portion of the labor force, has been dropping since the late 1990s.

An in-depth discussion of these issues will take place at a conference on Humans, Machines, and the Future of Work, Dec. 5-6, 2016.

Why Is That Relevant? Collecting Annotator Rationales for Relevance Judgments

When collecting human ratings or labels for items, it can be difficult to measure and ensure data quality control, due to both task subjectivity (i.e., lack of a gold standard against which answers can be easily checked), as well as lack of transparency into how individual judges arrive at their submitted ratings. Using a paid microtask platform like Mechanical Turk to collect data introduces further challenges: crowd annotators may be inexpert, minimally trained, and effectively anonymous, and only rudimentary channels may be provided to interact with workers and observe work as it is performed.

To address this challenge, we investigate a very simple idea: requiring judges to provide a rationale supporting each rating decision they make. We explore this idea for the specific task of collecting relevance judgments for search engine evaluation. A relevance judgment rates how relevant a given web page is to a particular search query. We define a rationale as a direct excerpt from a web page (e.g., which a worker might easily provide by copy-and-paste or highlighting).

To illustrate this idea, we show a simple example below in which a user is assumed to be searching for information about symptoms of jaundice, and a worker is asked to rate the relevance of the Wikipedia page on jaundice. The worker might rate the page as relevant and support his/her decision by highlighting an excerpt describing jaundice symptoms, as shown.

What are the symptoms of jaundice?

While this idea of rationales is quite simple and easy to implement — good things for translating ideas from research to practice! — rationales appear to be remarkably useful, offering a myriad of benefits:

  • Rationales enhance transparency and value. While traditional labor models involve high-quality interpersonal interactions and transparency, this comes at significant expense. On the other hand, microtasking typically greatly improves efficiency and costs, but at the cost of low transparency and poor communication channels. Rationales represent a middle-ground between these extremes, with textual excerpts comprising a very light-weight form of communication from worker to requester to enhance transparency. Moreover, because rationales enhance data interpretability, they increase the value of collected data.  This makes rationales useful to collect from traditional annotators as well as crowd annotators, and beyond ensuring initial data quality, rationales increase the enduring value of collected data for all future users as well.
  • Rationales enhance reliability and accountability. Identifying a focused context for explaining annotator decisions as they are made is generally helpful and especially so for subjective tasks without clear answers. By making work more verifiable, arbitrary answers can no longer be given without justification, making annotators more accountable for their work and decreasing any temptation to rush or cheat. In addition, rationales let us establish alternative truths, e.g., in cases of disagreement between raters, or when rating decisions seem surprising or unintuitive without explanation. Moreover, just as rationales allow requesters to understand worker thought processes, they also enable application of sequential task design in which workers refine or refute one another’s reasoning. Such sequential design is critical for minimizing reliance on “experts” to verify data quality when scaling up data collection.
  • Rationales enhance inclusivity. We hypothesized that the improved transparency and accountability of rationales would enable us to remove all traditional barriers to participation without compromising data quality. By allowing anyone interested to work, we support greater scalability of data collection, greater diversity of responses, and equal opportunity access to income. Our experiments on Mechanical Turk imposed no worker qualifications to restrict anyone from performing our tasks, and we were able to successfully collect high quality data without imposing any worker restrictions.

It gets even better. Despite the seemingly extra work required to collect rationales in addition to relevance judgments, in practice rationales do not require extra time (and therefore cost) to collect! Over a series of experiments collecting nearly 10,000 relevance judgments, we found that workers produce higher quality relevance judgments simply by being asked to provide a rationale, and prolific workers require virtually no extra time to provide rationales in addition to ratings. Intuitively, rationales appear to merely capture explicitly the implicit cognitive work inherent in providing ratings: workers already have to find relevant text to perform the rating task, and asking them to simply report that text requires essentially no extra work.

Finally, it’s intuitive that workers who select similar rationales are likely to produce similar relevance judgments, and we can exploit such overlap between worker rationales in aggregating worker judgments in order to further improve data quality.

In sum, our results suggest that rationales may be a remarkably simple yet powerful tool for crowdsourced data collection, particularly for difficult or subjective tasks. There’s much more to our study, so we encourage you to read our full paper! The link to our paper is below, along with links to the presentation slides and our collected data.

Tyler McDonnell, Matthew Lease, Mucahid Kutlu, and Tamer Elsayed. Why Is That Relevant? Collecting Annotator Rationales for Relevance Judgments. In Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP), pages 139-148, 2016. Best Paper Award. [ Slides & Data ]

Machine Learning for Dummies

By Eric Holloway and Robert J. Marks II

Dummies Commentary
Humans are slow and sloppy, why do we want human guided machine learning?

Since the 1970s, we’ve known humans can find approximate solutions to NP-Complete (really hard) problems more efficiently than the best algorithms (Krolak 1971). The best algorithms scale quadratically, while humans scale linearly (Dry 2006). We also know many of the most widely used and successful machine learning algorithms are NP-Hard to train optimally (Diettrich 2000). This suggests a human/machine hybrid can produce better models than machine learning alone.

Solution times
Polynomial regression of human solution times against problem sizes (Dry 2006).

However, there are problems with human interaction. The hardest problem is visualization. It is hard visualizing data with more than 3 dimensions. So, we perform dimension reduction by projecting data onto two dimensions. We then collect many weak models (humans draw boxes) from multiple projections to build a strong model; known as boosting in machine learning.

User Interface
User interface for Amazon Mechanical Turk HIT.

Combining crowdsourcing and boosting, we use Amazon’s Mechanical Turk to collect the models. The data is transformed by the models into a feature space. Then, we use linear regression to classify new data.

Linear Regression
Linear regression classification of human produced features.

We test the human/machine hybrid on one artificial dataset and four real world datasets, all with ten or more dimensions. This hybrid is competitive with machine only linear regression on the untransformed data.

Results
Results of linear regression classification using just machine, and using human/machine hybrid.

You can read more about our work in the HCOMP 2016 paper High Dimensional Human Guided Machine Learning.

A demo is available for a limited time.

References

Krolak, P., Felts, W., & Marble, G. (1971). A man-machine approach toward solving the traveling salesman problem. Communications of the ACM, 14(5), 327-334.

Dry, M., Lee, M. D., Vickers, D., & Hughes, P. (2006). Human performance on visually presented traveling salesperson problems with varying numbers of nodes. The Journal of Problem Solving, 1(1), 4.

Dietterich, T. G. (2000, June). Ensemble methods in machine learning. In International workshop on multiple classifier systems (pp. 1-15). Springer Berlin Heidelberg.

 

Rethinking experiment design as algorithm design

As experimentation in the behavioral and social sciences moves from brick-and-mortar laboratories to the web, techniques from human computation and machine learning can be combined to improve the design of experiments. Experimenters can take advantage of the new medium by writing complex computationally mediated adaptive procedures for gathering data: algorithms.

In a paper to be presented at CrowdML’16, we consider this algorithmic approach to experiment design. We review several experiment designs drawn from the fields of medicine, cognitive psychology, cultural evolution, psychophysics, computational design, game theory, and economics, describing their interpretation as algorithms. We then discuss software platforms for efficient execution of these algorithms with people. Finally, we consider how machine learning can optimize crowdsourced experiments and form the foundation of next-generation experiment design.

Consider the transmission chain, an experimental technique that, much like the children’s game Telephone, passes information from one person to the next in succession. As the information changes hands, it is transformed by the perceptual, inductive, and reconstructive biases of the individuals. Eventually, the transformation leads to erasure of the information contained in the input, leaving behind a signature of the transformation process itself. For this reason, transmission chains have been particularly useful in the study of language evolution and the effects of culture on memory.

When applied to functional forms, for example, transmission chains typically revert to a positive linear function, revealing a bias in human learning. In each row of the following figure, reprinted from Kalish et al. (2007), the functional relationship in the leftmost column is passed down a transmission chain of 9 participants, in all four instances reverting to a positive linear function.

Transmission chains can be formally modeled as a Markov chain by assuming that perception, learning, and memory follow the principles of Bayesian inference. Under this analysis, Bayes’ rule is used to infer the process that generated the observed data. A hypothesis is then sampled from the posterior distribution and used to generate the data passed to the next person in the chain. When a transmission chain is set up in this way, iterated learning is equivalent to a form of Gibbs sampling, a widely-used Markov chain Monte Carlo algorithm. The convergence results for the Gibbs sampler thus apply, with the prior as the stationary distribution of the Markov chain on hypotheses. This equivalence raises the question of whether other MCMC-like algorithm can form the basis of new experiment designs.

For more, attend CrowdML at NIPS in Barcelona or see our full paper: Rethinking experiment design as algorithm design.

Jordan W. Suchow, University of California, Berkeley
Thomas L. Griffiths, University of California, Berkeley

Probabilistic Modeling for Crowdsourcing Partially-Subjective Ratings

By An T. Nguyen, Matthew Halpern, Byron C. Wallace, Matthew Lease
University of Texas at Austin and Northeastern University

Imagine you are a graphics designer working on a logo for a major corporation. You have a preliminary design in mind, but now you need to hammer out the fine-grained details, such as the exact font, sizing, and coloring to use. In the past you might have solicited feedback from colleagues, in-person user studies, but you recently heard the power of the “crowd” and you are intrigued. You launch an Amazon Mechanical Turk task asking workers to rate different logo permutations on a 1-5 scale, where 1 and 5 stars are the least and most satisfactory ratings, respectively.

You now have a rich multi-variate dataset for the workers’s opinions, but did all workers undertake the task in good faith? For example, a disinterested worker could have just picked a score and a more malicious worker could have intentionally picked an opposite score. You might like to detect any such cases and filter them out of your data. You might even want to extrapolate from the collected scores to other versions not scored. How could you do any of this?

While the need to accurately model data quality is key, this is a challenging partially-subjective rating scenario where each response is open but partially constrained. In the logo example, the worker is allowed to select any score in the range 1-5. Personal opinions may differ significantly on a score for a specific logo design. This is in contrast to conventional crowdsourcing tasks, such as image tagging and finding email addresses, where one expects a single, correct answer to each question being asked. While a great deal of prior work has proposed ways to model data for such objective tasks, far less work has considered modeling data quality and worker performance under more subjective task scenarios.

The key observation we make in our paper is that the worker data for these partially-subjective tasks, where worker labels are partially ordered (i.e. scores from one to five), are heteroscedastic in nature. Therefore we propose a probabilistic, heteroscedastic model where the means and variances of worker responses are modeled as functions of instance attributes. In other words, the variability of scores can itself vary across the different parameters. Consider the results as the font size of a logo is varied. We would expect that most workers would give the logos with the smallest and largest font sizes low scores. However, the range of scores for the middle range of fonts is going to be more varied.

We demonstrate the effectiveness of our model on a large dataset of nearly 25,000 Mechanical Turk ratings of user experience when viewing videos on smartphones with varying hardware capabilities. Our results show that our method is effective at both predicting user ratings and in detecting unreliable respondents, which is particularly valuable and little studied in this domain of subjective tasks where there is no clear, objective answer.

The link to our full paper is below, along with links to shared data and code, and we welcome any comments or suggestions!

An Thanh Nguyen, Matthew Halpern, Byron C. Wallace, and Matthew Lease. Probabilistic Modeling for Crowdsourcing Partially-Subjective Ratings. In Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2016. 10 pages. [ bib | pdf | data | sourcecode ]

Pairwise, Magnitude, or Stars: What’s the Best Way for Crowds to Rate?

IS THE UBIQUITOUS FIVE STAR RATING SYSTEM IMPROVABLE?
We compare three popular techniques of rating content: five star rating, pairwise comparison, and magnitude estimation.

We collected 39 000 ratings on a popular crowdsourcing platform, allowing us to release a dataset that will be useful for many related studies on user rating techniques.
The dataset is available here.

METHODOLOGY
We ask each worker to rate 10 popular paintings using 3 rating methods:

  • Magnitude: Using any positive number (zero excluded).
  • Star: Choosing between 1 to 5 stars.
  • Pairwise: Pairwise comparisons between two images, with no ties allowed.

We run 6 different experiments (one for each combination of these three types) with 100 participants in each of them. We can thus analyze the bias given by the rating system order, and the results without order bias by using the aggregated data.

At the end of the rating activity in the task, we dynamically build the three painting rankings induced by the choices of the participant, and ask them which of the three rankings better reflects their preference (the ranking comparison is blind: There is no indication on how each ranking has been obtained, and their order is randomized).

Graphical interface to let the worker express their preference on the ranking induced by their own ratings
Graphical interface to let the worker express their preference on the ranking induced by their own ratings

WHAT’S THE PREFERRED TECHNIQUE?
Participants clearly prefer the ranking obtained from their pairwise comparisons.  We notice a memory bias effect: The last technique used is more likely to get the most accurate description of the real user preference. Despite this, the pairwise comparison technique obtained the maximum number of preferences in all cases.

Number of expression of preference of the ranking induced by the three different techniques, grouped by the order in which the tests have been run
Number of expression of preference of the ranking induced by the three different techniques

EFFORT
While the pairwise comparison technique clearly requires more time than the other techniques, it would be comparable in terms of time with the other techniques using a dynamic test system (of order NlogN).

Average time per test, grouped by the order in which the tests have been run
Average time per test

WHAT DID WE LEARN?

  • Star rating is confirmed to be the most familiar way for users to rate content.
  • Magnitude is unintuitive with no added benefit.
  • Pairwise comparison, while requiring a higher number of low-effort user ratings, best reflects intrinsic user preferences and seems to be a promising alternative.

For more, see our full paper, Pairwise, Magnitude, or Stars: What’s the Best Way for Crowds to Rate?

Alessandro Checco, Information School, University of Sheffield
Gianluca Demartini, Information School, University of Sheffield

Remembering David Martin and his ethnographic studies of crowdworkers

http://www.humancomputation.com/2016/martin.html

This post is a late tribute to Dr. David B. Martin, who passed away in June 2016. Dave was an ethnographer, inherently interested in the tension between tools, progress and the reality of everyday life. Dave had spent the last few years doing ethnomethodological studies of crowdwork and crowdworkers, together with Jacki O’Neill, Ben Hanrahan, and myself at Xerox Research Centre Europe, bringing forth the invisible, hidden work, and various issues of the worker communities into public view. Through ethnomethodology we studied how people collaborated with each other, how they communicated, and the mundane things they did to make crowdwork work. His vision and efforts in the area of crowdsourcing will continue to live on through his work, constantly inspiring us.

The team’s first encounter with crowdsourcing was in 2011, when Dave and Jacki began studying outsourcing work to investigate if crowdsourcing could be used for work that is traditionally outsourced. Following the tradition of the work practice studies at Xerox, they conducted ethnographic studies of BPO work [1], specifically of low-skilled, piece-rate work of healthcare form digitization [2] to see if it could be crowdsourced. Dave studied at-home work in the US and Jacki studied in-office data entry work in India. These studies showed how the outsourcers’ current workflow was sensitively orchestrated to achieve high quality and rapid turnaround times at minimum cost. Even in a low-skilled, piece-rate environment, it was the subtleties of the employer-employee relationship that enabled the work to be done on time and to quality.

The Xerox team then went on to explore crowdsourcing further, especially the lived work of microtask crowdwork: the work practices that make crowdwork work as experienced by crowdworkers. At this point the research team expanded from Dave and Jacki, and got Shourya Roy and colleagues from the Xerox India lab involved. And later joined, Ben Hanrahan, our in-house crowdwork developer at Xerox lab in Grenoble (and speaker at HCOMP at ‘Remembering David B. Martin and his Ethnographic Studies of Crowd Workers’ event [3] on the 31st of October 2016), and myself, PhD candidate from University of Nottingham, UK.

Our interest in crowdsourcing was to enquire: Who are the people who do crowdwork? What are their work practices? What are their personal and work lives like? Why do they this kind of work? What are their expectations from, and issues with crowdwork? How do the various technologies they use fit with and support these practices?

Our enquiries began with Dave’s study [4] of the Turker Nation Forum, predominantly Turkers based in the US, and my PhD, studying the Turkers based in India. Dave’s study was ground-breaking both methodologically (being an EM ethnography of a forum) and in its findings. It painted the first detailed picture of the lived world of crowdworking. Turkers oriented to turking as ‘work’ and AMT as a ‘labour market’ which came as a surprise to much of the research community. The study showed the motivations and ethical codes the workers followed, and the need for fairness and relationship-building in crowdwork. In platforms like Amazon Mechanical Turk not only does it offer no support for the management of rapid, high quality workflows, but also deliberately designs out the relationship between workers and the organisation, reduces accountability and replaces complex social, organisational & financial motivators almost solely with monetary ones.

Before publication of ‘Being a Turker’, Dave sought and got approval from members of the Turker Nation forum for his paper, as he wanted to ensure he was truly representing the concerns of the Turker community. He truly prized the emails and forum comments about his paper from the members of the turker community, and although we do not have these communications with us, it shows us what really mattered to him.

Dave, Jacki and Ben were part of the analysis of the ethnographic data I collected in India in 2013. Dave took over my phd supervision from Jacki, in 2014, honed by analytical skills and mentored me on the ethnomethodological journey through my phd. The rigorous discussions and debates with him on Zimmerman, Becker and so on helped frame the key discussions in my thesis. With Andy Crabtree and Tom Rodden from University of Nottingham we wrote the ‘Understanding Indian Crowdworkers’ as an introduction[5] to the Indian turkers’ story. The first major contribution from the phd was the ‘Turk-life in India’ paper [6] that showed the various levels of digital and English-language literacy amongst turkers, the role infrastructure and technology played in turking and social nature of turking. ‘Turking in a global labour market’ was a comparative[7] narrative of workers based in the US and India, the understandings they had of the market and of each other in a transnational labour platform that brought them head-on into a very competitive work environment and what that meant for designers, policy makers, researchers and activists. This also led us to design a conceptual tool [8] – TurkBench, meant to provide dynamic scheduling and provide crowdworkers with personalized market visualization and session management.

Our collective research into crowdsourcing showed that turkers were collaborative agents, who cared about their work, the money they earned, their reputation and relationship with the requesters. It also brought to the fore the discussion on design, since in crowdsourcing, the labour market is embodied through the technology you create, and thus have far-reaching consequences in the lives of the crowdworkers, not just in their ability to successfully complete quality work, but also in their working conditions and standards of living. Dave wanted to ensure that technology-designers were not designing in a bubble, and were aware of their responsibilities, and the consequences their design had on ‘real’ people.

Dave had been working to support this cause since his first brush with crowdsourcing. One of his final projects was the book chapter “Understanding The Crowd: ethical and practical matters in the academic use of crowdsourcing” which highlights the principles of ‘professional ethics’ that were put in place for the safety and well-being of the research subjects, in this case, the crowdworkers, reminding researchers of ‘ethical conduct’ during crowd-based research, providing references and further guidance to use when using the crowd in supporting empirical work. He ignited a spark in the conversations at a Dagstuhl crowdsourcing Seminar [9] in November 2015 “Evaluation in the Crowd: Crowdsourcing and Human-Centred Experiments” where computer scientists from the fields like visualization, psychology, graphics, multimedia assembled to discuss the role of crowdsourcing in empirical research work. Dave challenged the perspectives of the academicians present, reminding them to never overlook the people who provided experimental data for them. The book arising from that seminar is to be published by Springer in early 2017.

The last event he took part in at was at the University of Oxford at the Connected Life Conference [10] in June 2016, where he, along with colleagues, participated in an interdisciplinary debate on socio-digital practices of collective action, continuing his fight to design technologies and advance policy work for workers doing crowdwork, who are ignored, hidden behind algorithmic distribution and assembly of work.

Drawing to a close, I’d like to borrow Jacki’s words from her video at HCOMP 2016, something that echoes with everyone who had known him, met him at conferences or at a pub.
“Dave will be remembered for his passion for people, philosophy, a good argument and a pint. He conducted ethnomethodology without indifference. His sense of fairness and integrity permeated both his social and his work life. And in the fight to design technologies which give agency to typically undervalued workers, we have lost a kindred spirit. The greatest tribute we can give to Dave, is to remember that the crowd has a human face.”

We miss you Dave.

Neha Gupta

References:

[1] Relationship-based Business Process Crowdsourcing?
Jacki O.Neill and David Martin. In IFIP Conference on Human-Computer Interaction, pp. 429-446, 2013.

[2] Form digitization in BPO: from outsourcing to crowdsourcing?
Jacki O’Neill, Shourya Roy, Antonietta Grasso, and David Martin. In Proceedings of the 31st ACM SIGCHI conference on Human factors in computing systems, pp. 197-206, 2013.
[3] HCOMP talk http://www.humancomputation.com/2016/martin.html
[4] Being a Turker
David Martin, Benjamin V. Hanrahan, Jacki O’Neill, and Neha Gupta. In Proceedings of the 17th ACM CSCW Conference on Computer Supported Cooperative Work & Social Computing, pp. 224-235, 2014.

[5] Understanding Indian Crowdworkers
Neha Gupta, Andy Crabtree, Tom Rodden, David Martin, and Jacki O.Neill. Back to the Future of Organizational Work: Crowdsourcing and Digital Work Marketplaces Workshop at CSCW 2014.

[6] Turk-Life in India
Neha Gupta, David Martin, Benjamin V. Hanrahan, and Jacki O’Neill. In Proceedings of the 18th ACM GROUP: International Conference on Supporting Group Work, pp. 1-11. 2014.

[7] Turking in a Global Labour Market
David Martin, Jacki O.Neill, Neha Gupta, and Benjamin V. Hanrahan. In Proceedings of the 19th ACM CSCW Conference on Computer Supported Cooperative Work & Social Computing, pp. 39-77, 2016

[8] TurkBench: Rendering the Market for Turkers
Benjamin V. Hanrahan, Jutta K. Willamowski, Saiganesh Swaminathan, and David B. Martin. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI), pp.1613-1616.

[9] Dagstuhl seminar http://www.dagstuhl.de/en/program/calendar/semhp/?semnr=15481

[10] Oxford Conference http://connectedlife.oii.ox.ac.uk/2016conference/programme/

How to Best Serve Micro-tasks to the Crowd when there Is Class Imbalance

DOES CLASS IMBALANCE IN RELEVANT JUDGMENT AFFECTS PERFORMANCE?
We study the effect on crowd worker efficiency and effectiveness of the dominance of one class in the data they process. We aim at understanding if there is any bias in workers seeing many negative examples in the identification of positive labels.

We run comparative experiments where we measure label quality and work efficiency over different class distribution settings both including label frequency (i.e., one dominant class) as well as ordering (e.g., positive cases preceding negative ones).

batch types
Order of the document classes in each batch, in blue for ‘relevant’ and red for ‘non-relevant’

We used data from TREC8.  To measure effects of class imbalance, we used two different relevant/non-relevant ratios in a batch of judging tasks: 10%-90% and 50%-50%.

RESULTS
When the relevant documents are shown before the non-relevant ones we obtain the highest precision, while the worst precision is obtained when they are shown at the end of the batch.
Moreover, in batch 2 we observe a low number of true positives and a large number of false positive judgments by the workers, which shows how 90% of non-relevant documents shown at the beginning of the batch create a bias in the workers’ notion of relevance.

Mean judgment accuracy, precision and recall for each setting
Mean judgment accuracy, precision and recall for each setting

When classes are balanced, there is no statistically significant difference in the performance between different orders. On the other hand, seeing a similar number of positive and negative documents leads to good performance with more than 60% accuracy in all the three order settings.

WHAT DID WE LEARN?
When most of the documents are non-relevant and the few relevant ones are presented first, workers perform better. This is a positive result which can be easily applied in practice as in real IR evaluation settings most of the documents to be judged are non-relevant.

Including in the first positions documents known to be relevant will both prime workers on relevance as well as allow for training.

While in a real setting it is not possible to put relevant documents first, it would still be possible to order documents by attributes indicating their relevance (e.g., retrieval rank, number of IR systems retrieving the document, etc.) thus presenting first to the workers the documents with higher probability of being relevant.

For more, see our paper, THE EFFECT OF CLASS IMBALANCE AND ORDER ON CROWDSOURCED RELEVANCE JUDGMENTS

Rehab K. Qarout, Information School, University of Sheffield
Alessandro Checco, Information School, University of Sheffield
Gianluca Demartini, Information School, University of Sheffield

Social Sampling and the Multiplicative Weights Update Method

by Elisa Celis, Peter Krafft and Nisheeth Vishnoi

Understanding when social interactions lead to the emergence of group-level abilities beyond those of an individual is central to understanding human collaboration and collective intelligence. Towards this, in this post we consider a social behavior that is at once conspicuous in daily life, oft discussed in the social science literature, and also recently empirically verified: the behavior essentially boils down to taking suggestions from other people, and we study whether this leads to good decision-making as a whole.

For instance, consider the problem of choosing what stocks to purchase, or which restaurant to go to. From day to day the attractiveness of each option available might change, but overall may tend to vary around some mean. In such cases, there can be a large number of options for an individual to choose from, and it is impossible to hear about all the past experiences that individuals have had with each option. One simple strategy in these situations is to first seek out a recommendation, whether from friends or via web searches, and then evaluate the current information available about the recommended option.  In doing so, no individual has to consider all the different choices themselves, so the process is cognitively simple.

This social behavior can be broken down as follows:

  • people look to others for advice about what decisions to make,
  • privately gather further information about the recommended options,
  • and then make their final decisions.

We study a concrete model of this process called “social sampling” that was  recently formulated and validated empirically using a large behavioral dataset. More precisely, as depicted in the figure below, in every decision-making round each individual first chooses an option to consider by looking at the current decision of a random other person, then the current (stochastic) quality of each option is observed, and finally the individual randomly either keeps this option or remains undecided where the probability of keeping it is determined by the quality.

social-sampling

From the perspective of a single individual, social sampling is a simple heuristic, and it requires very limited cognitive overhead. As a whole, it is a priori unclear whether this process will result in society eventually converging to the best option, or in inferior options gaining popularity by being propagated from one person to the next.

Nevertheless, our results show that a group of individuals implementing social sampling in a diverse range of settings results in approximately optimal decisions, and hence the behavior is collectively rational. Our analysis shows that social sampling is a highly effective distributed algorithm for solving the problem of which decision is best to make, in the sense that social sampling achieves near-optimal regret in this sequential decision-making task. This behavioral mechanism that individuals use may therefore be highly effective in large groups.

Key to our results is the observation that social sampling can be viewed as a distributed implementation of the ubiquitous multiplicative weights update (MWU) method in which the popularity of each option implicitly represents the weight of that option. This relationship provides an algorithmic lens through which we can understand the emergent collective behavior of social sampling.  Beyond these scientific implications, the relationship to MWU could also suggest novel distributed MWU algorithms. Social sampling requires little communication and memory, and hence may be appropriate as a MWU algorithm for low-power devices such as sensor networks or the internet-of-things.

Social Sampling was one of the group projects pursued at the CMO-BIRS 2016 WORKSHOP ON MODELS AND ALGORITHMS FOR CROWDS AND NETWORKS.