Mere minutes after our awesome advisor,Dan Weld, mentioned The 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP), we were all-in. It’s rare to scroll through an event program and realize that each and every session is going to be so relevant and useful to your work, but that’s exactly how we were all feeling with this event’s agenda. And it did not disappoint!
Our decision to sponsor HCOMP required very little human computation, and we were thrilled to give a keynote talk on our tips for sourcing better training data. We’ve created an online version of our presentation deck for your reference; hope it’s helpful.
If you’d like to learn more about these ideas or have something to add, please give us a shout. We’re also particularly interested in the topic of bias in training data, so if this is a concern of yours as well, get in touch and let’s study it together (we’ll bring the data!).
Finally, as we noted in our talk, we’re hiring! We’re growing our data science team and looking for computer vision experts specifically. Check out our openings if you’re looking for your next great opportunity.
A big thanks to everyone at HCOMP. We had a great time and look forward to continuing the many discussions we started there.
Correlation does not imply causation. This phrase gets thrown around by scientists, statisticians, and laypeople all the time. It means that you shouldn’t use data about two things to infer that one thing causes the other, at least not without making a lot of limiting assumptions. But it is difficult to imagine ignoring causal inference when it seems to be such a key ingredient of intelligent decision-making. Machine learning approaches exist for using data to estimate causal structure, but we think it’s interesting that humans seem to judge causality without even looking at data. So, the goal of our CrowdCamp project was to gather some such judgements from real people.
To start, we make a list of variable names for which we hypothesize humans might have opinions about causal relationships without ever (or at least recently) having looked at the related data. Some of these variables include:
Real Daily Wages, Oil Prices, Internet Traffic, Residential Gas Usage, Power Consumption, Precipitation, Water Usage, Traffic Fatalities, Passenger Miles Flown in Aircraft, Auto Registration, Bus Ridership, Copper Prices, Wheat Harvest, Private Housing Units Started, Power Plant Expenditures, Price of Chicken, Sales of Shampoo, Beer Shipments, Percent of Men with Full Beards,
Pigs Slaughtered, Cases of Measles, Thickness of Ozone Layer, etc.
Using Amazon Mechanical Turk (AMT), we presented workers with sets of ten randomly chosen pairs of variables, and we asked them to choose the most fitting causal relationship between variable A and variable B between these four choices:
A causes B
B causes A
Other variable Z causes A and B
No causal relationship
Workers were advised “it’s possible that A and B may be related in several of the above ways. If you feel this is the case, choose the one that you believe is the strongest relationship.”
We collected 10 judgements from 50 workers, for a total of 500 judgements on pairs of 42 variables. When workers chose the option of a third variable causing both presented variables, we asked them to name the third variable (though we didn’t force them to). Of the 500 judgements, 74 of them were A->B, 85 were B->A, 34 were Z->A&B, and 307 were no causality. The most common one-directional causality judgements were:
1. Church Attendance -> Internet Traffic
2. Alcohol Demand -> Public Drunkenness
3. Federal Reserve Interest Rate -> Price of Chicken
4. Bus Ridership -> Oil Prices
5. Alcohol Demand -> Number of Forest Fires
6. Public Drunkenness -> Armed Robberies
7. Power Consumption -> Birth Rate
8. Church Attendance -> Armed Robberies
9. Bus Ridership -> Birth Rate
10. Price of Chicken -> Total Rainfall
Many of these are not surprising. Of course interest rates affect prices and alcohol consumption affects drunkenness. Others not so much… why would chicken prices affect rainfall? Also, we realize we only asked about the strength of the causal relationship, not the sign. So we have no way of knowing whether the workers believe going to church causes an increase or a decrease in armed robberies.
We also collected some interesting answers for the optional third variable Z causing both A and B. Most of the time it was some big general factor like population, economic conditions, geographical area, or fuel prices. There were some creative ones too:
A: Deaths from Homicides
B: Beer Shipments
Z: Thieves trying to intercept and steal beer shipments
So we collected all these judgements, now what do we do with them? As for machine learning applications, we see three options:
Use as training/testing labels for causal inference techniques.
See how well they serve for building informative priors to regularize regression problems.
Use them to guide structure learning in probabilistic graphical models.
In conclusion, it was interesting to see how workers on AMT perceived causal relationships between economic, demographic, and miscellaneous variables by only looking at the names of the variables rather than actual data. We think it would be useful to take such qualitative “common-sense” preconceptions into account when designing automatic models of inference.
Alex Braylan, University of Texas at Austin Kanika Kalra, Tata Research Tyler McDonnell, University of Texas at Austin
Semantic similarity or semantic relatedness are features of natural language that contribute to the challenge machines face when analyzing text. Although semantic relatedness is still a complex challenge only few ground truth data set exist. We argue that the available corpora used to evaluate the performance of natural language tools do not capture all elements of the phenomenon. We present a set of simple interventions that illustrate 1) framing effects influence similarity perception, 2) the distribution of similarity across multiple users is important and 3) semantic relatedness is asymmetric.
A number of metrics in the literature attempt to model and evaluate semantic similarity in natural languages. Semantic similarity has applications in areas such as semantic search, text mining, etc. The concept of semantic similarity has long been considered as a more specific concept than the concept of semantic relatedness. Semantic relatedness, as it includes the concepts of antonymy and meronymy, is more generic than semantic similarity.
Different approaches have been attempted to measure semantic relatedness and similarity. Some methods use structured taxonomies such as WordNet alternative approaches define relatedness between words using search engines (e.g., based on Google counts) or Wikipedia. All of these methods are evaluated based on the correlation with human ratings. Yet only few benchmark data sets exist. One of the most widely used being the WS-353 data-set . As the corpus is very small and the sample size per pair is low it is arguable if all relevant phenomena are in fact present in the provided data set.
In this study, we aim to understand how human raters perceive word-based semantic relatedness. We argue that asking simple word-based semantic similarity is beyond the scope of existing test sets. Our hypotheses in this paper are as follows:
(H1) The framing effect influences similarity rating by human assessors.
(H2) The distribution of similarity rating does not follow a normal distribution.
(H3) Semantic relatedness is not symmetric. The relatedness between words (e.g., tiger and cat) yields different similarity ratings in a different word order.
To verify our hypotheses, we collected similarity ratings on word pairs from the WS-353 data-set. We randomly selected 102 word pairs from the WS-353 data-set. We collected similarity ratings on the 102 word pairs through Amazon Mechanical Turk (MTurk). We collected 5 dataset for these 102 pairs. Each collection used a different task design and was separated into two batches of 51 words each. Each batch received ratings from 50 unique contributors so that each pair of word received 50 ratings in each condition.
The way the questions were asked to the crowd workers are shown in the following figure. For each question, 4 conditions were differently framed. The first two of these are “How is X similar to Y?” (sim) and “How is Y similar to X?” (inverted-sim). We further repeated them asking for the difference between both words (dissim and inverted-dissim, respectively). Since the scale is reversed in dissim and inverted-dissim, the dissimilarity ratings were converted into similarity ratings for comparison.
We compared the distributions of similarity ratings in the original WS-353 dataset and our dataset in order to confirm the framing effect. The mean values of 50 ratings were calculated for each pair in our dataset to compare with original similarity ratings in the WS-353 dataset. We filtered exactly the same 102 word pairs from the WS-353 to ensure the consistency between two settings. The distributions are found to be significantly different (p < 0.001, paired t-test).
Our preliminary results show that similarity ratings for some word pairs in the WS-353 dataset do not follow a normal distribution. Some of the distributions reveal that there are different perceptions of similarity, which gets highlighted by multiple peaks. A possible explanation is that the lower peak can be attributed to individuals that are aware of the factual differences between a “sun” or “star” and an actual planet orbiting a “star” while the others are not aware of it.
We compared the difference between the similarity ratings of sim (dssim) and that of inverted-sim (inverted-dissim) to verify third hypothesis. Scatter plot representations of similarity ratings in different word orders for the similarity question and the dissimilarity question reflect that the semantic relatedness in different orders do not take same mean values, indicating the semantic relatedness is asymmetric. The asymmetric relationship consistently appears in the different types of questions (i.e., similarity and dissimilarity.) The results show a remarkable difference between the similarity of “baby” to “mother” and the similarity of “mother” to “baby”. This indicates that the asymmetric relationship between mother and baby was reflected in the subjective similarity rating.
To measure the inter-rater reliability, we have computed the value of Krippendorff’s alpha for both the original dataset and for the one we obtained through the current analysis. Krippendorff’s alpha is a statistical measure that basically provides a highlight of the agreement achieved when encoding a set of units of analysis in terms of the values of a variable.
 L. Finkelstein, E. Gabrilovich, Y. Matias, E. Rivlin, Z. Solan, G. Wolfman, and E. Ruppin. Placing search in context: The concept revisited. ACM Transactions on Information Systems, 20(1), 2002.
Artificial intelligence is widely expected to reduce the need for human labor in a variety of sectors . Workers on virtual labor marketplaces unknowingly accelerate this process by generating training data for artificial intelligence systems, putting themselves out of a job.
Models such as Universal Basic Income  have been proposed to deal with the potential fallout of job loss due to AI. We propose a new model where workers earn ownership of the AI systems they help to train, allowing them to draw a long-term royalty from a tool that replaces their labor . We discuss four central questions:
How should we design the ownership relationship between workers and the AI system?
How can teams of workers find and market AI systems worth building?
How can workers fairly divide earnings from a model trained by multiple people?
Do workers want to invest in AI systems they train?
AI Systems Co-owned by Workers and Requesters
Current model (requester-owned): Under the terms of platforms like Amazon Mechanical Turk , the data produced (and trained AI systems that result) are owned entirely by requesters in exchange for a fixed price paid to workers for producing that data.
Proposed model (worker-owned): In a cooperative model for training AI systems, workers can choose to accept a fraction of that price in exchange for shares of ownership in the resulting trained system (smaller fractions = increased ownership). We can imagine interested outside investors (or even workers themselves) participating in such co-ops as well, bankrolling particular projects that have a significant chance of success.
Finding and Marketing AI Systems
Bounties vs. marketplaces: Platforms like Kaggle and Algorithmia allow interested parties to post a bounty (reward) for a trained AI system. Risks under this model include (1) the poster may not accept their solution, (2) the poster may choose another submission over their solution, or (3) the open call may expire. Alternately, Algorithmia also provides a marketplace enabling AI systems to earn money on a per-use basis. Risks here include identifying valuable problem domains with high earning potential.
Online vs. offline training models: In an online payment model, workers can provide answers initially and as the AI gains confidence in its predictions, work starts shifting from the crowd to the AI. In an offline payment model, the model can be marketed once it achieves sufficiently accurate predictions, or workers could market a dataset rather than a fully-trained AI system.
Fairly Dividing Earnings from AI Systems
Assigning credit: How to optimally assign credit for individual training examples is an open theoretical question. We see the opportunity for both model-specific and black-box solutions.
Measuring improvement: Measuring improvement to worker owned and trained AI systems will require methods that incentivize workers to provide the most useful examples, not simply ones that they may have gathered for a test set.
Example selection: Training examples could be selected by the AI system (active learning) or by workers. What are fair payment schemes for various kinds of mixed-initiative systems?
Data maintenance: Data may become stale over time, or change usefulness. Should workers be responsible for maintaining data, and what are fair financial incentives?
Do Workers Want to Invest in AI Systems?
We launched a survey on Mechanical Turk (MTurk) to gauge interest, and got feedback from 31 workers.
On average, workers were willing to give up 25% of their income if given the chance to double it over one year. Only 3 participants said they’d not be willing to give up any of their earnings, and age doesn’t seem to be a factor here.
When given a risk factor, over 48% chose to give up some current payment for a future reward.
In order to give up 100% of their current earnings, workers needed to be able to make back 3 times their invested amount.
45% of workers reported not being worried at all about AI taking over their jobs.
Organizers: Markus Krause (UC Berkeley), Praveen Paritosh (Google), and Adam Tauman Kalai (Microsoft Research)
Human computation and crowdsourcing as a field investigates aspects of the human in the loop. Consequently, we should use metaphors of computer science to describe human phenomena. These phenomena have been studied by other fields such as sociology and psychology for a very long time. Ignoring these fields not only blocks our access to valuable information but also results in simplified models we try to satisfy with artificial intelligence.
We focused this Crowdcamp on methodologically recognizing the human in the loop, by paying more attention to human factors in task design, and borrowing methodologies from scientific fields relying on human instruments, such as survey design, psychology, and sociology.
We believe that this is necessary for and will foster: 1) raising the bar for AI research, by facilitating more natural human datasets that capture the human intelligence phenomena more richly, 2) raising the bar for human computation methodology for collecting data using/via human instruments, and 3) improve the quality of life and unleashing the potential of crowdworkers by taking into consideration human cognitive, behavioral, and social factors.
This year’s Crowdcamp featured some new concepts. Beside of having a theme we also hold a pre workshop social event. The idea of the event was to get together and discuss ideas in an informal and cheerful setting. We found this very helpful to break the ice, form groups, and prepare ideas for the camp. It helped to keep us focused on the tasks without sacrificing social interactions.
We think the pre workshop social event really helped inspiring participants to get to work right away the next day. We are aware of at least one submitted work in progress paper 24 hours after the workshop! We are sure there are even more great results in the individual group reports published on this blog.
We expect to publish all of the data sets we collected in the next week or so, so please check back in a few days to see more of the results of our workshop. A forthcoming issue of AAAI magazine will include an extended version of this report. If you have feedback on the theme of this year’s CrowdCamp, you might find some further points in there to ruminate about. Feel free to share feedback directly or by commenting on this blog post.
Thanks to the many awesome teams that participated in this year’s CrowdCamp, and stay tuned as blog posts from each team describing their particular project will immediately follow this workshop overview post in the coming days.
The HCOMP Workshop on Mathematical Foundations of Human Computation was held at HCOMP 2016 in Austin, Texas on November 3, 2016. The goal of the workshop was to bring together researchers across disciplines to discuss the future of research on the mathematical foundations of human computation, with a particular emphasis on the ways in which theorists can learn from the existing empirical literature on human computation and the ways in which applied and empirical work on human computation can benefit from mathematical foundations.
There has been great progress in human computation in the past decade. However, human computation is still not well understood from a foundational mathematical perspective. Mathematical foundations are important to influence and shape the future of human computation. They provide frameworks for formalizing desirable properties of systems (e.g., correctness, optimality, scalability, privacy, fairness), predicting the impact of design decisions, designing systems with provable guarantees, and performing counterfactual analysis, which is notoriously hard to do empirically.
The workshop concluded with an open problem session in which participants delivered short informal pitches of open problems related to the workshop theme. The main themes of the open problems that were proposed include:
The characterization of the important properties of computational models of human cognition. For example, how do we classify or quantify the hardness of problems in human computation?
Evaluation metrics for human computation problems. What properties do we want human computation to achieve?
The design of models that accurately incorporate human behavior. At a minimum, this requires diving into the literature from other disciplines, such as psychology and behavioral economics. Ideally it should be explored in an interdisciplinary manner.
How to develop theory and design algorithms that are robust to errors in modeling and experiments.
When collecting human ratings or labels for items, it can be difficult to measure and ensure data quality control, due to both task subjectivity (i.e., lack of a gold standard against which answers can be easily checked), as well as lack of transparency into how individual judges arrive at their submitted ratings. Using a paid microtask platform like Mechanical Turk to collect data introduces further challenges: crowd annotators may be inexpert, minimally trained, and effectively anonymous, and only rudimentary channels may be provided to interact with workers and observe work as it is performed.
To address this challenge, we investigate a very simple idea: requiring judges to provide a rationale supporting each rating decision they make. We explore this idea for the specific task of collecting relevance judgments for search engine evaluation. A relevance judgment rates how relevant a given web page is to a particular search query. We define a rationale as a direct excerpt from a web page (e.g., which a worker might easily provide by copy-and-paste or highlighting).
To illustrate this idea, we show a simple example below in which a user is assumed to be searching for information about symptoms of jaundice, and a worker is asked to rate the relevance of the Wikipedia page on jaundice. The worker might rate the page as relevant and support his/her decision by highlighting an excerpt describing jaundice symptoms, as shown.
What are the symptoms of jaundice?
While this idea of rationales is quite simple and easy to implement — good things for translating ideas from research to practice! — rationales appear to be remarkably useful, offering a myriad of benefits:
Rationales enhance transparency and value. While traditional labor models involve high-quality interpersonal interactions and transparency, this comes at significant expense. On the other hand, microtasking typically greatly improves efficiency and costs, but at the cost of low transparency and poor communication channels. Rationales represent a middle-ground between these extremes, with textual excerpts comprising a very light-weight form of communication from worker to requester to enhance transparency. Moreover, because rationales enhance data interpretability, they increase the value of collected data. This makes rationales useful to collect from traditional annotators as well as crowd annotators, and beyond ensuring initial data quality, rationales increase the enduring value of collected data for all future users as well.
Rationales enhance reliability and accountability. Identifying a focused context for explaining annotator decisions as they are made is generally helpful and especially so for subjective tasks without clear answers. By making work more verifiable, arbitrary answers can no longer be given without justification, making annotators more accountable for their work and decreasing any temptation to rush or cheat. In addition, rationales let us establish alternative truths, e.g., in cases of disagreement between raters, or when rating decisions seem surprising or unintuitive without explanation. Moreover, just as rationales allow requesters to understand worker thought processes, they also enable application of sequential task design in which workers refine or refute one another’s reasoning. Such sequential design is critical for minimizing reliance on “experts” to verify data quality when scaling up data collection.
Rationales enhance inclusivity. We hypothesized that the improved transparency and accountability of rationales would enable us to remove all traditional barriers to participation without compromising data quality. By allowing anyone interested to work, we support greater scalability of data collection, greater diversity of responses, and equal opportunity access to income. Our experiments on Mechanical Turk imposed no worker qualifications to restrict anyone from performing our tasks, and we were able to successfully collect high quality data without imposing any worker restrictions.
It gets even better. Despite the seemingly extra work required to collect rationales in addition to relevance judgments, in practice rationales do not require extra time (and therefore cost) to collect! Over a series of experiments collecting nearly 10,000 relevance judgments, we found that workers produce higher quality relevance judgments simply by being asked to provide a rationale, and prolific workers require virtually no extra time to provide rationales in addition to ratings. Intuitively, rationales appear to merely capture explicitly the implicit cognitive work inherent in providing ratings: workers already have to find relevant text to perform the rating task, and asking them to simply report that text requires essentially no extra work.
Finally, it’s intuitive that workers who select similar rationales are likely to produce similar relevance judgments, and we can exploit such overlap between worker rationales in aggregating worker judgments in order to further improve data quality.
In sum, our results suggest that rationales may be a remarkably simple yet powerful tool for crowdsourced data collection, particularly for difficult or subjective tasks. There’s much more to our study, so we encourage you to read our full paper! The link to our paper is below, along with links to the presentation slides and our collected data.
Since the 1970s, we’ve known humans can find approximate solutions to NP-Complete (really hard) problems more efficiently than the best algorithms (Krolak 1971). The best algorithms scale quadratically, while humans scale linearly (Dry 2006). We also know many of the most widely used and successful machine learning algorithms are NP-Hard to train optimally (Diettrich 2000). This suggests a human/machine hybrid can produce better models than machine learning alone.
However, there are problems with human interaction. The hardest problem is visualization. It is hard visualizing data with more than 3 dimensions. So, we perform dimension reduction by projecting data onto two dimensions. We then collect many weak models (humans draw boxes) from multiple projections to build a strong model; known as boosting in machine learning.
Combining crowdsourcing and boosting, we use Amazon’s Mechanical Turk to collect the models. The data is transformed by the models into a feature space. Then, we use linear regression to classify new data.
We test the human/machine hybrid on one artificial dataset and four real world datasets, all with ten or more dimensions. This hybrid is competitive with machine only linear regression on the untransformed data.
By An T. Nguyen, Matthew Halpern, Byron C. Wallace, Matthew Lease
University of Texas at Austin and Northeastern University
Imagine you are a graphics designer working on a logo for a major corporation. You have a preliminary design in mind, but now you need to hammer out the fine-grained details, such as the exact font, sizing, and coloring to use. In the past you might have solicited feedback from colleagues, in-person user studies, but you recently heard the power of the “crowd” and you are intrigued. You launch an Amazon Mechanical Turk task asking workers to rate different logo permutations on a 1-5 scale, where 1 and 5 stars are the least and most satisfactory ratings, respectively.
You now have a rich multi-variate dataset for the workers’s opinions, but did all workers undertake the task in good faith? For example, a disinterested worker could have just picked a score and a more malicious worker could have intentionally picked an opposite score. You might like to detect any such cases and filter them out of your data. You might even want to extrapolate from the collected scores to other versions not scored. How could you do any of this?
While the need to accurately model data quality is key, this is a challenging partially-subjective rating scenario where each response is open but partially constrained. In the logo example, the worker is allowed to select any score in the range 1-5. Personal opinions may differ significantly on a score for a specific logo design. This is in contrast to conventional crowdsourcing tasks, such as image tagging and finding email addresses, where one expects a single, correct answer to each question being asked. While a great deal of prior work has proposed ways to model data for such objective tasks, far less work has considered modeling data quality and worker performance under more subjective task scenarios.
The key observation we make in our paper is that the worker data for these partially-subjective tasks, where worker labels are partially ordered (i.e. scores from one to five), are heteroscedastic in nature. Therefore we propose a probabilistic, heteroscedastic model where the means and variances of worker responses are modeled as functions of instance attributes. In other words, the variability of scores can itself vary across the different parameters. Consider the results as the font size of a logo is varied. We would expect that most workers would give the logos with the smallest and largest font sizes low scores. However, the range of scores for the middle range of fonts is going to be more varied.
We demonstrate the effectiveness of our model on a large dataset of nearly 25,000 Mechanical Turk ratings of user experience when viewing videos on smartphones with varying hardware capabilities. Our results show that our method is effective at both predicting user ratings and in detecting unreliable respondents, which is particularly valuable and little studied in this domain of subjective tasks where there is no clear, objective answer.
The link to our full paper is below, along with links to shared data and code, and we welcome any comments or suggestions!
An Thanh Nguyen, Matthew Halpern, Byron C. Wallace, and Matthew Lease. Probabilistic Modeling for Crowdsourcing Partially-Subjective Ratings. In Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2016. 10 pages. [ bib | pdf| data| sourcecode ]
IS THE UBIQUITOUS FIVE STAR RATING SYSTEM IMPROVABLE?
We compare three popular techniques of rating content: five star rating, pairwise comparison, and magnitude estimation.
We collected 39 000 ratings on a popular crowdsourcing platform, allowing us to release a dataset that will be useful for many related studies on user rating techniques.
The dataset is available here.
We ask each worker to rate 10 popular paintings using 3 rating methods:
Magnitude: Using any positive number (zero excluded).
Star: Choosing between 1 to 5 stars.
Pairwise: Pairwise comparisons between two images, with no ties allowed.
We run 6 different experiments (one for each combination of these three types) with 100 participants in each of them. We can thus analyze the bias given by the rating system order, and the results without order bias by using the aggregated data.
At the end of the rating activity in the task, we dynamically build the three painting rankings induced by the choices of the participant, and ask them which of the three rankings better reflects their preference (the ranking comparison is blind: There is no indication on how each ranking has been obtained, and their order is randomized).
WHAT’S THE PREFERRED TECHNIQUE?
Participants clearly prefer the ranking obtained from their pairwise comparisons. We notice a memory bias effect: The last technique used is more likely to get the most accurate description of the real user preference. Despite this, the pairwise comparison technique obtained the maximum number of preferences in all cases.
While the pairwise comparison technique clearly requires more time than the other techniques, it would be comparable in terms of time with the other techniques using a dynamic test system (of order NlogN).
WHAT DID WE LEARN?
Star rating is confirmed to be the most familiar way for users to rate content.
Magnitude is unintuitive with no added benefit.
Pairwise comparison, while requiring a higher number of low-effort user ratings, best reflects intrinsic user preferences and seems to be a promising alternative.