AskSheet: Efficient Human Computation for Decision Making with Spreadsheets

For some decisions, we know what we want; the real “work” is in digging through the wealth of available information to find one that meets our criteria. The process can be time-consuming, especially if there are many alternatives to choose from, with the details spread among different locations.

One of the recurring challenges of adapting any complex job to a microtask platform is that crowd workers can’t see the big picture. They don’t know your situation. Furthermore, knowledge gained in one task doesn’t necessarily help a worker doing the next task. For decision making, this makes it difficult to pare down the options based on just a few of the most influential criteria.

AskSheet is a system for coordinating workers on Mechanical Turk to gather the inputs to data-driven decisions. The user (someone in charge of a decision) creates a skeleton spreadsheet model, including spreadsheet formulas that would compute the decision result if all of the inputs were already known. Cells in need of input are marked by entering a special =ASK(…) formula, the parameters to which specify the type and usually the range of information requested, as well as cues that help AskSheet group related inputs into HITs that will be efficient for workers.

This decision model finds any pediatrician who (1) has good ratings on two rating sites, (2) is within 15 minutes’ drive, and (3) accepts my insurance. Once the “root” cell (F53) can be evaluated, we know that one doctor must fit, so AskSheet stops posting HITs.

The key innovation is a method for prioritizing the inputs by leveraging the syntactic structure of the formulas. It exploits the fact that spreadsheets have no guaranteed order of evaluation for operations such as =AND(…), =OR(…), =MIN(…), and =MAX(…). When evaluating an operand entails posting a HIT, short-circuit evaluation becomes a powerful tool for reducing human effort (and cost).  Existing platforms (e.g., Smartsheet) would gather all of the inputs. The benefit of AskSheet is that it produces the same conclusive answer, while eliminating many of the inputs.

The paper details three field trials for choosing a pediatrician, smartphone, and a car. The savings, which depend entirely on the specifics of these models (described in the paper), ranged from 37% to 82% of the inputs. We priced our HITs at rates such that workers effectively earned $4.75 to $7.38 per hour. At those rates, the cost savings ranged from $4.83 and $25.83.

AskSheet supports models with several hundred inputs and a subset of spreadsheet formulas, including =AND(…), =OR(…), =MIN(…), =IF(…), =INDEX(…), =MATCH(…), and a few others. We are working on solutions to underlying computational challenges that will allow its use with larger, more expressive models. In addition, a key part of the vision is to allow delegating some inputs to paid crowd workers and direct other more sensitive or subjective inputs to trusted collaborators.

For more, see our full paper, AskSheet: Efficient Human Computation for Decision Making with Spreadsheets.

Alex Quinn, University of Maryland
Ben Bederson, University of Maryland

About the author

Alex Quinn

Alex Quinn is an assistant professor at Purdue University in the School of Electrical & Computer Engineering. His research is about human computation and crowdsourcing.

View all posts