CrowdCamp Report: DesignOracle: Exploring a Creative Design Space with a Crowd of Non-Experts

For an unconstrained design problem like writing a fiction story or designing a poster, can a crowd of non-experts be shepherded to generate a diverse collection of high quality ideas? Can the crowd also generate the actual stories or posters? With the goal of building crowd-powered creativity support tools, a group of us, Paul André, Robin N. Brewer, Krzysztof Gajos, Yotam Gingold, Kurt Luther, Pao Siangliulue, and Kathleen Tuite, set out to answer these questions. We based our approach on a technique described by Keith Johnstone in his book Impro: Improvisation and the Theatre for extracting creativity from people who believe themselves to be uncreative. An uncreative person is told that a story is already prepared and he or she has merely to guess it via yes or no questions. Unbeknown to the guesser, there is no story; guesses are met with a yes or no response, essentially randomly. For example:

  1. Is it about CSCW? Yes
  2. Is it about CrowdCamp? No
  3. Is it about a bitter rivalry? Yes

As questioning proceeds, a consistent story is revealed, entirely due to the guesser generating and then externalizing an internally consistent mental model of a story (or poster, etc.) that justifies the given answers.

To evaluate the potential of this “StoryOracle” approach, we ran a series of experiments on Amazon Mechanical Turk:

  1. We extracted dozens of surprising and creative stories and poster designs using the technique.
  2. We explored the design space in a directed manner by generating variations on well-liked stories. Every question and yes or no answer provides a possible branch-point. To branch, we selected an important “pivot” question and showed all questions and answers up to the pivot, followed by the pivot question and the same or the opposite answer, to a set of new participants with instructions to continue guessing the story. For example: A story and branches
  3. We converted question-based stories into “normal” stories. For example:

    Jack is a scientist who has dedicated his life to discovering how to generate energy using fusion power because he believes it would benefit humanity. Eventually he discovers how to do it and tells his wife, Jane, the good news. Jane is not aware that this is a secret and talk about it with Carl (to whom she was in love before meeting Jack). Carl tries to steal the secret from Jack but Jane fought with him and is able to impede him.
    However, Jack discovers that Jane told Carl about the secret and they have a fight because of it. Eventually, Carl, who is angry with both of them, decides to kill Jack and Jane. He manages to kill the couple but when he is about to steal Jack’s secret, the power fusion discovery is released to the public through the internet.
    Thus, all the world is able to produce energy using Jack’s discovery and eventually his dream of providing a better quality of life to everyone comes true.

  4. We evaluated stories’ quality.
  5. We devised domain-specific prompts for questioners, such as the setting, theme, and characters in a story.

Taken together, in the two days of CrowdCamp we managed to build the foundation for a crowd-powered tool to explore a creative design space, in an undirected or directed manner, and generate a variety of high quality artifacts.

Paul André, Carnegie Mellon University
Robin N. Brewer, University of Maryland, Baltimore County
Krzysztof Gajos, Harvard University
Yotam Gingold, George Mason University
Kurt Luther, Carnegie Mellon University
Pao Siangliulue, Harvard University
Kathleen Tuite, University of Washington

Diamonds From the Rough: Improving Drawing, Painting, and Singing via Crowdsourcing

Yotam Gingold, George Mason University
Etienne Vouga, Columbia University
Eitan Grinspun, Columbia University
Haym Hirsh, Rutgers University

Does the combination of many drawings, paintings, or singings produce a “more beautiful” output than any of the inputs? Many physical actions—we focus on artistic ones—are inherently limited in quality by the skill, time, resources, and motor disabilities (such as Parkinson’s) of the humans performing them. In this work, we show that we can produce “diamonds” from rough input. We consider many different people each contributing one input via crowdsourcing, as well as a single person contributing many inputs (the crowd within).

Continue reading

Micro Perceptual Human Computation for Visual Tasks

Yotam Gingold, Columbia University & Rutgers University
Ariel Shamir, The Interdisciplinary Center, Herzliya
Daniel Cohen-Or, Tel-Aviv University

Many graphics and vision problems are very simple for humans and difficult for computers. For example, it is simple for people to recognize, organize, segment, align, or correspond images and 3D objects. In contrast, the amount of research in the field towards solving these problems indicates how difficult they are for pure machine computation.

We advocate a tight integration of human computation into online, interactive graphics and vision algorithms. Our key idea is to decompose a problem into a massive number of very simple, carefully designed, human micro-tasks that are based on perception, and whose answers can be combined algorithmically to solve the original problem. We present three specific examples for the design of Micro Perceptual Human Computation algorithms to extract depth layers and image normals from a single photograph, and to detect bilateral symmetry of objects in an image.

A user sits at his computer and uses an application which is written in code that runs on both electronic and human processors.

Continue reading