CrowdCamp 2016: Understanding the Human in the Loop

Report on CrowdCamp 2016: The 7th Workshop on Rapidly Iterating Crowd Ideas, held in conjunction with AAAI HCOMP 2016. Held November 3, 2016 in Austin, TX.

Organizers: Markus Krause (UC Berkeley), Praveen Paritosh (Google), and Adam Tauman Kalai (Microsoft Research)giphy

Human computation and crowdsourcing as a field investigates aspects of the human in the loop. Consequently, we should use metaphors of computer science to describe human phenomena. These phenomena have been studied by other fields such as sociology and psychology for a very long time. Ignoring these fields not only blocks our access to valuable information but also results in simplified models we try to satisfy with artificial intelligence.

We focused this Crowdcamp on methodologically recognizing the human in the loop, by paying more attention to human factors in task design, and borrowing methodologies from scientific fields relying on human instruments, such as survey design, psychology, and sociology.

We believe that this is necessary for and will foster: 1) raising the bar for AI research, by facilitating more natural human datasets that capture the human intelligence phenomena more richly, 2) raising the bar for human computation methodology for collecting data using/via human instruments, and 3) improve the quality of life and unleashing the potential of crowdworkers by taking into consideration human cognitive, behavioral, and social factors.

This year’s Crowdcamp featured some new concepts. Beside of having a theme we also hold a pre workshop social event. The idea of the event was to get together and discuss ideas in an informal and cheerful setting. We found this very helpful to break the ice, form groups, and prepare ideas for the camp. It helped to keep us focused on the tasks without sacrificing social interactions.

We think the pre workshop social event really helped inspiring participants to get to work right away the next day. We are aware of at least one submitted work in progress paper 24 hours after the workshop! We are sure there are even more great results in the individual group reports published on this blog.

We expect to publish all of the data sets we collected in the next week or so, so please check back in a few days to see more of the results of our workshop. A forthcoming issue of AAAI magazine will include an extended version of this report. If you have feedback on the theme of this year’s CrowdCamp, you might find some further points in there to ruminate about. Feel free to share feedback directly or by commenting on this blog post.

Thanks to the many awesome teams that participated in this year’s CrowdCamp, and stay tuned as blog posts from each team describing their particular project will immediately follow this workshop overview post in the coming days.

A Workshop Connecting Crowdsourcing and Online Education at HComp 2014

The online education and crowdsourcing communities are addressing similar problems in educating, motivating and evaluating students and workers. The online learning community succeeds in increasing the supply side of the cognitively skilled labor market, and the crowdsourcing at scale community creates a larger marketplace for cognitively skilled work. WorkLearn is held at HComp 2014 in Pittsburgh, November 2, 2014.


Call for Proposals

WorkLearn 2014 is a full-day workshop at HCOMP 2014 which will bring together researchers and practitioners from crowdsourcing and online education communities to explore connections between learning and working online. We want to spark knowledge sharing and discussions on topics such as: integrating online learning platforms and online work platforms; solving shared problems like training and evaluation of both students and high-skill crowd workers; how crowdsourcing methodologies can be used to scale the labor-intensive components of education. We invite submission of short (1-2 page) position papers which identify and motivate key problems related to the intersection of issues between crowd work/human computation and online learning/education. You are invited to include a short bio in your submission to provide context for your fit with the workshop. Please send your submission to Submissions invited to participate in the workshop will notified in September. We encourage submission of position papers focusing on:

  • Challenges and demands of industry
    • What skills do we need to train (crowd and online) workers for?
    • What can crowdsourcing do for learning at scale?
  • Proposals for platforms and software to connect online work and learning
    • How can a platform for online learning be linked to a platform for crowd work in a way that creates a more skilled workforce and better crowd work?
    • Visionary papers on the future of online work and learning

We are looking forward seeing you in Pittsburgh: Markus Krause, Leibniz University, Germany Praveen Paritosh, Google, USA Joseph Jay Williams, Stanford University, USA

Call for Papers: Disco 2013 Human Computation Games Workshop At HComp 2013

Extending and building upon the focus of past workshops on games and human computation, the workshop Disco aims at exploring the intersection of entertainment, learning and human computation. Disco is held at HComp2013 in Palm Springs, November 9, 2013.



Both long (6 pages) and short/position papers (2 pages) can be submitted
In AAAI format:
Via easychair:

Important Dates:

  • Paper submission

    September 20, 2013

  • Notification of acceptance

    October 10, 2013

  • Camera ready submission (tentative)

    October 15, 2013

  • Workshop

    November 9, 2013

With the Internet being used worldwide, the way we think about communication, computation, artificial intelligence and research is changing. Human computation has emerged as a powerful approach to solving problems that would not be tractable without humans in the loop. Within human computation, games called games with a purpose or serious games are a successful approach to incite people to collaborate in human computation. Games are also for human means to learn.

Digital games are interaction machines and, how implicit it might be, always contain a learning component. Whether one is stacking blocks, exploring dungeons, or building cities games provide a variety of human machine interactions that range from simple puzzles to complex problem spaces. The challenges that emerge through these mechanics are precisely what foster human learning during the course of a game. Recently, human computation systems have tried to leverage the insights people have for solving these problems by observing, and automatically learning from, the interactions between players and choices made by players.

The workshop Disco is devoted to exploring the relationships between entertainment, learning, and human computation. The workshop has several goals. First, the workshop will investigate games as powerful incentives for human both to learn and to engage in human computation. Second, the workshop will pay attention at how learning can be seamlessly integrated into human computation tasks so as to improve both a player’s experience and a human computation system. Third, the workshop will explore how learning relates to entertainment and games. Finally, to close the loop, the workshop will investigate how human computation can improve the content, design and playability of games.

We are looking forward seeing you in Palm Springs:
François Bry, (Ludwig-Maximilians-University)
Markus Krause, (Leibniz University)


GCI 2012: Call for Paper

Harnessing Collective Intelligence with Games is the 1st international Workshop on Systems with Homo Ludens in the Loop. We invite scientists and practitioners to submit their papers on the topics of gamification and games for collective intelligence. The paper deadline is April 13th.

Continue reading