CrowdCamp Report: Protecting Humans – Worker-Owned Cooperative Models for Training AI Systems

Artificial intelligence is widely expected to reduce the need for human labor in a variety of sectors [2]. Workers on virtual labor marketplaces unknowingly accelerate this process by generating training data for artificial intelligence systems, putting themselves out of a job.

Models such as Universal Basic Income [4] have been proposed to deal with the potential fallout of job loss due to AI. We propose a new model where workers earn ownership of the AI systems they help to train, allowing them to draw a long-term royalty from a tool that replaces their labor [3]. We discuss four central questions:

  1. How should we design the ownership relationship between workers and the AI system?
  2. How can teams of workers find and market AI systems worth building?
  3. How can workers fairly divide earnings from a model trained by multiple people?
  4. Do workers want to invest in AI systems they train?

Crowd Workers gain ownership shares in the AI they help train and reap long-term monetary gains, while Requesters can avail of lower initial training costs.

AI Systems Co-owned by Workers and Requesters

  • Current model (requester-owned): Under the terms of platforms like Amazon Mechanical Turk [1], the data produced (and trained AI systems that result) are owned entirely by requesters in exchange for a fixed price paid to workers for producing that data.
  • Proposed model (worker-owned): In a cooperative model for training AI systems, workers can choose to accept a fraction of that price in exchange for shares of ownership in the resulting trained system (smaller fractions = increased ownership). We can imagine interested outside investors (or even workers themselves) participating in such co-ops as well, bankrolling particular projects that have a significant chance of success.

Finding and Marketing AI Systems

  • Bounties vs. marketplaces: Platforms like Kaggle and Algorithmia allow interested parties to post a bounty (reward) for a trained AI system. Risks under this model include (1) the poster may not accept their solution, (2) the poster may choose another submission over their solution, or (3) the open call may expire. Alternately, Algorithmia also provides a marketplace enabling AI systems to earn money on a per-use basis. Risks here include identifying valuable problem domains with high earning potential.
  • Online vs. offline training models: In an online payment model, workers can provide answers initially and as the AI gains confidence in its predictions, work starts shifting from the crowd to the AI. In an offline payment model, the model can be marketed once it achieves sufficiently accurate predictions, or workers could market a dataset rather than a fully-trained AI system.

Fairly Dividing Earnings from AI Systems

  • Assigning credit: How to optimally assign credit for individual training examples is an open theoretical question. We see the opportunity for both model-specific and black-box solutions.
  • Measuring improvement: Measuring improvement to worker owned and trained AI systems will require methods that incentivize workers to provide the most useful examples, not simply ones that they may have gathered for a test set.
  • Example selection: Training examples could be selected by the AI system (active learning) or by workers. What are fair payment schemes for various kinds of mixed-initiative systems?
  • Data maintenance: Data may become stale over time, or change usefulness. Should workers be responsible for maintaining data, and what are fair financial incentives?

 

Do Workers Want to Invest in AI Systems?

We launched a survey on Mechanical Turk (MTurk) to gauge interest, and got feedback from 31 workers.

  • On average, workers were willing to give up 25% of their income if given the chance to double it over one year. Only 3 participants said they’d not be willing to give up any of their earnings, and age doesn’t seem to be a factor here.
  • When given a risk factor, over 48% chose to give up some current payment for a future reward.
    hit-1dollar
  • In order to give up 100% of their current earnings, workers needed to be able to make back 3 times their invested amount.
  • 45% of workers reported not being worried at all about AI taking over their jobs.

ai-worried

References

[1] Amazon Mechanical Turk. 2014. Participation Agreement. Retrieved November 4, 2016 from https://www.mturk.com/mturk/conditionsofuse.

[2] Executive Office of the President National Science and Technology Council Committee on Technology. October 2016. Preparing for the Future of Artificial Intelligence.

[3] Anand Sriraman, Jonathan Bragg, Anand Kulkarni. 2016. Worker-Owned Cooperative Models for Training Artificial Intelligence. Under review.

[4] Wikipedia. Basic Income. https://en.m.wikipedia.org/wiki/Basic_income

Anand Sriraman, TCS Research – TRDDC, Pune, India
Jonathan Bragg, University of Washington, USA
Anand Kulkarni, University of California, Berkeley, USA