Visual Challenges in the Everyday Lives of Blind People

Blind people often need to access visual information, but don’t have a sighted companion available to answer questions.  While automated tools like OCR and object recognition can make some visual information accessible, other complex questions still require a human’s input.

We created VizWiz Social to connect blind users’ visual questions with sighted workers or friends who can answer them.  VizWiz Social is a mobile phone application that allows the user to take a photograph of something, record a question about it, and send it to anonymous crowd workers or to their social network for answers.  Our paper discusses what we learned from the 40,748 questions asked by 5,329 blind users in the first year of the app’s availability.

The VizWiz Social user interface allows a blind user to (a) take a photograph of something they have a question about, (b) record their question, (c) choose where to send their question, and (d) wait for answers to arrive. The user can access the application through the VoiceOver screen reader.

We created a taxonomy of the types of questions asked to find out what visual information was most desired by our blind users.  We also analyzed the features of the photographs and questions send in by user, and user behaviors over time.  Some interesting results include:

  • The most popular type of questions were Identification questions (41%), where users asked for simple identifications of objects (such as food or drink items).  However, there were also a large number of Description questions (24%) and Reading questions (17%), which required more complex analysis and human insight to answer.
  • Most questions were perceived by our raters to be objective and time-sensitive, perhaps due to the fast response times available from object recognition or pre-recruited crowd workers.
  • Users who had a poor first experience with our application – either because their photographs were low-quality or or because the answers they received were unsatisfactory – had a higher-than-typical abandonment rate.

These results improve our understanding of the problems blind people face, and may help motivate new projects more accurately targeted to help blind people live more independently in their everyday lives.

For more, see our full paper, Visual Challenges in the Everyday Lives of Blind People.

Erin Brady, ROCHCI @ University of Rochester
Meredith Ringel Morris, Microsoft Research
Yu Zhong, ROCHCI @ University of Rochester
Samuel White, ROCHCI @ University of Rochester
Jeffrey Bigham, ROCHCI @ University of Rochester


About the author

Erin Brady

Graduate student in the ROCHCI lab at the University of Rochester, focused on using the combination of crowdsourcing and friendsourcing to help further accessibility.

View all posts


  • Really exciting work. I love the brief anecdote in the paper about a sudden increase of hotel shampoo bottle identification questions. Apparently the service started a bit of a meme at a conference as new people tried out the tool for an interesting task. This is the sort of tantalizing evidence that a service is really being integrated in a person’s life that is not only super exciting, but also bodes well for future crowdsourced human services.

    I wonder how we might use domain knowledge or expertise to get at some of the tougher questions. Maybe we want to look at friend or Internet social networks to find the right fashion experts to answer the “do these clothes match” questions. How do we find the experts, make sure they are willing to answer, and identify which questions need expertise to complete?

    • Thanks for reading, Jeffrey!

      Matching the right workers to questions is something we’ve definitely been trying to improve. For more subjective questions (like the fashion ones you mentioned), you not only want to find people with some level of clothing expertise, but you also have to try to reconcile different types of styles – for example an older, professional woman might give different recommendations than a college-aged sorority girl! If you’re interested, you can learn more about how trust and other factors impact question askers in a paper I worked on with Michele Williams from UMBC:

      In terms of identifying which questions need expertise – definitely an open question. For the time being, we’ve left all decisions about who to choose as answerers to the user themselves, but it might be interesting to provide ‘suggested’ sources based on the type of question asked, or the contents of the photograph.

      Thanks again for your comment! Really interesting stuff to consider.