diagrams

Crowdsourcing the Location of Photos and Videos

How can crowdsourcing help debunk fake news and prevent the spread of misinformation? In this paper, we explore how crowds can help expert investigators verify the claims around visual evidence they encounter during their work.

A key step in image verification is geolocation, the process of identifying the precise geographic location where a photo or video was created. Geotags or other metadata can be forged or missing, so expert investigators will often try to manually locate the image using visual clues, such as road signs, business names, logos, distinctive architecture or landmarks, vehicles, and terrain and vegetation.

However, sometimes there are not enough clues to make a definitive geolocation. In these cases, the expert will often draw an aerial diagram, such as the one shown below, and then try to find a match by analyzing miles of satellite imagery.

An aerial diagram of a ground-level photo, and the corresponding satellite imagery of that location.

Source: Bellingcat

This can be a very tedious and overwhelming task – essentially finding a needle in a haystack. We proposed that crowdsourcing might help, because crowds have good visual recognition skills and can scale up, and satellite image analysis can be highly parallelized. However, novice crowds would have trouble translating the ground-level photo or video into an aerial diagram, a process that experts told us requires lots of practice.

Our approach to solving this problem was right in front of us: what if crowds also use the expert’s aerial diagram? The expert was going to make the diagram anyway, so it’s no extra work for them, but it would allow novice crowds to bridge the gap between ground-level photo and satellite imagery.

To evaluate this approach, we conducted two experiments. The first experiment looked at how the level of detail in the aerial diagram affected the crowd’s geolocation performance. We found that in only ten minutes, crowds could consistently narrow down the search area by 40-60%, while missing the correct location only 2-8% of the time, on average.

diagrams

In our second experiment, we looked at whether to show crowds the ground-level photo, the aerial diagram, or both. The results confirmed our intuition: the aerial diagram was best. When we gave crowds just the ground-level photo, they missed the correct location 22% of the time – not bad, but probably not good enough to be useful, either. On the other hand, when we gave crowds the aerial diagram, they missed the correct location only 2% of the time – a game-changer.

Bar chart showing the diagram condition performed significantly better than the ground photo condition.

For next steps, we are building a system called GroundTruth (video) that brings together experts and crowds to support image geolocation. We’re also interested in ways to synthesize our crowdsourcing results with recent advances in image geolocation from the computer vision research community.

For more, see our full paper, Supporting Image Geolocation with Diagramming and Crowdsourcing, which received the Notable Paper Award at HCOMP 2017.

Rachel Kohler, Virginia Tech
John Purviance, Virginia Tech
Kurt Luther, Virginia Tech

About the author

Kurt Luther

Kurt Luther is an assistant professor of computer science at Virginia Tech, where he directs the Crowd Intelligence Lab.

View all posts

Leave a Reply

Your email address will not be published. Required fields are marked *