Multilingual Social Tagging of Art Images: Cultural Bridges and Diversity

Irene Eleta, iSchool, University of Maryland
Jennifer Golbeck, Human-Computer Interaction Lab and iSchool, University of Maryland

Digital image collections are becoming available for the general public in growing numbers thanks to the digitization and dissemination efforts of cultural heritage institutions around the world. People access these collections through text (for instance, typing words), which requires images to be annotated. However, the volume of the collections poses many challenges for indexing, access and use. Crowdsourcing the annotation of images of artworks to different language communities has the potential to bridge language borders and reach wider audiences. Our study shows that multilingual social tagging is a promising approach for obtaining human translations of index terms and for enriching the meaning of images with different perspectives. Also, we suggest creative ways of using multilingual social tags in digital libraries and museums.

The UNESCO World Digital Library appproaches the problem of access to its digital objects by carefully translating the professional annotations, but their collection is relatively small and offers a limited number of languages. This approach is infeasible in the fascinating and challenging project Europeana, the portal to Europe’s cultural collections, which has over 20 million digital objects from 32 countries as of today.

Multilingual social tags could enable multilingual search (i.e. the search functionality supports more than one language) and cross-lingual search with the help of machine translation, where users are able to find images annotated in a language different from their own.

Social tagging could add the value of different cultural interpretations, which would be more accentuated when engaging different language communities and in various countries. The steve.museum project, a collaboration of several American Museums of Art, has explored the use of social tags in English for digital images of artworks. They wonder “how this new sort of engagement with museum objects might help … draw contributors who bring a multi-cultural perspective to looking at our works of art“.

steve.museum social tagging project
steve.museum social tagging project

We need to design user interfaces that support diversity and encourage sharing of perspectives for the construction of the meaning of artworks. The caveat is that the social tagging environment has to stimulate participation, and we need natural language processing for “cleaning” the tags and making sense of text in diferent languages.

The study we are presenting at CSCW 2012 uses a collection of digital images of paintings with tags in Spanish and English. We compared the social tagging patterns in both languages. First, we detected that the social tags referring to ¨persons or things¨, mostly associated with realistic paintings, facilitates the agreement in the vocabulary across languages. This type of social tags could be used as crowdsourced translations of index terms.

Secondly, social tags referring to ¨emotions and abstract ideas¨, such as melancholy, fear, or peace, showed great vocabulary divergence. When aggregating tags from two language communities for an image, we could indentify the social tags that unveil cultural differences. In these cases, machine translation of tags in one language may be used to provide additional cultural understanding for users in another language. For example, this image of the painting “The Cotton Pickers” by Winslow Homer, from Los Angeles County Museum of Art, has richer cultural references in the English set of social tags: African American, racism, southern, civil war, etc.

Tags in English and Spanish for the image of the painting The Cotton Pickers
Tags in English and Spanish for the image of the painting The Cotton Pickers, by Winslow Homer, Los Angeles County Museum of Art

These social tags could facilitate the discovery of images across languages. For instance, if a user types the keyword “racism”, the system could display on the results page all the images that are tagged with the word “racism” (like “The Cotton Pickers”) and, additionally, the images tagged in Spanish with “racismo”, showing the translation when needed. Alternatively, users could browse tag clouds in their language to access the images, for example, by selecting the tag “racism”, which would direct to a cluster of images created using all the translations of “racism” available in the index of social tags.

Finally, we recommend having separate language versions of the tagging environment, while using multilingual social tags in the back-end for indexing and retrieval. This separation helps the processing of tags and eliminates the language identification problem. After processing the social tags, they can be used for filtering search results (i.e. return images only about social life and 20th century), suggesting terms for query expansion, recommending similar images, and clustering images to facilitate browsing.

For more, come to the CSCW 2012 session on Supporting Art and Literature, Wednesday February 15 at Seattle, WA.

About the author


Irene Eleta is a Ph.D. candidate at the University of Maryland iSchool. Her research interests span a variety of approaches for accessing and promoting multilingual information on the Internet, including multilingual social tagging, cross-language information retrieval, transnational social networks, and user studies of language choice in search and communication.

View all posts