Analyzing naturally crowdsourced instructions at scale

“The proportion of ingredients is important, but the final result is also a matter of how you put them together,” said Alain Ducasse, one of only two chefs in the world with 21 Michelin stars. In fact, for cooking professionals like chefs, cooking journalists, and culinary students, it is important to understand not only the culinary characteristics of the ingredients but also the diverse cooking processes. For example, categorizing different approaches to cooking a dish or identifying usage patterns of particular ingredients and cooking methods are crucial tasks for them.

However, these analysis tasks require extensive browsing and comparison which is very demanding. Why? It’s because there are thousands of recipes available online even for something seemingly as simple as making a chocolate chip cookie.

screen-shot-2018-04-12-at-01-17-22Figure 1: search results for “chocolate chip cookie” in leading recipe websites.

In essence, these recipes are naturally crowdsourced instructions for a shared goal, like making a chocolate chip cookie. They exist in diverse contexts (no oven, no gluten, from scratch, etc.), diverse level of details and lengths, different levels of required skills and expertise, and in different writing styles.

We devised a computational pipeline which a) constructs graphical representation which captures semantic structure for each recipe in natural language text using machine-assisted human computation techniques. b) compares structural and semantic similarities between every recipe.

On top of the pipeline, we built an analytics dashboard which invites cooking professionals to analyze a large collection of recipes for a single dish.

ss_dashboardFigure 2: RecipeScape is an interface for analyzing cooking processes at scale with three main visualization components: (a) RecipeMap provides clusters of recipes with respect to their structural similarities, where each point on the map is a clickable recipe, (b) RecipeDeck provides in-depth view and pairwise comparisons of recipes, (c) RecipeStat provides usage patterns of cooking actions and ingredients.

The interface aims to provide analysis on three different levels. It aims to a) provide statistical information of individual ingredients, cooking actions. b) provide analysis for structural comparison of recipes to examine the representative, outlier recipes. c) provide clusters of recipes to examine fundamental similarities and differences of the various approaches.

Our user study with 4 cooking professionals and 7 culinary students suggest that RecipeScape broadens browsing and analysis capabilities. For example, users have reported RecipeScape enables explorations of both common and exotic recipes, comparisons of substitute ingredients and cooking actions, discovers fundamentally different approaches to cooking a dish and assists imagination and mental simulations of diverse cooking processes.

Our approach is not only bounded to cooking but is applicable to other how-tos like software tutorials, makeup instructions, furniture assemblies and more.

This work was presented at CHI 2018 in Montreal as “RecipeScape: An Interactive Tool for Analyzing Cooking Instructions at Scale”. For more detail, please visit our project website,