Pedestrian navigation presents many unique challenges. Traditional maps are optimized for structured areas with good signage, which is uncommon for pedestrian areas. This leads to difficulties recognizing a particular place and in interpreting directions. We explored the use of landmarks in navigation systems, which studies have shown to be easier than map or distance-and-turn directions. Our own user studies also found images augmented with directional information were an effective navigation aid.

Building this system introduced several challenging problems across various fields. We developed a flexible framework for image collections to intelligently select landmarks and images based on a variety of factors. Our database operates with several thousand images and produces a navigational image (augmented with directions) in less than 220 milliseconds.

In order to augment images with arrows and navigation information, a world aligned camera pose was required. We developed a method to obtain quality camera poses by utilizing existing structure-from-motion reconstruction algorithms and aligning the results to a world coordinate system. This often increased camera location accuracy by more than 10 meters over standard GPS readings. This camera pose also allowed us to synthesize new instruction types and extend the usable range of images by showing them from new perspectives, both through “zoom-out” views of single images and views rendered from automatically created 3D models.

In addition to path information for navigation, we also enable rendering of building boundaries, labels, and even hyperlinks in an image captured by a user, thus creating a mobile augmented reality platform. We based the live-matching method on an image matching system designed to run in real time on a mobile phone. Due to differences in indoor environments, we developed a separate indoor system to perform live image-based localization by matching features to a floorplan.

A series of user studies informed our choices about direction generation and system design. Additional user studies evaluated the results of our automatic generation system. The studies showed image-based navigation was reviewed favorably, and the extensions beyond basic navigation had significant impact. Our improved instructional images were ranked as more accurate than the baseline images in 94% of cases.