Abstract: Interpreting ultrasound data presents a significant challenge to medical personnel, which limits the clinical applications of the technology. In order to address this issue, we have developed a novel, flexible, and efficient view-based high-level representation for anatomical knowledge and used this model to create a semi-automated ultrasound interpretation system intended to aid non-expert medical practitioners in using ultrasound devices in a variety of different diagnostic situations. The design of the system incorporates techniques from computer graphics, computer vision, and machine learning along with results from a study of approaches used by human experts to interpret ultrasound examinations. Essentially, the system treats the collection of images generated during an ultrasound examination as an ordered sequence of views of the anatomical environment and picks out key views in which the contents of the scan image changes. It stores descriptions of expected key views and robustly matches incoming images to this key view sequence during an orientation phase of an examination. The system also uses information in the stored view descriptions to label the anatomical structures present in an input image and generates simple 3D anatomical models registered to the patient in order to create structure labels that align with incoming images during a subsequent free scanning phase. The prototype can guide a novice user through an examination of a patient's abdomen and automatically identify anatomical structures within the region. In addition, we have performed a pilot user study to determine the impact of the system's labels on non-experts' performances on a representative ultrasound task and found that the labels improve participants' efficiency and accuracy. Overall, the design represents a novel approach to processing and augmenting ultrasound data and to representing spatial knowledge, and it lays the groundwork for future efforts to develop fully automated medical imaging systems.