A fundamental goal of Computer Vision is to provide scene understanding and situational awareness. In order to deliver on this promise, traditional monitoring systems were designed for specific environmental situations, such as a specific time, place, or activity scenario. A well versed expert defines the events of interest by hand for the particular application. While effective, these techniques do not scale well, they typically have poor generalization, are inflexible to behavioral changes, and the analysis rules may not reflect the true nature of the scene but a priori expectations. The conventional methods to understand activities must be scaled to match growing need. Society's rapid acceptance of video use in a wide variety of locations and applications has promoted the deployment of large camera networks. These networks monitor complex scenes and deliver volumes of video data that can not be digested without automated assistance.

This dissertation investigates unsupervised activity understanding by analyzing patterns of motion trajectories. A practical approach is introduced and carefully developed to overcome the difficulties with trajectory learning, namely the definition of a simple activity model that can be robustly inferred from crude measurements, the automatic determination of the number of typical activities in a scene, and methods to observe dynamic scenes over long periods. The activity analysis framework is able to process and analyze activity, providing activity characterization, prediction, and abnormality detection, in real-time for real world utility.

The efficacy of the trajectory learning framework is demonstrated in three distinct arenas. Highway traffic is monitored using a single camera with the VECTOR system, multiple sensors are integrated and combined in a unified space with CANVAS, and driving maneuvers analyzed from within a moving automobile. This extension of the trajectory learning paradigm to a broad range of (untouched) application spaces further highlights the dissertation contributions. Finally, extensive performance evaluation and characterization is conducted to provide a missing benchmark for the field.