Abstract: Selecting and encoding local features that preserve salient characteristics of an object, but are invariant to the typical variability of its appearance is crucial for recognition. This dissertation develops techniques for choosing good features for visual object categorization and identification. The first chapter of this work focuses on the problem of choosing encodings and comparison functions for local patches that help overcome the typically high intra-class variability of object categories. We present an experimental framework for evaluating various local image descriptors, comparing several widely used methods, and we suggest reasons for their varying performance. Based on patterns discovered during these tests, we offer a new representation that incorporates the most successful aspects of the other methods. Finally, we verify our results by constructing a very simple classifier for detecting faces, and show that the performance of the classifier using different local patch descriptors is consistent with our earlier results. The remainder of this work is dedicated to the problem of object identification, where the category is known and the algorithm recognizes an object's exact identity. Identification is characterized by two special challenges: (1) Inter-class variation is often small and may be dwarfed by illumination or pose changes; (2) There may be many classes, but few or just one positive “training” examples per class. Due to (1), a solution must locate possibly subtle object-specific salient features (e.g. a door handle) while avoiding distracting ones (e.g. a specular highlight). However, (2) rules out direct techniques of feature selection. We describe an on-line algorithm that takes one model image from a known category and builds an efficient “same” vs. “different” classification cascade by predicting the most discriminative feature set for that object. Our method not only estimates the saliency and scoring function for each candidate feature, but also models the dependency between features, building an ordered feature sequence unique to a specific model image, maximizing cumulative information content. Learned stopping thresholds make the classifier very efficient. To make this possible, category-specific characteristics are learned automatically in an off-line training procedure from labeled image pairs of the category, without prior knowledge about the category.