Since the recent advent of digital cameras, large collections of images are stored in personal computers in an un-organized fashion. To structure these image datasets for retrieval purpose, two line of researches have been conducted: the purely text-based methods and the content-based methods. The purely text-based methods rely solely on manual image annotations which is too simplistic to solve many practical applications. On the other hand, the traditional content-based methods take advantage of the rich image content information but are still severely limited by the semantic gap between low level features and high level concepts.

To address this issue, we have developed a novel ontology-based image organization framework that exploits both sources of information. First, the ontology is extracted from noisy image annotations to form a concept space that covers interesting image semantics in the image datasets. Then we learn the mapping between raw image data and image concepts by first breaking the entire semantic gap into multiple smaller gaps, then designing specialized algorithms to bridge each of these gaps in a bottom-up fashion.

Specifically we have developed: (a) complementary salient object detectors to capture a diverse set of middle-level image semantics; (b) a hierarchical joint boosting and feature selection algorithm to effectively and efficiently perform weak classifier fusioning; (c) a Bayesian formulation for multi-level image annotation reasoning.

With the text and image analysis result, we have developed a novel ontology-based visualization system to help users to navigate within the image datasets. This visualization system uses the concept ontology in hyperbolic visualization space as a global summary of the entire dataset where users can then "visually'' submit queries. To provide intuitive presentations of image search results, a kernel-based image projection algorithm is also developed for similarity-based groupings of the returned image dataset. We discover that this visualization system together with a kernel-based relevance feedback mechanism can be used to address the problem of training data acquisition for large-scale concept learnings.

By implementing all these ideas into one integrated system, the original unstructured image collections can be then readily accessed by users through keyword-based search, ontology-based browsing and dynamic visualization in a multi-modal fashion. Experiments on large scale image datasets have demonstrated that the proposed system is effective in structuring diverse image content and improving user experience in content-based image organization and retrieval.