With an aging population, prostate cancer has become a major medical and socioeconomic problem. As a widespread but less aggressive cancer, the early diagnosis of prostate cancer is critically important for the treatment of this disease. On the other hand, modern imaging and computer techniques open the window for improving diagnostic accuracy of prostate cancer using computer-aided systems. The thesis investigates medical image analysis methods that are important components of the computer-aided prostate cancer diagnostic system. Work presented in this thesis is a part of a larger effort to design a computer-aided biopsy system, which aims to increase the diagnostic accuracy of prostate biopsy using population-based statistical information and patient-specific image information.

This thesis makes several major contributions. First, we develop a statistical image analysis method to construct a statistical atlas of prostate cancer distribution using a large cohort of histological images of prostatectomy specimens. Based on this statistical atlas, an optimized biopsy strategy was derived to maximize the cancer detection rate. Compared to random systematic biopsy strategies, the atlas-based biopsy strategy achieves significantly higher diagnostic accuracy. Second, a deformable model is designed to segment prostates from transrectal ultrasound (TRUS) images. In contrast to existing deformable models that are guided by edge information, the proposed deformable model is guided by texture analysis results. Using abundant texture information in prostate TRUS images, the proposed method provides more robust and accurate segmentation results. Third, a reformulated support vector machine (SVM) is proposed to increase the classification speed of the SVM. The objective function of the SVM is reformulated to prevent extreme outliers in training sets dominating the objective function. A deterministic annealing framework is proposed to train the reformulated SVM. Compared to the standard SVM, the reformulated SVM is much faster and has better generalization ability. Fourth, we design a method for the registration of prostate histological and MR images. This registration method is guided by landmark points, which can be automatically detected from both histological and MR images using a scale-space analysis method. Compared to related work, our registration method is more effective in mapping the cancer ground truth from prostate histological images to MR images.

Image analysis methods proposed in this thesis not only relate to an important clinical application, prostate cancer diagnosis, but also share common research themes, e.g., probabilistic optimization, texture analysis, deformable matching, pattern recognition and scale-space analysis. We hope the work of this thesis will benefit the clinical community and promote future developments in the research areas of medical image analysis, computer vision and pattern recognition.