This thesis addresses dimensionality reduction problems in classification for both high-dimensional multivariate and functional data.

High-dimensional data refers to data with a large number of variables, often larger than the number of observations. High-dimensional data are encountered in a wide range of areas such as engineering, biometrics, psychometrics, and neuroimaging. Classifying these data is a difficult problem because the enormous number of variables poses challenges to conventional classification methods and renders many classical techniques impractical. A natural solution is to add a dimensionality reduction step before a classification technique is applied.

In order to deal with multivariate data, two approaches are proposed. One is a simulated annealing (SA) based method and the other is a multivariate adaptive stochastic search (MASS) method. They both utilize stochastic search algorithms to select a handful of optimal transformation directions from a large number of random directions in each iteration. One advantage of the proposed methods is that they can accurately project the data onto very low-dimensional non-linear, as well as linear, spaces. These methods are designed to mimic variable selection type methods, such as the Lasso, or variable combination methods, such as PCA, or a method that combines the two approaches. Particularly, MASS can adaptively adjust the model complexity level, and hence performs well in situations where variable selection or variable combination methods fail. We demonstrate the strengths of SA and MASS on an extensive range of simulation and real studies by comparing them to many classical and modern classification methods.

Classification problems associated with functional data are also addressed. We propose a functional adaptive classification (FAC) approach which takes the functional response into consideration and produces highly accurate and interpretable results. FAC is also based on a stochastic search procedure guided by the evaluation of model complexity. This often results in a simple relationship between functional covariates and the reduced data and makes the model interpretable. Simulation studies and an fMRI time course study are also provided to show the effectiveness of the proposed method.