It is well known that observations gathered in the real world are not perfectly independent as is assumed in many data analyses. Typically a linear mixed model (LMM) is used when the clustering structure is known and it provides a powerful tool. Small area estimation, that is, predicting population means of a variable of interest, when only a few observations from each population are sampled, is one application of LMMs. In contrast to LMMs, most supervised learning methods do not take into account whether the data have some clustered structure. Random forests, one such method which uses many randomized decision trees, is a popular algorithm used to model large complex data sets because it tends to produce accurate predictions.

In this dissertation, two estimators of residual variance are proposed using random forest and they are studied through simulations. A robust modeling technique for mixed-effects data is then proposed, called Mixed Random Forest (MRF), that uses regression trees and accounts for the data's clustered structure. The performance of the MRF algorithm is compared to that of LMMs for different underlying functions and different values of the variance components. The MRF method is shown to perform better in terms of mean squared prediction error (MSPE) when the underlying function is complex, such as conditionally linear. The theoretical MSPE of the predicted group mean is also derived and an estimator of the MSPE is proposed. The performance of the MSPE estimator is investigated via simulations and the results back up the theoretical result. The MRF method is applied to data from the American Community Survey in the small area estimation context.