In many classification problems data samples cannot be collected easily, example in drug trials, biological experiments and study on cancer patients. In many situations the data set size is small and there are many outliers. When classifying such data, example cancer vs normal patients the consequences of misclassification are probably more important than any other data type, because the data point could be a cancer patient or the classification decision could help determine what gene might be over expressed and perhaps a cause of cancer. These mis-classifications are typically higher in the presence of outlier data points. The aim of this thesis is to develop a maximum margin classifier that is suited to address the lack of robustness of discriminant based classifiers (like the Support Vector Machine (SVM)) to noise and outliers. The underlying notion is to adopt and develop a natural loss function that is more robust to outliers and more representative of the true loss function of the data. It is demonstrated experimentally that SVM's are indeed susceptible to outliers and that the new classifier developed, here coined as Robust-SVM (RSVM), is superior to all studied classifier on the synthetic datasets. It is superior to the SVM in both the synthetic and experimental data from biomedical studies and is competent to a classifier derived on similar lines when real life data examples are considered.