Operating systems rely heavily on access control mechanisms to achieve security goals and defend against remote and local attacks. The complexities of modern access control mechanisms and the scale of policy configurations are often overwhelming to system administrators and software developers. Therefore, mis-configurations are common, and the security consequences are serious. It is critical to have models and tools to analyze thoroughly the effectiveness of access control policies in operating systems and to eliminate configuration errors.

In this dissertation, we propose an approach to systematically analyze access control policies in operating systems. The effectiveness of a policy can be evaluated under attack scenarios. An attack scenario consists of the initial resources an attacker has and the attacker's objective. Attacks under an attack scenario are encoded in a host attack graph. Compared to existing solutions, our approach is more comprehensive and does not rely on manually defined attack patterns.

Based on the model, a tool called VulSAN is implemented to analyze policies in Linux systems, and a tool calledWACCA is implemented to analyze policies inWindows systems. We analyze policies in Ubuntu, Fedora, SUSE Linux and Windows Vista. We discuss the results and show the possibilities to improve the quality of protection. The results are also used to compare the effectiveness of SELinux and AppArmor policies in a version of Ubuntu Linux.