The growing complexity of interactions between computers and networks makes the subject of network security a very interesting one. As our dependence on the services provided by computing networks grows, so does our investment in such technology. In this situation, there is a greater risk of occurrence of targeted malicious attacks on computers and networks, which could result in system failure. At the user level, the goal of network security is to prevent any malicious attack by a virus or a worm. However, at the network level, total prevention of such malicious attacks is an impossible and impractical objective to achieve. A more attainable objective would be to prevent the rampant proliferation of a malicious attack that could cripple the entire network.

Traditional Intrusion Detection Systems (IDSs) focus on the detection of attacks at the individual nodes, after a malicious code has entered individual machines in a network. However, repeated failures of conventional IDSs have led researchers to develop methods that integrate detection systems in networks and use their collective intelligence to defend against malicious attacks. Such approaches utilize the synergistic power generated by the network, as nodes share prior and current knowledge of detected attacks and related information with other nodes.

This dissertation investigates the practical application of a cooperative approach, used to defend computer networks against attacks from external agents. In this dissertation I focus on the detection of metamorphic NOP (No OPeration) sleds, which are common in buffer overflow attacks, and the role of topology on the rate of spread of a malicious attack. The aim of this study is to use the results to provide recommendations that can be utilized to develop optimal network security policies.