Performance optimization of communication networks involves challenges at both the engineering level and the human level. In the first part of the dissertation, we study a network security game where strategic players choose their investments in security. Since a player's investment can reduce the propagation of computer viruses, a key feature of the game is the positive externality exerted by the investment. With selfish players, unfortunately, the overall network security can be far from optimum. First, we characterize the price of anarchy (POA) in the strategic-form game under an "effective-investment" model and a "bad-traffic" model, and give insight on how the POA depends on the network topology, the cost functions of the players, and their mutual influence (or externality). We show that the POA in general cannot be offset by the improvement of security technology. Second, in a repeated game, users have more incentive to cooperate. We characterize the socially best outcome that can be supported by the repeated game, as compared to the social optimum. We also introduce a Folk Theorem which only requires local punishments and rewards, but supports the same payoff region as the usual Folk Theorem. Finally, with a social planner who implements a due-care scheme which mandates the minimal investments, we study how the performance bound improves. Although our primary focus is Internet security, many results are generally applicable to games with positive externalities.

In the second part of the dissertation, we consider the problem of achieving the maximum throughput and utility in a class of networks with resource-sharing constraints. This is a classical problem which had lacked an efficient distributed solution. First, we propose a fully distributed scheduling algorithm that achieves the maximum throughput. Inspired by CSMA (Carrier Sense Multiple Access) which is widely deployed in today's wireless networks, our algorithm is simple, asynchronous and easy to implement. Second, using a novel maximal-entropy technique, we combine the CSMA scheduling algorithm with congestion control to approach the maximum utility. Also, we further show that CSMA scheduling is a modular MAC-layer algorithm that can work with other protocols in the transport layer and network layer. Third, for wireless networks where packet collisions are unavoidable, we establish a general analytical model and extend the above algorithms to that case.

Stochastic Processing Networks (SPNs) model manufacturing, communication, and service systems. In manufacturing networks, for example, service activities require parts and resources to produce other parts. SPNs are more general than queueing networks and pose novel challenges to throughput-optimum scheduling. In the third part of the dissertation, we proposes a "deficit maximum weight" (DMW) algorithm to achieve throughput optimality and maximize the net utility of the production in SPNs.