Electric power systems are subject to disturbances in the operation, and may encounter system failures such as power outages and blackouts due to disturbances. Power system security analysis plays an important role in improving the survivability to disturbances. This dissertation proposes advanced computational and optimization techniques that can be applied to mitigate instabilities in power systems subject to disturbances. The research work has been integrated into a general framework for power system dynamic security analysis. The proposed methods cover strategies for both power system instability assessment and control, and provide a fast simulation algorithm and coordinated optimization techniques to improve power system security. In the assessment phase of power system security analysis, a fast algorithm is proposed to identify power system dynamic behavior using decoupled time domain simulation method. Traditional time domain simulation algorithms can be categorized as explicit and implicit methods. While explicit methods are fast, the simulation results cannot be guaranteed to be correct for stiff dynamical systems. On the other hand, implicit methods may give correct qualitative behavior with slow performance. As a hybrid method, the proposed decoupled method improves the computational efficiency and achieves numerical stability of time domain simulation by combining the advantages of traditional explicit and implicit methods, and the decomposition is accomplished through invariant subspace partition with rigorous mathematical analysis. In mitigation phase of system security analysis, a coordinated control strategy based on trajectory optimization is proposed. Power system dynamic performance is improved by the proposed method within the constraints imposed on system transition. In addition to the equilibrium conditions, inequality constraints in power system dynamics such as voltage level are considered in the formulation and solved through penalty function method. As one of the applications, power quality such as voltage dip in power system dynamics can be improved. Cascading events may also be prevented by including transitional constraints in the trajectory optimization. Numerical examples of test power systems are presented to demonstrate the applications of the proposed methods.