Abstract: Increasing needs for autonomy in future aerospace systems and immense progress in computing technology have motivated the development of on-line adaptive control techniques to account for modeling errors, changes in system dynamics, and faults occurring during system operation. After extensive treatment of the inner-loop adaptive control dealing mainly with stable adaptation towards desired transient behavior, adaptive optimal control has started receiving attention in literature. Motivated by the problem of optimal control of the magneto-hydrodynamic (MHD) generator at the inlet of the scramjet engine of a hypersonic flight vehicle, this thesis treats the general problem of efficiently combining off-line and on-line optimal control methods. The predictive control approach is chosen as the off-line method for designing optimal controllers using all the existing system knowledge. This controller is then adapted on-line using policy-iteration-based Q-learning, which is a stable model-free reinforcement learning approach. The combined approach is first illustrated in the optimal control of linear systems, which helps in the analysis as well as the validation of the method. A novel neural-networks-based parametric predictive control approach is then designed for the off-line optimal control of non-linear systems. The off-line approach is illustrated by applications to aircraft and spacecraft systems. This is followed by an extensive treatment of the off-line optimal control of the MHD generator using this neuro-control approach. On-line adaptation of the controller is implemented using several novel schemes derived from the policy-iteration-based Q-learning. The implementation results demonstrate the success of these on-line algorithms for adapting towards modeling errors in the off-line design.