Notions of optimal behavior expressed in natural systems led researchers to develop reinforcement learning (RL) as a computational tool in machine learning to learn actions by trial and error interactions yielding either a reward or punishment. RL provides a way for learning agents to optimally interact with uncertain complex environments, and hence, can address problems from a variety of domains, including artificial intelligence, controls, economics, operations research, etc.

The focus of this work is to investigate the use of RL methods in feedback control to improve the closed-loop performance of nonlinear systems. Most RL-based controllers are limited to discrete-time systems, are offline methods, require knowledge of system dynamics and/or lack a rigorous stability analysis. This research investigates new control methods as an approach to address some of the limitations associated with traditional RL-based controllers.

A robust adaptive controller with an adaptive critic or actor-critic (AC) architecture is developed for a class of uncertain nonlinear systems with disturbances. The AC structure is inspired from RL and uses a two pronged neural network (NN) architecture—an action NN, also called the actor, which approximates the plant dynamics and generates appropriate control actions; and a critic NN, which evaluates the performance of the actor, based on some performance index.

In the context of current literature on RL-based control, the contribution of this work is the development of controllers which learn the optimal policy (approximately) for uncertain nonlinear systems. In contrast to model learning strategies for RL-based control of uncertain systems, the requirement of model knowledge is obviated in this work by the development of a robust identification-based state derivative estimator. The robust identifier is designed to yield asymptotically convergent state derivative estimates which are leveraged for model-free formulation of the Bellman error. The identifier is combined with the traditional actor-critic resulting in a novel actor-critic-identifier architecture, which is used to approximate the infinite-horizon optimal control for continuous-time uncertain nonlinear systems. The method is online, partially model-free, and is the first ever indirect adaptive control approach to continuous-time RL.