Reinforcement Learning (RL) is a machine learning technology in which a computer agent learns, through trial and error, the best way to accomplish a particular task [1].  The recent development of Deep Reinforcement Learning (DRL) technology, in which deep learning networks [2] are used to parameterize the RL agent’s policy and value functions, has enabled superhuman performance for some tasks, most especially games such as chess and GO [3].  Researchers have devised DRL algorithms specifically for continuous control problems [4,5], and this has led many in the process control community to wonder what impact DRL technology will have on our industry.  Two recent papers provide an introduction to DRL technology and an analysis of its appropriateness for industrial process control applications [6,7].  This presentation builds on the previous analysis efforts by proposing a that an industrial process control technology must be:

•    Intelligent
•    Consistent
•    Offset-free
•    Nominally stable
•    Flexible

DRL technology will be assessed with respect to each requirement, exposing existing deficiencies, useful research directions, and potential solutions.  It will be shown that DRL technology is not likely to replace currently used Proportional-Integral-Derivative (PID) or Model-Predictive Control (MPC) algorithms, however it may prove useful in managing control systems by helping to tune controllers [8,9,10], advising operators during upsets or transient operations [6], and by managing plant-wide disturbances such as diurnal changes and weather events [6].

A final section will outline several promising research directions for DRL technology [11].

References

[1] RS Sutton, AG Barto, “Reinforcement Learning – An Introduction”, The MIT Press, (2018).
[2] I Goodfellow, Y Bengio, A Courville, “Deep Learning”, The MIT Press, (2016).
[3] D Silver, A Huang, CJ Maddison, A Guez, L Sifre, G Van Den Driessche, J Schrittwieser, I. Antonoglou, V Panneershelvam, M Lanctot, “Mastering the game of Go with deep neural networks and tree search”, Nature 529, 484–489 (2017).
[4] TP Lillicrap, JJ Hunt, A Pritzel, N Heess, T Erez, Y Tassa, D Silver, D Wierstra, “Continuous control with deep reinforcement learning”, arXiv:1509.02971 (2015).
[5] L Busoniu, T deBruin, D Tolic, J Kober, I Palunko, “Reinforcement learning for control: Performance, stability, and deep approximators”, Annual Reviews in Control, 46, 8-28, (2018).
[6] J Shin, TA Badgwell, KH Liu, JH Lee, “Reinforcement Learning – Overview of recent progress and implications for process control”, Computers & Chemical Engineering, 127, 282-294 (2019).
[7] RN Nian, J Liu, B Huang, “A review on reinforcement learning: Introduction and applications in industrial process control”, Computers & Chemical Engineering 139, 106886 (2020).
[8] T Badgwell, K Liu, N Subrahmanya, M Kovalski, “Adaptive PID controller tuning via deep reinforcement learning”, US Patent 10,915,073, (2021).
[9] O Dogru, K Velswamy, F Ibrahim, Y Wu, AS Sundaramoorthy, B Huang, S Xu, M Nixon, N Bell, “Reinforcement learning approach to autonomous PID tuning”, Computers & Chemical Engineering, 161, 107760 (2022).
[10] NP Lawrence, MG Forbes, PD Loewen, DG McClement, JU Backstrom, RB Gopaluni, “Deep Reinforcement Learning with Shallow Controllers: An Experimental Application to PID Tuning”, arXiv, 2111.07171v1 (2021).
[11] A Mesbah, KP Wabersich, AP Schoellig, MN Zeilinger, S Lucia, TA Badgwell, JP Paulson, “Fusion of Machine Learning and MPC under Uncertainty: What Advances Are on the Horizon?”, American Control Conference, June 8-10, Atlanta, GA (2022).
 

We are currently in the midst of a fourth industrial revolution (Industry 4.0 [1]), involving the large-scale automation of traditional manufacturing and industrial practices, made possible by recent developments in mathematical algorithms, computer hardware, and internet connectivity (Industrial Internet of Things (IIoT) [2]).  While much of this work can be considered evolutionary in nature, in this presentation we highlight three emerging technologies that appear to be truly disruptive; that is, they are likely to have such a large impact that they will change the way theoreticians and practitioners think about and accomplish manufacturing and process control.  These technologies are Economic Model Predictive Control (EMPC), Deep Reinforcement Learning (DRL), and Open Process Automation (OPA).

Economic MPC (EMPC) is a relatively new technology that combines economic optimization with Model Predictive Control [3], two functions that are traditionally implemented separately.  While the theory was worked out a few years ago [4], EMPC applications have only begun to appear recently.  Professor Jim Rawlings and co-workers, for example, presented a successful EMPC implementation for the Stanford University campus heating and cooling system [5].  Recent theoretical work has shown that scheduling problems, which are usually approached from the point of view of static optimization, can also be considered as a special case of closed-loop EMPC [6].  This unification of closed-loop scheduling, economic optimization, and dynamic control has shed new light on such problems as rescheduling in the face of disturbances, and provides academics with a completely new framework for viewing and analyzing scheduling problems. Practitioners now have, for the first time, the hope of combining three disparate levels in the traditional control hierarchy into a single, harmonious layer.

Reinforcement Learning (RL) is a Machine Learning (ML) technology in which a computer agent learns, through trial and error, the best way to accomplish a particular task [7].  Deep Learning (DL) is a technology in which neural networks with a large number of intermediate layers are used to model relationships [8].  Using DL to parametrize the policy and value function of an RL agent leads to Deep Reinforcement Learning (DRL) technology, which allows an agent to achieve superhuman performance for some tasks.  In 2017, for example, a DRL agent named AlphaGo soundly defeated the reigning world champion Go player [9].  Applications of this technology to manufacturing and process control systems are currently under study [10].  It is likely that DRL will not replace currently successful control algorithms such as PID and MPC, but will rather takeover some of the mundane tasks that humans perform to manage automation and control systems.  For example, it appears that a DRL agent can learn how to tune PID loops effectively [11].  Other possibilities include advising operators during transient and upset conditions, mitigating disturbances such as weather events, and detecting and mitigating unsafe operations [10].

The industrial automation marketplace, comprised of Distributed Control System (DCS), Programmable Logic Controller (PLC), and Supervisory Control and Data Acquisition (SCADA) technology offerings, will soon experience a historic, game-changing disruption with the emergence of Open Process Automation (OPA) technology.  Manufacturers, whose innovations have been constrained for decades by the limitations of closed, proprietary systems, will soon experience the benefits of open, interoperable, resilient, secure-by-design automation systems, made possible by the development of the consensus-based Open Process Automation Standard (O-PAS) by the Open Process Automation Forum (OPAF) [12].  Once O-PAS certified automation systems become widespread, vendors will see the market for their products and services expand significantly as the visions of I4.0 and the IIoT are realized.  Academics and technology developers will see more opportunities to test their solutions as it becomes easier to deploy them.  Dr. Don Bartusiak, co-director of OPAF, summarizes their progress to date in a paper published recently in Control Engineering Practice [12].  

References

[1] Y Liao, F Deschamps, EdFR Loures, LFP Ramos, “Past, present, and future of Industry 4.0 - a systematic literature review and research agenda proposal”, Intl J Production Research, 55 (12), 3609-3629, (2017).
[2] H Boyes, B Hallaq, J Cunningham, T Watson, “The industrial internet of things (IIoT): An analysis framework”, Computers in Industry, 101, 1–12, (2018).
[3] SJ Qin, TA Badgwell, “A survey of industrial model predictive control technology”, Control Engineering Practice 11 (7), 733-764, (2003).
[4] JB Rawlings, D Angeli, CN Bates, “Fundamentals of economic model predictive control”, 51st Conference on Decision and Control, 3851-3861, (2012).
[5] JB Rawlings, NR Patel, MJ Risbeck, CT Maravelias, MJ Wenzel, RD Turney, “Economic MPC and real-time decision making with application to large-scale HVAC energy systems”, Computers & Chemical Engineering, 114 (6), 89-98, (2018).
[6] MJ Risbeck, CT Maravelias, JB Rawlings, “Unification of Closed-Loop Scheduling and Control:  State-space Formulations, Terminal Constraints, and Nominal Theoretical Properties”, Computers & Chemical Engineering, 129 (10), (2019).
[7] RS Sutton, AG Barto, “Reinforcement Learning – An Introduction”, The MIT Press, (2018).
[8] I Goodfellow, Y Bengio, A Courville, “Deep Learning”, The MIT Press, (2016).
[9] D Silver, A Huang, CJ Maddison, A Guez, L Sifre, G Van Den Driessche, J Schrittwieser, I. Antonoglou, V Panneershelvam, M Lanctot, “Mastering the game of Go with deep neural networks and tree search”, Nature 529, 484–489 (2017). 
[10] J Shin, TA Badgwell, KH Liu, JH Lee, “Reinforcement Learning – Overview of recent progress and implications for process control”, Computers & Chemical Engineering, 127, 282-294 (2019).
[11] TA Badgwell, KH Liu, NA Subrahmanya, WD Liu, MH Kovalski, “Adaptive PID Controller Tuning via Deep Reinforcement Learning”, U.S. patent 1095073, granted February 9, 2021. 
[12] RD Bartusiak, S Bitar, DL DeBari, BG Houk, D Stevens, B Fitzpatrick, P Sloan, “Open Process Automation: A Standards-Based, Open, Secure, Interoperable Process Control Architecture”, Control Engineering Practice 121, 105034, (2022).

We consider data-driven methods for modeling discrete-valued dynamical systems evolving over networks. The spread of viruses and diseases, the propagation of ideas and misinformation, the fluctuation of stock prices, and correlations of financial risk between banking and economic institutions are all examples of such systems. In many of these systems, data may be widely available, but approaches to identify relevant mathematical models, including the underlying network topology, are not widely established or agreed upon. Classic system identification methods focus on identifying continuous-valued dynamical systems from data, where the main analysis of such approaches largely focuses on asymptotic properties, i.e., consistency. More recent identification approaches have focused on sample complexity, i.e., how much data is needed to achieve an acceptable model approximation. In this talk, we will discuss the problem of identifying a mathematical model from data for a discrete-valued, discrete-time dynamical system evolving over a network. Specifically, under maximum likelihood estimation approaches, we will demonstrate guaranteed consistency conditions and sample complexity bounds. Applications to the aforementioned examples will be further discussed as time allows.

The study of epidemic processes has been of interest over a wide range of fields for the past century, including in mathematical systems, biology, physics, computer science, social sciences and economics. Recently there has been renewed interest in the study of epidemic processes focused on the spread of viruses over networks, motivated not only by recent outbreaks of infectious diseases, but also by the rapid spread of opinions over social networks, and the security threats posed by computer viruses. In this talk we will discuss modeling and convergence analysis results for epidemic processes over both static and time-varying networks, with the goal being to elucidate the behavior of such spread processes. Multi-strain models, and issues arising from epidemic modeling and prediction based on the use of data from ongoing viral outbreaks will also be discussed as time allows. Simulation results and potential mitigation actions will be reviewed to conclude the talk.

We consider the problem of clustering data sets where the data points are dynamic, or essentially time-varying. Our approach is to incorporate features of both the deterministic annealing algorithm as well as control theoretic methods in our computational solution. Extensions of our method can be made to the problem of aggregating time-varying graphs, for which we have developed a quantitative measure of dissimilarity that allows us to compare directed graphs of differing sizes. In this talk, an overview of our dynamic clustering algorithm will be given, along with some analysis of the algorithm properties. We will conclude with a few highlighted applications, and further extensions as time allows.

Evolutionary dynamics in large populations of decision-making autonomous agents have become a powerful model to study complex interactions in natural, social, economic and engineering systems. In this talk I focus on showing how evolutionary game theoretic models can be studied using systems and control theory. We look into how feedback actions can be incorporated and demonstrate that the closed-loop population dynamics may exhibit drastically different collective outcomes.

Team movement control, including navigation and path-following, are fundamental functions for mobile robots carrying out environmental monitoring and sampling tasks. New challenges arise when control algorithms have to be designed for a team of robots with limited communication capacity and the environment may contain obstacles. In this talk, I show how to design guiding vector fields to enable motion coordination among robots; I also show how to construct composite guiding vector fields to avoid colliding with obstacles of arbitrary shapes. Both theoretical guarantees and experimental validations are discussed for practical scenarios.

Google search for the term ‘software defined vehicles’ returned about 646M results in 0.33 seconds. The automotive industry is awash with this new buzz phrase. In their report “Rewiring car electronics and software architecture for the Roaring 2020s,” futurists from McKinsey & Company are heralding sweeping industry transformation both technically and commercially. We, GM, are making a splash with our Ultifi Platform that reimagines what it means to own and experience a vehicle and to grow revenue beyond vehicle sales. But what does all this mean for our vehicles – where the rubber meets the road, literally?!

This talk will explore the software defined future from a vehicle point of view. Using examples of core vehicle functions like motion control – embedded software that makes our vehicles stop, turn & go – we will dive into the challenges and opportunities that lie ahead and how we can all engage and shape this exciting future.

Distributed control is a classical research topic. While a rich theory is available, some assumptions such as availability of subsystem dynamics and topology and the subsystems following the prescribed controllers exactly have proven difficult to remove. An interesting direction in recent times to get away from these assumptions has been the utilization of learning for control. In this talk, we consider some problems in control design for distributed systems using learning. Our core message is that utilizing control-relevant properties in learning algorithms can not only guarantee concerns such as stability, performance, safety, and robustness that are important in control of physical systems, but also help with issues such as data sparsity and sample complexity that are concerns during the implementation of learning algorithms.

Fluid-like models, such as the Lighthill-Whitham-Richards (LWR) model and their discretizations like Daganzo’s Cell Transmission Model (CTM), have proven successful in modeling traffic networks. In general, these models are not linear; they employ discontinuous dynamics or nonlinear terms to describe phenomena like shock waves and phantom jams. Given the complexity of the dynamics, it is not surprising that the stability properties of these models are not yet well characterized. Recent results have shown the existence of a unique equilibrium in the free flow regime for certain classes of networks modeled by the CTM; however, these results restrict inflows to the system to be bounded or constant. Further, it is of interest to understand the system behavior in congested regimes since links in various practical networks are often congested.

Many learning algorithms have been proposed for design of control policies in cooperative and competitive multi-agent systems. We explore the robustness of some such algorithms to the presence of strategic agents. First, we show that some recently proposed multi-agent reinforcement learning algorithms are vulnerable to being hijacked by even one agent that prioritizes individual utility function over the team utility function and propose a way to make the algorithms robust to such attacks. Then we consider a game set up in which agents are employing a fictitious play-based learning algorithm and show that an agent can move the game to a more favorable equilibrium by deviating from the prescribed algorithm.

This talk focuses on early research into control challenges of large transport aircraft that progressed from low to high Technology Readiness Levels (TRLs). We look at example projects such as autonomous in-air refuelling and autonomous formation flight, and highlight the approaches that were followed towards progressing up the TRL ladder. The interface between academia and industry is often complex, requiring academics and industry partners to step out of their comfort zones and to have a multi-disciplinary approach to solving problems. It is at this interface where we turn challenges into very interesting research problems and eventually into very useful industrialised systems. The talk includes high level discussions on the application of various basic and advanced control systems techniques (without resorting to mathematics) as well as general lessons learned from industry collaboration. It is suitable to an audience with a generalised knowledge of post-graduate- level control systems and an interest in effective industry collaboration.

Most conventional aircraft were created to be flown by human pilots. This affects the fundamental design configurations of aircraft and their handling qualities. In this anthology talk, we first look at various flight control examples where we could learn from human behaviour to create simple, safe and effective flight control systems for conventional aircraft. We also see how conventional aircraft can be extended to improve their performance by relying on unconventional approaches. Eventually this line of thinking leads us to creating unconventional high performance and energy efficient aircraft. This talk is suited to an

unconventional high performance and energy efficient aircraft. This talk is suited to an audience with a general control systems background, including graduate students of various engineering disciplines, with an interest in the fundamentals and practical qualities of flight stabilisation and control algorithms.

This lecture will survey the area of quantum control engineering. It will discuss models for quantum systems using both the Schrodinger and Heisenberg pictures of quantum mechanics including finite level quantum systems and continuous linear quantum systems. It will also discuss the open loop quantum control of quantum systems including robust and learning based approaches. In addition, it will discuss closed loop approaches to quantum control including measurement based feedback control and quantum filtering along with coherent quantum feedback control in which the controller is also a quantum system. In the area of coherent control of quantum linear quantum systems, it will discuss quantum H-infinity control, quantum LQG control and coherent quantum observers and coherent quantum state estimation. The lecture will also cover the quantum Kalman decomposition. Applications in the areas of quantum optics and quantum electromechanical systems will be presented.

This lecture presents a survey of some of the main results in the theory of negative imaginary systems. The lecture also presents some applications of negative imaginary systems theory in the design of robust controllers. In particular, the lecture concentrates on the application of negative imaginary systems theory in the area of control of atomic force microscopes.

While autonomous agents that perform solo missions can yield significant benefits, greater efficiency and operational capability will be realized from teams of autonomous agents operating in a coordinated fashion. Potential applications for networked multiple autonomous agents include environmental monitoring, search and rescue, space-based interferometers and hazardous material handling. Networked multi-agent systems place high demands on features such as low cost, high adaptivity and scalability, increased flexibility, great robustness, and easy maintenance. To meet these demands, the current trend is to design distributed control algorithms that rely on only local interaction to achieve global group behavior.

The purpose of this talk is to overview our research in distributed control of multi-agent systems. Theoretical results on distributed leaderless consensus with agent dynamics including first- and second-order linear dynamics, rigid body attitude dynamics, and Euler-Lagrange dynamics, distributed single-leader collective tracking with reduced interaction and partial measurements, distributed multi-leader containment control with local interaction, distributed average tracking with multiple time-varying reference signals, and distributed optimization with non-identical constraints will be introduced. Application examples in multi-vehicle cooperative control will also be introduced.

The problem of distributed dynamic state estimation using networked local agents with sensing and communication abilities, has become a popular research area in recent years due to its wide range of applications such as target tracking, region monitoring and area surveillance. Specifically, we consider the scenario where the local agents take local measurements and communicate with only their nearby neighbors to estimate the state of interest in a cooperative and fully distributed manner. A distributed hybrid information fusion algorithm is proposed in the scenario where the process model of the target and the sensing models of the local agents are linear and time varying. The proposed distributed hybrid information fusion algorithm is shown to be fully distributed and hence scalable, to be run in an automated manner and hence adaptive to locally unknown changes in the network, to have agents communicate for only once during each sampling time interval and hence inexpensive in communication, and to be able to track the interested state with uniformly upper bounded estimate error covariance. It is also explored very mild conditions on general directed time-varying graphs and joint network observability/detectability to guarantee the stochastic stability of the proposed algorithm.

In this talk by "using delays" I understand either Time-Delay Approaches to control problems (that originally may be free of delays) or intentional inserting delays to the feedback. I will start with an old Time-Delay approach - to sampled-data control. In application to network-based control with communication constraints, this is the only approach that allows treating transmission delays larger than the sampling intervals. I will continue with "using artificial delays" via simple Lyapunov functionals that lead to feasible LMIs for small delays and to simple sampled-data implementation.

Finally I will present a New Time-Delay approach - this time to Averaging. The existing results on averaging (that have been developed for about 60 years starting from the works of Bogoliubov and Mitropolsky) are qualitative: the original system is stable for small enough values of the parameter if the averaged system is stable. Our approach provides the first quantitative bounds on the small parameter making averaging-based control (including vibrational and extremum seeking control) reliable.

Many important plants (e.g. flexible manipulators or heat transfer processes) are governed by partial differential equations (PDEs) and are often described by models with a significant degree of uncertainty. Some PDEs may not be robust with respect to arbitrary small time-delays in the feedback. Robust finite-dimensional controller design for PDEs is a challenging problem.

In this talk two constructive methods for finite-dimensional control will be presented:

1. Spatial decomposition (or sampling in space) method, where the spatial domain is divided into N subdomains with N sensors and actuators located in each subdomain;
2. Modal decomposition method, where the controller is designed on the basis of a finite-dimensional system that captures the dominant dynamics of the infinite-dimensional one. Sufficient conditions ensuring the stability and performance of the closed-loop system are established in terms of simple linear matrix inequalities that are always feasible for appropriate choice of controllers.

We will discuss delayed and sampled-data implementations as well as application to network-based deployment of multi-agents.

One of the central societal challenges is to prolong independent living for elderly and promote well. Personalized robotic rehabilitation and assistance is considered one of the enabling technologies with control design playing a significant role. Focusing on sensorimotor rehabilitation and assistance, personalized control should be able to adapt to the high inter-personal variability in human motor behavior but also to intra-personal changes over time. Control adaptation is further challenged by the sparsity of person-specific data because calibration routines need to be brief for user acceptance. Above all, guaranteed safety is one of the key requirements.

In this talk we will present recent results on learning-based control with performance and safety guarantees for highly uncertain systems with particular focus on challenges arising from personalized rehabilitation and assistance. In order to achieve high sample efficiency as well as transparency of the system, available knowledge of dynamic models will be exploited and and augmented by Bayesian non-parametric model components. Epistemic uncertainty due to limited training data will explicitly be taken into account in the control design in order to achieve uncertainty-aware behavior of the closed loop system.

Online learning as well as realtime capabilities are further important aspects discussed in this talk. The results will be demonstrated in user intention-driven shared control designs for upper limb rehabilitation and assistance with exoskeletons.

Achieving a high level of autonomy of robots operating in extreme environments is particularly desirable but also particularly challenging due to uncertain and potentially varying operating conditions. By extreme environments we mean remote or hardly accessible environments where robots need to rely largely on local limited resources for their control implementation as for example underwater robots for collecting litter in marine environments. In this scenario the strong influence of nonlinear hydrodynamics on the motion of underwater robots and (often unpredictable) influences like currents as well as the distorted perception of the environment pose significant challenges for precise control and safe operation.

Recent progress in machine learning for control promises high performance in such uncertain conditions, yet many of the available approaches cannot directly be applied due to the limited available resources in terms of local computational power and communication. Hence apart from the challenge of providing safety and performance guarantees for learning control, also the efficient implementation plays an important role.

In this talk we will present results on learning-based control with performance guarantees for nonlinear systems in uncertain environment and under resource constraints on the example of an underwater robotic system with manipulation capabilities. We will introduce approaches to evaluate data-efficiency in non-parametric modeling techniques and show that the control task matters in this respect. The promises of physics-informed learning techniques to improve learning performance in terms of data efficiency and under noisy training conditions will be discussed.

Furthermore, different approaches to achieve real-time performance of non-parametric machine learning techniques given limited resources will be presented. While the proposed approaches promise to bring us a step further towards implementable high performance control for robots in extreme environments we will also discuss the remaining challenges as well as their limits.

Online learning in closed loop control systems is very attractive because it allows the automated identification of highly nonlinear dynamical systems as well as a fast adaptation to dynamically changing environments. Yet, depending on the application the data collection and the training of models is costly if not even prohibitive. On the one hand, the training is computationally expensive and might compromise real-time performance.

In particular in non-parametric learning approaches as e.g. in Gaussian Processes, the computational tractability is tied to the number of training data. As such it is important to understand how informative training samples are and further how to improve algorithmic efficiency of training and prediction.

In this talk we will demonstrate that the control task in addition to the underlying system dynamics has a strong influence on the required sample complexity. Employing Bayesian principles, we explore methods to quantify epistemic uncertainty with respect to control objectives and how they can be exploited to achieve a high sample efficiency for learning in the closed loop system. Additionally, approaches for efficient non-parametric online learning algorithms are proposed to allow the application of the presented methods under real-time constraints.

Stochastic approximation algorithms are used to approximate solutions to fixed point equations that involve expectations of functions with respect to possibly unknown distributions. Reinforcement learning algorithms such as TD- and Q-learning are two of its most famous applications.

This talk provides an overview of stochastic approximation, with focus on optimizing the rate of convergence. Based on this general theory, the well known slow convergence of Q-learning is explained: the variance of the algorithm is typically infinite.   New algorithms with provably fast (even optimal) convergence rate have been developed in recent years:   stochastic Newton-Raphson,  Zap SNR,  and acceleration techniques inspired by Polyak and Nesterov will be discussed (as time permits, and depending on the interests of the audience).

This work concerns design of control systems for "Demand Dispatch" to obtain ancillary services to the power grid by harnessing inherent flexibility in many loads.  With careful design, the grid operator can harness this flexibility to regulate supply-demand balance.  The deviation in aggregate power consumption can be controlled just as generators provide ancillary service today.  

Distributed control techniques are called for, much like those used today to provide congestion control in communication networks.  The main message is that intelligence should be concentrated as much as possible at the load.  In this way it is possible to design local control loops so that the aggregate of loads appears as a passive input-output system, while strict QoS constraints are maintained for each load.