We present differentiable predictive control (DPC), a method for offline learning of constrained neural control policies for nonlinear dynamical systems with performance guarantees. We show that the sensitivities of the parametric optimal control problem can be used to obtain direct policy gradients. Specifically, we employ automatic differentiation (AD) to efficiently compute the sensitivities of the model predictive control (MPC) objective function and constraints penalties. To guarantee safety upon deployment, we derive probabilistic guarantees on closed-loop stability and constraint satisfaction based on indicator functions and Hoeffding’s inequality. We empirically demonstrate that the proposed method can learn neural control policies for various parametric optimal control tasks. In particular, we show that the proposed DPC method can stabilize systems with unstable dynamics, track time-varying references, and satisfy nonlinear state and input constraints. Our DPC method has practical time savings compared to alternative approaches for fast and memory-efficient controller design. Specifically, DPC does not depend on a supervisory controller as opposed to approximate MPC based on imitation learning. We demonstrate that, without losing performance, DPC is scalable with greatly reduced demands on memory and computation compared to implicit and explicit MPC while being more sample efficient than model-free reinforcement learning (RL) algorithms.
Physics-constrained graph modeling for building thermal dynamics
Ziyao Yang, Amol D. Gaidhane, Ján Drgoňa, and 4 more authors
In this paper, we propose a graph model embedded with compact physical equations for modeling the thermal dynamics of buildings. The principles of heat flow across various components in the building, such as walls and doors, fit the message-passing strategy used by Graph Neural networks (GNNs). The proposed method is to represent the multi-zone building as a graph, in which only zones are considered as nodes, and any heat flow between zones is modeled as an edge based on prior knowledge of the building structure. Furthermore, the thermal dynamics of these components are described by compact models in the graph. GNNs are further employed to train model parameters from collected data. During model training, our proposed method enforces physical constraints (e.g., zone sizes and connections) on model parameters and propagates the penalty in the loss function of GNN. Such constraints are essential to ensure model robustness and interpretability. We evaluate the effectiveness of the proposed modeling approach on a realistic dataset with multiple zones. The results demonstrate a satisfactory accuracy in the prediction of multi-zone temperature. Moreover, we illustrate that the new model can reliably learn hidden physical parameters with incomplete data.
Finding MIDDLE Ground: Scalable and Secure Distributed Learning
Marco Bornstein, Nawaf Nazir, Jan Drgona, and 2 more authors
In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, 2024
Edge-computing methods allow devices to efficiently train a high-performing, robust, and personalized model for predictive tasks. However, these methods succumb to privacy and scalability concerns such as adversarial data recovery and expensive model communication. Furthermore, edge computing methods unrealistically assume that all devices train an identical model. In practice, edge devices have varying computational and memory constraints, which may not allow certain devices to have the space or speed to train a specific model. To overcome these issues, we propose MIDDLE, a model-independent distributed learning algorithm that allows heterogeneous edge devices to assist each other in training while communicating only non-sensitive information. MIDDLE unlocks the ability for edge devices, regardless of computational or memory constraints, to assist each other even with completely different model architectures. Furthermore, MIDDLE does not require model or gradient communication, significantly reducing communication size and time. We prove that MIDDLE attains the optimal convergence rate of stochastic gradient descent for convex and non-convex smooth optimization. Finally, our experimental results demonstrate that MIDDLE attains robust and high-performing models without model or gradient communication.
Modeling dynamical systems is crucial for a wide range of tasks, but it remains challenging due to complex nonlinear dynamics, limited observations, or lack of prior knowledge. Recently, data-driven approaches such as Neural Ordinary Differential Equations (NODE) have shown promising results by leveraging the expressive power of neural networks to model unknown dynamics. However, these approaches often suffer from limited labeled training data, leading to poor generalization and suboptimal predictions. On the other hand, semi-supervised algorithms can utilize abundant unlabeled data and have demonstrated good performance in classification and regression tasks. We propose TS-NODE, the first semi-supervised approach to modeling dynamical systems with NODE. TS-NODE explores cheaply generated synthetic pseudo rollouts to broaden exploration in the state space and to tackle the challenges brought by lack of ground-truth system data under a teacher-student model. TS-NODE employs an unified optimization framework that corrects the teacher model based on the student’s feedback while mitigating the potential false system dynamics present in pseudo rollouts. TS-NODE demonstrates significant performance improvements over a baseline Neural ODE model on multiple dynamical system modeling tasks.
Extreme Risk Mitigation in Reinforcement Learning using Extreme Value Theory
Karthik Somayaji NS, Yu Wang, Malachi Schram, and 4 more authors
Transactions on Machine Learning Research, Mar 2024
Risk-sensitive reinforcement learning (RL) has garnered significant attention in recent years due to the growing interest in deploying RL agents in real-world scenarios. A critical aspect of risk awareness involves modelling highly rare risk events (rewards) that could potentially lead to catastrophic outcomes. These infrequent occurrences present a formidable challenge for data-driven methods aiming to capture such risky events accurately. While risk-aware RL techniques do exist, they suffer from high variance estimation due to the inherent data scarcity. Our work proposes to enhance the resilience of RL agents when faced with very rare and risky events by focusing on refining the predictions of the extreme values predicted by the state-action value distribution. To achieve this, we formulate the extreme values of the state-action value function distribution as parameterized distributions, drawing inspiration from the principles of extreme value theory (EVT). We propose an extreme value theory based actor-critic approach, namely, Extreme Valued Actor-Critic (EVAC) which effectively addresses the issue of infrequent occurrence by leveraging EVT-based parameterization. Importantly, we theoretically demonstrate the advantages of employing these parameterized distributions in contrast to other risk-averse algorithms. Our evaluations show that the proposed method outperforms other risk averse RL algorithms on a diverse range of benchmark tasks, each encompassing distinct risk scenarios.
Machine Learning for Smart and Energy-Efficient Buildings
Hari Prasanna Das, Yu-Wen Lin, Utkarsha Agwan, and 7 more authors
Energy consumption in buildings, both residential and commercial, accounts for approximately 40% of all energy usage in the United States, and similar numbers are being reported from countries around the world. This significant amount of energy is used to maintain a comfortable, secure, and productive environment for the occupants. So, it is crucial that energy consumption in buildings must be optimized, all the while maintaining satisfactory levels of occupant comfort, health, and safety. Machine learning (ML) has been proven to be an invaluable tool in deriving important insights from data and optimizing various systems. In this work, we review some of the most promising ways in which ML has been leveraged to make buildings smart and energy-efficient. For the convenience of readers, we provide a brief introduction to the relevant ML paradigms and the components and functioning of each smart building system we cover. Finally, we discuss the challenges faced while implementing machine learning algorithms in smart buildings and provide future avenues for research in this field.
2023
Neuro-physical dynamic load modeling using differentiable parametric optimization
Shrirang Abhyankar, Ján Drgoňa, Aaron Tuor, and 1 more author
In 2023 IEEE Power & Energy Society General Meeting (PESGM), Mar 2023
Normalizing flows (NF) build upon invertible neural networks and have wide applications in probabilistic modeling. Currently, building a powerful yet computationally efficient flow model relies on empirical fine-tuning over a large design space. While introducing neural architecture search (NAS) to NF is desirable, the invertibility constraint of NF brings new challenges to existing NAS methods whose application is limited to unstructured neural networks. Developing efficient NAS methods specifically for NF remains an open problem. We present AutoNF, the first automated NF architectural optimization framework. First, we present a new mixture distribution formulation that allows efficient differentiable architecture search of flow models without violating the invertibility constraint. Second, under the new formulation, we convert the original NP-hard combinatorial NF architectural optimization problem to an unconstrained continuous relaxation admitting the discrete optimal architectural solution, circumventing the loss of optimality due to binarization in architectural optimization. We evaluate AutoNF with various density estimation datasets and show its superior performance-cost trade-offs over a set of existing hand-crafted baselines.
ACC
Physics-Informed Machine Learning for Modeling and Control of Dynamical Systems
Truong X. Nghiem, Ján Drgoňa, Colin Jones, and 10 more authors
In 2023 American Control Conference (ACC), Jun 2023
As buildings account for approximately 40% of global energy consumption and associated greenhouse gas emissions, their role in decarbonizing the power grid is crucial. The increased integration of variable energy sources, such as renewables, introduces uncertainties and unprecedented flexibilities, necessitating buildings to adapt their energy demand to enhance grid resiliency. Consequently, buildings must transition from passive energy consumers to active grid assets, providing demand flexibility and energy elasticity while maintaining occupant comfort and health. This fundamental shift demands advanced optimal control methods to manage escalating energy demand and avert power outages. Reinforcement learning (RL) emerges as a promising method to address these challenges. In this paper, we explore ten questions related to the application of RL in buildings, specifically targeting flexible energy management. We consider the growing availability of data, advancements in machine learning algorithms, open-source tools, and the practical deployment aspects associated with software and hardware requirements. Our objective is to deliver a comprehensive introduction to RL, present an overview of existing research and accomplishments, underscore the challenges and opportunities, and propose potential future research directions to expedite the adoption of RL for building energy management.
ACM e-Energy
Power Grid Behavioral Patterns and Risks of Generalization in Applied Machine Learning
Shimiao Li, Ján Drgoňa, Shrirang Abhyankar, and 1 more author
In Companion Proceedings of the 14th ACM International Conference on Future Energy Systems, Jun 2023
Recent years have seen a rich literature of data-driven approaches designed for power grid applications. However, insufficient consideration of domain knowledge can impose a high risk to the practicality of the methods. Specifically, ignoring the grid-specific spatiotemporal patterns (in load, generation, and topology, etc.) can lead to outputting infeasible, unrealizable, or completely meaningless predictions on new inputs. To address this concern, this paper investigates real-world operational data to provide insights into power grid behavioral patterns, including the time-varying topology, load, and generation, as well as the spatial differences (in peak hours, diverse styles) between individual loads and generations. Then based on these observations, we evaluate the generalization risks in some existing ML works caused by ignoring these grid-specific patterns in model design and training.
Constructing Neural Network Based Models for Simulating Dynamical Systems
Christian Legaard, Thomas Schranz, Gerald Schweiger, and 6 more authors
Dynamical systems see widespread use in natural sciences like physics, biology, and chemistry, as well as engineering disciplines such as circuit analysis, computational fluid dynamics, and control. For simple systems, the differential equations governing the dynamics can be derived by applying fundamental physical laws. However, for more complex systems, this approach becomes exceedingly difficult. Data-driven modeling is an alternative paradigm that seeks to learn an approximation of the dynamics of a system using observations of the true system. In recent years, there has been an increased interest in applying data-driven modeling techniques to solve a wide range of problems in physics and engineering. This article provides a survey of the different ways to construct models of dynamical systems using neural networks. In addition to the basic overview, we review the related literature and outline the most significant challenges from numerical simulations that this modeling paradigm must overcome. Based on the reviewed literature and identified challenges, we provide a discussion on promising research areas.
Structural inference of networked dynamical systems with universal differential equations
J. Koch, Z. Chen, A. Tuor, and 2 more authors
Chaos: An Interdisciplinary Journal of Nonlinear Science, Feb 2023
Networked dynamical systems are common throughout science in engineering; e.g., biological networks, reaction networks, power systems, and the like. For many such systems, nonlinearity drives populations of identical (or near-identical) units to exhibit a wide range of nontrivial behaviors, such as the emergence of coherent structures (e.g., waves and patterns) or otherwise notable dynamics (e.g., synchrony and chaos). In this work, we seek to infer (i) the intrinsic physics of a base unit of a population, (ii) the underlying graphical structure shared between units, and (iii) the coupling physics of a given networked dynamical system given observations of nodal states. These tasks are formulated around the notion of the Universal Differential Equation, whereby unknown dynamical systems can be approximated with neural networks, mathematical terms known a priori (albeit with unknown parameterizations), or combinations of the two. We demonstrate the value of these inference tasks by investigating not only future state predictions but also the inference of system behavior on varied network topologies. The effectiveness and utility of these methods are shown with their application to canonical networked nonlinear coupled oscillators.
Domain-aware Control-oriented Neural Models for Autonomous Underwater Vehicles
Wenceslao Shaw Cortez, Soumya Vasisht, Aaron Tuor, and 3 more authors
IFAC-PapersOnLine, Feb 2023
12th IFAC Symposium on Nonlinear Control Systems NOLCOS 2022
Conventional physics-based modeling is a time-consuming bottleneck in control design for complex nonlinear systems like autonomous underwater vehicles (AUVs). In contrast, purely data-driven models require a large number of observations and lack operational guarantees for safety-critical systems. Data-driven models leveraging available partially characterized dynamics have potential to provide reliable systems models in a typical data-limited scenario for high value complex systems, thereby avoiding months of expensive expert modeling time. In this work we explore this middle-ground between expert-modeled and pure data-driven modeling. We present control-oriented parametric models with varying levels of domain-awareness that exploit known system structure and prior physics knowledge to create constrained deep neural dynamical system models. We employ universal differential equations to construct data-driven blackbox and graybox representations of the AUV dynamics. In addition, we explore a hybrid formulation that explicitly models the residual error related to imperfect graybox models. We compare the prediction performance of the learned models for different distributions of initial conditions and control inputs to assess their suitability for control.
Homotopy Learning of Parametric Solutions to Constrained Optimization Problems
Shimiao Li, Ján Drgoňa, Aaron R Tuor, and 2 more authors
We present differentiable predictive control (DPC) as a deep learning-based alternative to the explicit model predictive control (MPC) for unknown nonlinear systems. In the DPC framework, a neural state-space model is learned from time-series measurements of the system dynamics. The neural control policy is then optimized via stochastic gradient descent approach by differentiating the MPC loss function through the closed-loop system dynamics model. The proposed DPC method learns model-based control policies with state and input constraints, while supporting time-varying references and constraints. In embedded implementation using a Raspberry-Pi platform, we experimentally demonstrate that it is possible to train constrained control policies purely based on the measurements of the unknown nonlinear system. We compare the control performance of the DPC method against explicit MPC and report efficiency gains in online computational demands, memory requirements, policy complexity, and construction time. In particular, we show that our method scales linearly compared to exponential scalability of the explicit MPC solved via multiparametric programming.
Data-driven Stabilization of Discrete-time Control-affine Nonlinear Systems: A Koopman Operator Approach
Subhrajit Sinha, Sai Pushpak Nandanoori, Ján Drgona, and 1 more author
In 2022 European Control Conference (ECC), Aug 2022
The availability of computational power, and a wealth of data from sensors have boosted the development of model-based predictive control for smart and effective control of advanced buildings in the last decade. More recently occupant-behavior models have been developed for including people in the building control loops. However, while important objectives of scientific research are reproducibility and replicability of results, not all information is available from published documents. Therefore, the aim of this paper is to propose a guideline for a thorough and standardized occupant-behavior model documentation. For that purpose, the literature screening for the existing occupant behavior models in building control was conducted, and the occupant behavior modeling processes were studied to extract practices and gaps for each of the following phases: problem statement, data collection, and preprocessing, model development, model evaluation, and model implementation. The literature screening pointed out that the current state-of-the-art on model documentation shows little unification, which poses a particular burden for the model application and replication in field studies. In addition to the standardized model documentation, this work presented a model-evaluation schema that enabled benchmarking of different models in field settings as well as the recommendations on how OB models are integrated with the building system.
Dissipative Deep Neural Dynamical Systems
Ján Drgoňa, Aaron Tuor, Soumya Vasisht, and 1 more author
The problem of synthesizing stochastic explicit model predictive control policies is known to be quickly intractable even for systems of modest complexity when using classical control-theoretic methods. To address this challenge, we present a scalable alternative called stochastic parametric differentiable predictive control (SP-DPC) for unsupervised learning of neural control policies governing stochastic linear systems subject to nonlinear chance constraints. SP-DPC is formulated as a deterministic approximation to the stochastic parametric constrained optimal control problem. This formulation allows us to directly compute the policy gradients via automatic differentiation of the problem’s value function, evaluated over sampled parameters and uncertainties. In particular, the computed expectation of the SP-DPC problem’s value function is back propagated through the closed-loop system rollouts parametrized by a known nominal system dynamics model and neural control policy which allows for direct model-based policy optimization. We demonstrate the computational efficiency and scalability of the proposed policy optimization algorithm in three numerical examples, including systems with a large number of states or subject to nonlinear constraints.
Learning Constrained Adaptive Differentiable Predictive Control Policies With Guarantees
Deciding on a suitable algorithm for energy demand prediction in a building is non-trivial and depends on the availability of data. In this paper we compare four machine learning models, commonly found in the literature, in terms of their generalization performance and in terms of how using different sets of input features affects accuracy. This is tested on a data set where consumption patterns differ significantly between training and evaluation because of the Covid-19 pandemic. We provide a hands-on guide and supply a Python framework for building operators to adapt and use in their applications.
Physics-constrained deep learning of multi-zone building thermal dynamics
Ján Drgoňa, Aaron R. Tuor, Vikas Chandan, and 1 more author
We present a physics-constrained deep learning method to develop control-oriented models of building thermal dynamics. The proposed method uses systematic encoding of physics-based prior knowledge into a structured recurrent neural architecture. Specifically, our method incorporates structural prior knowledge from traditional physics-based building modeling into the architecture of the deep neural network model. Further, we also use penalty methods to provide inequality constraints, thereby bounding predictions within physically realistic and safe operating ranges. We observe that stable eigenvalues accurately characterize the dissipativeness of the system, and use a constrained matrix parameterization based on the Perron-Frobenius theorem to bound the dominant eigenvalues of the building thermal model parameter matrices. We demonstrate the effectiveness and physical interpretability of the proposed data-driven modeling approach on a real-world dataset obtained from an office building with 20 thermal zones. The proposed data-driven method can learn interpretable dynamical models that achieve high accuracy and generalization over long-term prediction horizons. We show that using only 10 days’ measurements for training, our method is capable of generalizing over 20 consecutive days. We demonstrate that the proposed modeling methodology is achieving state-of-the-art performance by significantly improving the accuracy and generalization compared to classical system identification methods and prior advanced methods reported in the literature. compared to prior state-of-the-art methods reported in the literature.
L4DC
Automating Discovery of Physics-Informed Neural State Space Models via Learning and Evolution
Elliott Skomski, Ján Drgoňa, and Aaron Tuor
In Proceedings of the 3rd Conference on Learning for Dynamics and Control, 07 – 08 june 2021