ClimAIte Control
Year 2023
Author(s) Sebastian Stripp
Link https://repository.tudelft.nl/islandora/object/uuid:963a53d8-2dc0-4d2a-988d-612c72856e0d
ML Tags
Q-Learning
Neural Networks
Topic Tags
Operational Energy / Building Energy Control
Digital Twins
> Software & Plug-ins Used
SketchUp and Rhino for 3D Modelling
Grasshopper with plug-ins Honeybee and Ladybug for Visual Programming
Energy Plus for Energy Modelling
Python using Pytorch for Deep Q-Learning
> Workflow
The thesis investigates if it is possible to reduce the energy use of existing buildings by optimizing their operation through deep reinforcement learning (DRL). A reinforcement learning model is trained by having an agent take actions in a simulated environment (digital twin) with select decisions (opening ventilation louvres, setting the thermostat), weather forecast information and building use information (if the building is occupied or not). The goal of the model is to reduce energy usage and improve occupant comfort. The images below show parts of this process.
Thesis Report Figure 5.1: Diagram of agent and environment state, action and reward cycle
Thesis Report Figure 5.3: Diagram of agent and EnergyPlus interaction in detail
Thesis Report Figure 5.4: Diagram of the deep recurrent Q neural network of the agent
> Case study
The workflow is tested on a case study project. The building used is a 2-story timber-framed school building. It is a Passivhaus-certified project and serves as a primary school.
Thesis Report Figure 3.1: Case study school exterior view, by Lowfield Timber Frames
> Summary
The thesis project uses a digital twin of a case study building to test how Q-Learning (a form of reinforcement learning) can be used to make building control decisions to reduce energy consumption.
To simulate building performance, it was necessary to create a digital twin. First, the building was modelled in Rhino and divided into two zones. The surfaces were then incorporated into Ladybug and Honeybee to provide the building energy model. This model was calibrated against historical data measured on-site and ASHRAE guidelines were used to validate the digital twin.
In order to investigate saving energy while maintaining comfort, Reinforcement Learning agents were designed, coded, and tested together with 9 building variations of the digital twin. The main method of Reinforcement Learning used is Deep Q-Learning (DQN) which approximates the reward of different actions and chooses the one with the highest reward. To take the action with the highest reward, a fully-connected simple Neural Network (NN) is trained to understand the relationships between actions and expected rewards. Additionally, at each step the time-based data from the weather forecast are processed using a recurrent Neural Networks (RNN). Combining these two methods results in a Deep Recurrent Q-Learning (DRQN) model.
LIMITATIONS: Only a few different RL algorithm types were evaluated in a small number of combinations.
> Possible Applications
For ideas on how to implement some of the above mentioned techniques, please see “Possible applications for students to try with Reinforcement Learning”