RL-BCA (Work In Progress...)

Please redirect yourself to RL-EmsPy https://github.com/mechyai/RL-EmsPy, which is the updated and better maintained version of this. I was having trouble with version control on this repo so I have simply started a new, cleaner, more organized one as RL-EmsPy.

This repository is for Reinforcement Learning (RL) algorithm development and testing of BCAs (Building Control Agent) in EnergyPlus (E+) building energy simulator using Python Energy Management System (EMS) API with a meta-class wrapper, EmsPy.

This repo was constructed by someone with little experience with EnergyPlus and software/programming, but wanted to assist in creating an easily interfacable RL 'environment' for intelligent HVAC control research. Any feedback/criticism or improvements to the repo is appreciated.

The meta-class wrapper, EmsPy, is meant to simplify and somewhat constrain the E+ EMS API. The popular/intended use of EMS is to interface with a running E+ building simulation, not so easily done otherwise. Recently, a Python API was created for EMS so users aren't constrained to using the E+ Runtime Language (ERL) and can more readily interact with a running building simulation to gather state information and implement custom control at each simulation timestep (subhourly). This API can be used to create Python plugins or use E+ as a library and run simulations from Python - EmsPy utilizies the latter.
EMS exposes E+ data such as variables, internal variables, meters, actuators, and weather. Please see the documentation hyperlinks below to learn more.

Although this repo is meant to facilitate in interfacing with E+, making this environment more accessible to AI and controls people, a good understanding of E+ and building modeling may still be necessary, especially if you intend to create, link, and control your own building models. Eventually, some standard building models and template scripts will be created so that user's can simply experiment with them through Python for control purposes with no E+ experience needed. This may also help standardize performance benchmarks.

Regardless of your use case, you will need to have the proper versioned E+ simulation engine downloaded onto your system https://energyplus.net/downloads.

Further Documentation:

• EnergyPlus
• EnergyPlus Documentation (including EMS Application Guide!)
• EnergyPlus EMS Python API 0.2 Documentation
• EnerrgyPlus EMS API Homepage
• OpenStudio SDK Documentation (for building model creation and simulation GUI)
• OpenStudio Coalition
• Unmet Hours Help Forum (community forum for EnergyPlus related help)

Dependencies:

• EnergyPlus 9.5 (building energy simulation engine)
• EnergyPlus EMS Python API 0.2 (included in E+ 9.5 download)
• Python 3
• pyenergyplus Python package (included in E+ download)
• openstudio Python package (not currently used, but plan to add functionality)

Usage Explanation:

The diagram below depicts the RL-interaction-loop within a simulation timestep at runtime. Because of the technicalities of the interaction between EMS and the simulator - mainly the use of callback function(s) and the multiple calling points available per timestep - the RL algorithm must be implemented in a very specific manner, which will be explained in detail below.

There are likely 4 main use-cases for this repo, if you are hoping to implement RL algorithms at runtime. In order of increasing complexity:

• You want to use an existing template and linked building model to purely implement RL control
• You have an existing E+ building model (with no model or .idf modification needed) that you want to link and implement RL control on
• You have an existing E+ building model (with some amount of model or .idf modification needed) that you want to link and implement RL control on
• You want to create a new E+ building model to integrate and implement RL control on (another project in itself)

EmsPy usage for these use-cases is all the same, the difference is what must be done beforehand. Creating building models, understanding their file makeup, modifying .idf files, and adding/linking EMS variables and actuators brings extra challenges. This guide will focus on utilizing EmsPy (EMS API meta-class wrapper) and the latter components needed, before EmsPy, will be discussed briefly at the end, with basic guidance to get you started in the right direction.

At the least, even if solely using EmsPy for a given model, it is important to understand the EMS metrics of a given model: variables, internal variables, meters, actuators, and weather. These are used to build the state and action space of your control framework. See the EMS Application Guide and Input Output Reference documents for detailed information on these elements https://energyplus.net/documentation.

How to use EmsPy with an E+ Model:

This guide follows the design of the template Python scripts provided. The integration of the control (RL) algorithm and the flow of the calling points and callback functions at runtime is depicted in the image above. The image below loosely represents the logic of EmsPy and its usage.

1. First, you will create an EmsPy object from proper inputs (this acts as your building simulation environment/agent). The inputs include paths to the E+ directory and the building model file to be simulated, information about desired EMS metrics, simulation timestep, and actuation functions with calling points:

agent = emspy.BcaEnv(ep_path: str, ep_idf_to_run: str, timesteps: int, tc_var: dict, tc_intvar: dict, tc_meter: dict, tc_actuator: dict, tc_weather: dict)

• ep_path sets the path to your EnergyPlus 9.5 installation directory
• ep_idf_to_run sets the path to your EnergyPlus building model, likely .idf file
• timesteps the number of timesteps per hour of the simulation. This must match the timestep detailed in your model .idf
• define all EMS metrics you want to call or interact with in your model:
  • Build the Table of Contents (ToC) dictionaries for EMS variables, internal variables, meters, actuators, and weather
  • Note: this requires an understanding of EnergyPlus model input and output files, especially for actuators
  • Each EMS category ToC should be a dictionary of each EMS metric attribute name (key) its required arguments (value) for fetching the 'handle' or data from the model. See Data Transfer API documentation for more info
    • Variables: 'user_var_name': ['variable_name', 'variable_key'] elements of tc_vars dict
    • Internal Variables: 'user_intvar_name': ['variable_type', 'variable_key'] elements of tc_intvars dict
    • Meters: 'user_meter_name': 'meter_name' element of tc_meter dict
    • Actuators: 'user_actuator_name': ['component_type', 'control_type', 'actuator_key'] elements of tc_actuator dict
    • Weathers: 'user_weather_name': 'weather_name' elements of tc_weather dict

Once this has been completed the meta-class, EmsPy, has all it needs to build out your basic class - implementing various data collection/organization and dataframes attributes, as well as finding the EMS handles from the ToCs, etc. It may be helpful to run this 'agent/environment' object initialization and then review its contents to see all that the meta-class has created.

Note: At this point, the simulation can be ran but nothing useful will happen (in terms of control or data collection) as no calling points, callback functions, or actuation functions have been defined and linked.

2. Next, you must define the Calling Point & Actuation Function dictionary to define and enable callback functionality at runtime. This dictionary links a calling point(s) to a callback function(s) and the arguments related to data/actuation update frequencies. A given calling point defines when a linked callback function (and optionally an embedded actuation function) will be ran during the simulation timestep calculations. The diagram above represents the simulation flow and RL integration with calling points and callback functions.

The Calling Point & Actuation Function dictionary should be built one key-value at a time using the method:

TODO update new method arguments

BcaEnv.set_calling_point_and_callback_function(
   calling_point: str, actuation_fxn, update_state: bool, update_state_freq: int = 1, update_act_freq: int = 1)

• calling_point a single calling point from the available list EmsPy.available_calling_points
• actuation_fxn the control algorithm function (one of potentially many throughout a timestep), which must take no argument and return a dictionary (or None if no custom actuation) of actuator name(s) (key) and floating point setpoint value(s) (value) to be implemented at the linked calling point. Be sure to pass the function, not its result.
  • Note: due to the scope and passing of the callback function, please use a custom class and instantiate a global object in your script to encapsulate any custom data for the control algorithm (RL agent parameters) and then utilize the global object in your actuation function. The callback functions can reference object/class data at runtime.
  • Warning: actual actuator setpoint values can be floating point, integer, and boolean values (or None to relinquish control back to E+) and have a variety of input domain spans. Since the API input must be floating point, the setpoint values will be automatically cast to nearest integer (1/2 rounds up) and all but ~1.0 casts to False, respective to the specific actuator's needs. Internal variables may be able to be used to understand an actuators input domain. You must have an understanding of the actuator(s) to function as intended.
• update_state T/F to whether or not the entire EMS ToCs should be updated for that calling point, this acts as a complete state update (use BcaEnv.update_ems_data for more selective udpates at specific calling points, if needed)
• update_state_freq the number of simulation timesteps in between each state update, default is every timestep
• update_act_freq set to the number of simulation timesteps in between each actuation function call, default is every timestep

Note: there are multiple calling points per timestep, each signifying the start/end of an event in the process. The majority of calling points occur consistently throughout the simulation, but several occur once before during simulation setup.

The user-defined actuation_function should encapsulate any sort of control algorithm (more than one can be created and linked to unique calling points, but it's likely that only 1 will be used as the entire RL algorithm). Using the 'agent/environment' object attributes, or better, the methods BcaEnv.get_ems_data and BcaEnv.get_weather_forecast, to collect state information, a control algorithm/function can be created and passed. Using a decorator function, this actuation function will automatically be attached to a base callback function and linked to the defined calling point. At that calling point during runtime, the actuation function will be ran and the returned actuator dict will be passed to the simulation to update actuator setpoint values. The rest of the arguments are also automatically passed to the base-callback function to dictate the update frequency of state data and actuation. This means that data collection or actuation updates do not need to happen every timestep.

BcaEnv.get_ems_data(ems_metric_list: list, time_rev_index: list=[0]) -> list

• This method will return an ordered nested list of ordered data points for then given EMS metrics and timing index(s), or entire EMS type ToC
• Its intended use is to return updated state information at each timestep, it can be called as many times as need in a actuation function.
• This method must be used during runtime from an actuation function
• ems_metric_list pass one or more EMS metrics (of any type) OR ONLY a single EMS type (var, intvar, meter, actuator, weather) in a list. Passing an EMS type will utilize and return data for that entire EMS ToC
• time_rev_index indicates the time index of the data you want to return, indexing backwards from the most recent timestep at 0. Leaving this list empty [ ] will return the entire data list collected thus far in the simulation for each given EMS metric. Note that data will only be returned once the number of simulation timesteps has surpassed the maximum prior-time index given

BcaEnv.get_weather_forecast(when: str, weather_metrics: list, hour: int, zone_ts: int) -> list

• This method is used to fetch and return an ordered list of future weather data, resembling weather forecasts. Weather events that have already occurred in simulation can be gathered using BcaEnv.get_ems_data
• This method must be used during runtime from an actuation function
• weather_metrics is the list of user-defined weather variable names, defined in the weather ToC, you want to fetch data for
• when either 'today' or 'tomorrow' dictates which day is in question, relative to current simulation time
• hour the hour of the day to collect the weather forecast data
• zone_ts the timestep within the given hour you want to collect weather forecast data for

Note : If you wish to use callback functions just for defualt data collection pass None as the actuation function. If you wish to use the callback functions for custom data collection and/or other actions other than any actuation/control at a specific calling point, implement an actuation function that returns None as the actuation dict.

Also, if there is a need to update specific EMS metrics at a certain calling point separately from the rest (all EMS ToCs), you can use the method below within an actuation function to update specific EMS metrics. However, this does not also exclude them from the state_update that updates ALL EMS metrics.

BcaEnv.update_ems_data(ems_metric_list: list, return_data: bool) -> list

• This method will update the given EMS metrics, or entire EMS type ToC and optionally return an ordered list of the updated data
• Its intended use is if you want to update specific EMS data (or types) at a unique calling point, separate from the default state update of all EMS ToCs at another calling point.
• This method must be used during runtime from an actuation function
• ems_metric_list pass one or more EMS metrics (of any type) OR ONLY a single EMS type (var, intvar, meter, actuator, weather) in a list. Passing an EMS type will utilize that entire EMS ToC
• return_data if True, this will automatically return the ordered list of data from Bca.Env.get_ems_data

Warning : EMS data (and actuation) can be 'updated' by the user (but not necessarily internally by the simulation) for each calling point (and actuation function) assigned within a single timestep. You likely want to avoid this and manually only implement one state update state_update=True per timestep. Otherwise, you will screw up zone timestep increments (with current software design) and may accidentally be collecting data and actuating multiple times per timestep. Just because you want to update data/actuation does not necessary mean it will be implmented at all or how you intended. An understanding of calling points and when to collect data or actuate is crucial - Please see the EMS Application Guide for more information on calling points.

TIPS:

• (in progress)

CAUTION:

• Make sure your hourly timestep matches that of your EnergyPlus .idf model
• EMS data (and actuation) can be 'updated' by the user (but not necessarily internally by the simulation) for each calling point (and actuation function) assigned within a single timestep. You likely want to avoid this and manually only implement one state update state_update=True per timestep. Otherwise, you will screw up zone timestep increments (with current software design) and may accidentally be collecting data and actuating multiple times per timestep. Just because you want to update data/actuation does not necessary mean it will be implmented at all or how you intended. An understanding of calling points and when to collect data or actuate is crucial - Please see the EMS Application Guide for more information on calling points.

Future Planned Functionality & Repo Improvements:

• EmsPy improvements
  • more automatic user oversight to verify that user's have not violated logical errors in calling points, callback functions, and/or EMS updates
  • verify that given model timestep matches the .idf file, OR have it overwrite the model if not (say using openstudio somehow)
  • assist users in understanding actuator input ranges
  • further detailed documentation
• Data Dashboard class to automatically compile E+ performance and RL learning data into subplots via Matplotlib
• Openstudio wrapper class to assist in simple modifications of the .idf/.osm that impact simulation experiments (timesteps, start-end dates, etc.)
• Provide tips to documentation on how to construct and/or modify building models to be linked with EmsPy
• A handful of various building models already set up with EmsPy so that user's can just focus on control algorithms given readily available state and action space, and pre-linked calling points.

Creating an E+ Building Energy Model:

• TOOD

Setting up a E+ Model for EMS API Usage:

• TODO

Linking EMS Metrics to Your EmsPy Script:

• TODO

References:

• (in progress)