Merge pull request #359 from singularity-energy/development

v0.4.0

CITATION.cff

@@ -22,6 +22,6 @@ authors:
 identifiers:
   - type: doi
     value: 'https://zenodo.org/doi/10.5281/zenodo.7062459'
-version: 0.3.3
+version: 0.4.0
 license: MIT
-date-released: '2024-02-27'
+date-released: '2024-04-04'

Pipfile

@@ -6,6 +6,7 @@ name = "pypi"
 [packages]
 cvxopt = "*"
 cvxpy = "*"
+dask = "< 2024.3.0"
 osqp = "*"
 ipykernel = "*"
 notebook = "*"

README.md

@@ -68,18 +68,21 @@ There are many ways that you can contribute!
 
 ## Repository Structure
 ### Modules
+- `anomaly_screening`: classes use to flag timeseries for anomalies as proposed in Tyler H. Ruggles et al. Developing reliable hourly electricity demand data through screening and imputation (2020)
 - `column_checks`: functions that check that all data outputs have the correct column names
 - `constants`: specifies conversion factors and constants used across all modules
 - `data_pipeline`: main script for running the data pipeline from start to finish
 - `download_data`: functions that download data from the internet
 - `data_cleaning`: functions that clean loaded data
 - `eia930`: functions for cleaning and formatting EIA-930 data
 - `emissions`: functions used for imputing emissions data
-- `filepaths`: Used to identify where repository files are located on the user's computer
-- `gross_to_net_generation`: Functions for identifying subplants and gross to net generation conversion factors
+- `filepaths`: used to identify where repository files are located on the user's computer
+- `gross_to_net_generation`: functions for identifying subplants and gross to net generation conversion factors
+- `helpers`: functions that are used across modules
 - `impute_hourly_profiles`: functions related to assigning an hourly profile to monthly data
 - `load_data`: functions for loading data from downloaded files
 - `output_data`: functions for writing intermediate and final data to csvs
+- `subplant_identification`: functions for identifying subplant IDs
 - `validation`: functions for testing and validating data outputs
 - `visualization`: functions for visualizing data in notebooks
 

docs/docs/Methodology/Data Aggregation/Subplant Aggregation.md

@@ -29,4 +29,4 @@ At a high level, we update and expand the `subplant_id` mapping based on the fol
 4. All of the new unique ids are renumbered in consecutive ascending order
 
 
-> Note: Subplant_id are not meant to be stable identifiers across multiple years. A generator that is assigned subplant_id 0 in one year could be assinged a different subplant_id in another year.
+> Note: `subplant_id` should be stable across all data years within a single version of OGE. However, `subplant_id` may change from one version of OGE to the next as we get better data about subplant mappings. 

docs/docs/Overview/Using the Data.md

@@ -6,13 +6,17 @@ stoplight-id: using_the_data
 
 ## Data Availability
 
-The latest release includes data for year 2019-2020 covering the contiguous United States, Alaska, and Hawaii. In future releases, we plan to expand the geographic coverage to additional U.S. territories (track progress [here](https://github.com/singularity-energy/open-grid-emissions/issues/79)), and to expand the historical coverage of the data.
+The latest release includes data for year 2019-2022 covering the contiguous United States, Alaska, and Hawaii. In future releases, we plan to expand the geographic coverage to additional U.S. territories (track progress [here](https://github.com/singularity-energy/open-grid-emissions/issues/79)), and to expand the historical coverage of the data.
 
-This initial release only includes 2019 and 2020 data because as of August 2022, 2020 is the most recent year of complete input data available, and 2019 is the earliest complete year for which EIA-930 data is available (which is used to assign an hourly profile to non-CEMS data). You can track progress on integrating pre-2019 data [here](https://github.com/singularity-energy/open-grid-emissions/issues/117)
+Currently, 2019 is the earliest available year because it is the first year for which complete EIA-930 data is available (which is used to assign an hourly profile to non-CEMS data). You can track progress on integrating pre-2019 data [here](https://github.com/singularity-energy/open-grid-emissions/issues/117)
 
 ## Data Release Schedule
 
-Parts of the input data used for the Open Grid Emissions dataset is released by the U.S. Energy Information Administration in the Autumn following the end of each year (2021 data should be available Autumn 2022). Each release will include the most recent year of available data as well as updates of all previous available years based on any updates to the OGE methodology. All previous versions of the data will be archived on Zenodo.
+In general, annual data releases will be published in Q4 of the following year (i.e. 2023 data should be published in Q4 of 2024)
+
+Parts of the input data used for the Open Grid Emissions dataset is released by the U.S. Energy Information Administration in the Autumn following the end of each year (2022 data was published in September 2023). Each release will include the most recent year of available data as well as updates of all previous available years based on any updates to the OGE methodology. All previous versions of the data will be archived on Zenodo.
+
+Updated datasets will also be published whenever a new minor version of the open-grid-emissions code is released, usually representing methdological enhancements. These releases may happen mutliple times throughout a year.
 
 ## Files