Merge pull request #325 from singularity-energy/development

v0.3.0 with 2022 data

.github/workflows/lint.yml

@@ -0,0 +1,19 @@
+name: Lint
+
+on:
+  push:
+  pull_request:
+    branches:
+      - development
+      - main
+
+jobs:
+  formatting:
+    if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name != github.event.pull_request.base.repo.full_name
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
+      - run: python -m pip install --upgrade ruff
+      - run: ruff check . --diff
+      - run: ruff format . --diff

.gitignore

@@ -1,5 +1,7 @@
-data/*
-!data/manual/
+src/oge/__pycache__
+
+dist/*
+
 
 example/.ipynb_checkpoints/
 test/__pycache__/

CITATION.cff

@@ -12,16 +12,16 @@ authors:
     family-names: Pease
     orcid: 'https://orcid.org/0000-0003-3528-6048'
     affiliation: "Singularity Energy"
-  - given-names: Milo
-    family-names: Knowles
-    orcid: 'https://orcid.org/0000-0003-4052-5517'
+  - given-names: Benjamin
+    family-names: Rouille d'Orfeuil
     affiliation: "Singularity Energy"
   - given-names: Wenbo
     family-names: Shi
+    orcid: 'https://orcid.org/0000-0003-4889-0834'
     affiliation: "Singularity Energy"
 identifiers:
   - type: doi
-    value: 'https://doi.org/10.5281/zenodo.7495818'
-version: 0.2.0
+    value: 'https://doi.org/10.5281/zenodo.7692997'
+version: 0.3.0
 license: MIT
-date-released: '2022-12-30'
+date-released: '2023-12-29'

LICENSE.md

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2022 Singularity Energy
+Copyright (c) 2023 Singularity Energy
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

Pipfile

@@ -0,0 +1,31 @@
+[[source]]
+url = "https://pypi.org/simple"
+verify_ssl = true
+name = "pypi"
+
+[packages]
+cvxopt = "*"
+cvxpy = "*"
+osqp = "*"
+ipykernel = "*"
+notebook = "*"
+numpy = "*"
+openpyxl = "*"
+pandas = "*"
+plotly = "*"
+pyarrow = "*"
+qdldl = "*"
+requests = "*"
+seaborn = "*"
+sqlalchemy = "*"
+statsmodels = "*"
+coloredlogs = "*"
+"catalystcoop.pudl" = {git = "git+https://github.com/singularity-energy/pudl.git@oge_release"}
+gridemissions = {git = "git+https://github.com/singularity-energy/gridemissions"}
+
+[dev-packages]
+ruff = "*"
+pytest = "*"
+
+[requires]
+python_version = "3.11"

README.md

@@ -5,31 +5,46 @@
 
 The Open Grid Emissions Initiative seeks to fill a critical need for high-quality, publicly-accessible, hourly grid emissions data that can be used for GHG accounting, policymaking, academic research, and energy attribute certificate markets. The initiative includes this repository of open-source grid emissions data processing tools that use peer-reviewed, well-documented, and validated methodologies to create the accompanying public dataset of hourly, monthly, and annual U.S. electric grid generation, GHG, and air pollution data.
 
-Please check out [our documentation](https://docs.singularity.energy/docs/open-grid-emissions-docs) for more details about the Open Grid Emissions methodology.
+Please check out [our documentation](https://docs.singularity.energy/docs/open-grid-emissions) for more details about the Open Grid Emissions methodology.
 
 The Open Grid Emissions Dataset can be [downloaded here](https://singularity.energy/open-grid-emissions). An archive of previous versions of the dataset and intermediate data outputs (for research and validation purposes) can be found on [Zenodo](https://zenodo.org/communities/singularity-energy?page=1&size=20).
 
 ## Installing and running the data pipeline
+A couple of installation methods are available. First, navigate to the folder where you want to save the repository and run the following commands:
 
-To install and run the pipeline on your computer, open anaconda prompt, navigate to the folder where you want to save the repository, and run the following commands:
-
-```
+### If you are using conda
+```bash
 conda install git
 git clone https://github.com/singularity-energy/open-grid-emissions.git
 conda update conda
 cd open-grid-emissions
 conda env create -f environment.yml
 conda activate open_grid_emissions
+```
+
+### If you are using pipenv
+Note that this option requires to have Python and git installed on your machine.
+```bash
+pip install pipenv
+git clone https://github.com/singularity-energy/open-grid-emissions.git
+cd open-grid-emissions
+pipenv sync
+pipenv shell
+```
+
+The pipeline can be run as follows:
+```bash
 cd src
-python data_pipeline.py --year 2021
+python data_pipeline.py --year 2022
 ```
+independently of the installation method you chose.
 
-A more detailed walkthough of these steps can be found below in the "Development Setup" section.
+A more detailed walkthrough of these steps can be found below in the "Development Setup" section.
 
 ## Data Availability and Release Schedule
-The latest release includes data for year 2019-2021 covering the contiguous United States, Alaska, and Hawaii. In future releases, we plan to expand the geographic coverage to additional U.S. territories (dependent on data availability), and to expand the historical coverage of the data. 
+The latest release includes data for year 2019-2021 covering the contiguous United States, Alaska, and Hawaii. In future releases, we plan to expand the geographic coverage to additional U.S. territories (dependent on data availability), and to expand the historical coverage of the data.
 
-Parts of the input data used for the Open Grid Emissions dataset is released by the U.S. Energy Information Administration in the Autumn following the end of each year (2022 data should be available Autumn 2023). Each release will include the most recent year of available data as well as updates of all previous available years based on any updates to the OGEI methodology. All previous versions of the data will be archived on Zenodo.
+Parts of the input data used for the Open Grid Emissions dataset is released by the U.S. Energy Information Administration in the Autumn following the end of each year (2022 data was published in September 2023). Each release will include the most recent year of available data as well as updates of all previous available years based on any updates to the OGE methodology. All previous versions of the data will be archived on Zenodo.
 
 Updated datasets will also be published whenever a new version of the open-grid-emissions repository is released.
 
@@ -43,7 +58,7 @@ There are many ways that you can contribute!
  - Contribute your subject matter expertise to the discussion about [open issues and questions](https://github.com/singularity-energy/open-grid-emissions/issues?q=is%3Aissue+is%3Aopen+label%3Aquestion)
  - Submit a pull request to help us fix open issues
 
-# Repository Structure
+## Repository Structure
 ### Modules
 - `column_checks`: functions that check that all data outputs have the correct column names
 - `data_pipeline`: main script for running the data pipeline from start to finish
@@ -69,71 +84,95 @@ Notebooks are organized into five directories based on their purpose
 - `work_in_progress`: temporary notebooks being used for development purposes on specific branches
 
 ### Data Structure
-- `data/downloads` contains all files that are downloaded by functions in `load_data`
-- `data/manual` contains all manually-created files, including the egrid static tables
-- `data/outputs` contains intermediate outputs from the data pipeline... any files created by our code that are not final results
-- `data/results` contains all final output files that will be published
+All manual reference tables are stored in `src/oge/reference_tables`. 
 
-# Development Setup
+All files downloaded/created as part of the pipeline are stored in your HOME directory (e.g. users/user.name/):
+- `HOME/open_grid_emissions_data/downloads` contains all files that are downloaded by functions in `load_data`
+- `HOME/open_grid_emissions_data/outputs` contains intermediate outputs from the data pipeline... any files created by our code that are not final results
+- `HOME/open_grid_emissions_data/results` contains all final output files that will be published
 
+## Development Setup
 If you would like to run the code on your own computer and/or contribute updates to the code, the following steps can help get you started.
 
-## Users unfamiliar with git / python
-
-### Install conda and python
-
+### Setup with conda
+This installation is recommended if you are unfamiliar with git and Python.
+#### Install conda and python
 We suggest using miniconda or Anaconda to manage the packages needed to run the Open Grid Emissions code. Anaconda and Miniconda install a similar environment, but Anaconda installs more packages by default and Miniconda installs them as needed. These can be downloaded from [miniconda](https://docs.conda.io/en/latest/miniconda.html) or [Anaconda](https://www.anaconda.com/products/distribution)
 
-### Install a code editor
-
-If you want to edit the code and do not already have an integrated development environment (IDE) installed, one good option is Visual Studio Code (download: https://code.visualstudio.com/). 
-
-### Install and setup git software manager
-
-In order to download the repository, you will need to use git. You can either install Git Bash from https://git-scm.com/downloads, or you can install it using conda. To do so, fter installing Anaconda or Miniconda, open an Anaconda Command Prompt (Windows) or Terminal.app (Mac) and type the following command:
-
-```
+#### Install and setup git software manager
+In order to download the repository, you will need to use git. You can either install Git Bash from https://git-scm.com/downloads, or you can install it using conda. To do so, after installing Anaconda or Miniconda, open an Anaconda Command Prompt (Windows) or Terminal.app (Mac) and type the following command:
+```bash
 conda install git
 ```
-
 Then you will need set up git following these instructions: https://docs.github.com/en/get-started/quickstart/set-up-git
 
-## Once you have git and conda installed
-
-### Download the codebase to a local repository
-
+#### Download the codebase to a local repository
 Using Anaconda command prompt or Git Bash, use the `cd` and `mkdir` commands to create and/or enter the directory where you would like to download the code (e.g. "Users/myusername/GitHub"). Then run:
+```bash
+git clone https://github.com/singularity-energy/open-grid-emissions.git
+```
 
+#### Setup the conda environment
+Open anaconda prompt, use `cd` to navigate to the directory where your local files are stored (e.g. "GitHub/open-grid-emissions"), and then run:
+```bash
+conda update conda
+conda env create -f environment.yml
 ```
-git clone https://github.com/singularity-energy/open-grid-emissions.git
+and activate the environment:
+```bash
+conda activate open_grid_emissions
 ```
+Installation requires that the conda channel-priority be set to "flexible". This is the default behavior, so if you've never manually changed this, you shouldn't have to worry about this. However, if you receive an error message like "Found conflicts!" when trying to install the environment, try setting your channel priority to flexible by running the following command:`conda config --set channel_priority flexible` and then re-running the above commands.
 
-### Setup the conda environment
+### Setup with pipenv
+#### Install python and git
+We recommend that you use Python 3.11. If you don't have Python installed, we recommend that you use [**pyenv**](https://github.com/pyenv/pyenv). It lets you easily switch between multiple versions of Python. You will also need to use git to clone the repository. It can be installed from https://git-scm.com/downloads,
 
-Open anaconda prompt, use `cd` to navigate to the directory where your local files are stored (e.g. "GitHub/open-grid-emissions"), and then run:
+#### Install pipenv
+This can be done via:
+```bash
+pip install pipenv
+```
 
+#### Download the codebase
+As mentioned previously, clone the repository with:
+```bash
+git clone https://github.com/singularity-energy/open-grid-emissions.git
 ```
-conda update conda
-conda env create -f environment.yml
+and navigate to the root of the directory:
+```bash
+cd open-grid-emissions
 ```
 
-Installation requires that the conda channel-priority be set to "flexible". This is the default behavior, 
-so if you've never manually changed this, you shouldn't have to worry about this. However, 
-if you receive an error message like "Found conflicts!" when trying to install the environment,
-try setting your channel priority to flexible by running the following command:
-`conda config --set channel_priority flexible` and then re-running the above commands.
+#### Setup the environment
+In the root of the directory, create and activate the environment with:
+```bash
+# set up virtual environment (use whichever version of python 3.11 you have installed)
+pipenv --python 3.11.4
 
-## Running the complete data pipeline
+# if you have updated the pipfile and need to update pipfile.lock, run
+pipenv install
+# Otherwise, if you just want to install packages from the pipfile.lock, run
+pipenv sync
 
-If you would like to run the full data pipeline to generate all intermediate outputs and results files, open anaconda prompt, navigate to `open-grid-emissions/src`, and run the following (replacing 2021 with whichever year you want to run):
+# activate virtual environment
+pipenv shell
 
+# install an editable version of the oge package
+pip install build
+python -m build
+pip install –-editable .
+```
+
+If you ever need to remove and reinstall the environment, run `pipenv --rm` from the root directory then follow the directions above.
+
+### Running the complete data pipeline
+If you would like to run the full data pipeline to generate all intermediate outputs and results files, navigate to `open-grid-emissions/src`, and run the following (replacing 2021 with whichever year you want to run):
 ```
-conda activate open_grid_emissions
 python data_pipeline.py --year 2021
 ```
 
-## Keeping the code updated
-
+### Keeping the code updated
 From time to time, the code will be updated on GitHub. To ensure that you are keeping your local version of the code up to date, open git bash and follow these steps:
 ```
 # change the directory to where ever your local git repository is saved
@@ -147,16 +186,18 @@ git add .
 git pull origin main
 ```
 
-# Contribution Guidelines
+### Install a code editor
+If you want to edit the code and do not already have an integrated development environment (IDE) installed, one good option is Visual Studio Code (download: https://code.visualstudio.com/).
 
+## Contribution Guidelines
 If you plan on contributing edits to the codebase that will be merged into the main branch, please follow these best practices:
 
 1. Please do not make edits directly to the main branch. Any new features or edits should be completed in a new branch. To do so, open git bash, navigate to your local repo (e.g. `cd GitHub/open-grid-emissions`), and create a new branch, giving it a descriptive name related to the edit you will be doing:
 
 	`git checkout -b branch_name`
 
-2. As you code, it is a good practice to 'save' your work frequently by opening git bash, navigating to your local repo (`cd GitHub/open-grid-emissions`), making sure that your current feature branch is active (you should see the feature name in parentheses next to the command line), and running 
-	
+2. As you code, it is a good practice to 'save' your work frequently by opening git bash, navigating to your local repo (`cd GitHub/open-grid-emissions`), making sure that your current feature branch is active (you should see the feature name in parentheses next to the command line), and running
+
 	`git add .`
 
 3. You should commit your work to the branch whenever you have working code or whenever you stop working on it using:
@@ -168,9 +209,9 @@ If you plan on contributing edits to the codebase that will be merged into the m
 
 	`git push`
 
-5. Now open the GitHub repo web page. You should see the branch you pushed up in a yellow bar at the top of the page with a button to "Compare & pull request". 
-	- Click "Compare & pull request". This will take you to the "Open a pull request" page. 
-	- From here, you should write a brief description of what you actually changed. 
+5. Now open the GitHub repo web page. You should see the branch you pushed up in a yellow bar at the top of the page with a button to "Compare & pull request".
+	- Click "Compare & pull request". This will take you to the "Open a pull request" page.
+	- From here, you should write a brief description of what you actually changed.
 	- Click "Create pull request"
 	- The changes will be reviewed and discussed. Once any edits have been made, the code will be merged into the main branch.
 
@@ -180,5 +221,5 @@ If you plan on contributing edits to the codebase that will be merged into the m
 - All pandas merge operations should include the `validate` parameter to ensure that unintentional duplicate entries are not created (https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.merge.html)
 - All pandas groupby operations should include the `dropna=False` parameter so that data with missing groupby keys are not unintentionally dropped from the data.
 - All code should be formatted using `black`
-- Clear all outputs from notebooks before committing your work. 
+- Clear all outputs from notebooks before committing your work.
 - Any manual changes to reported categorical data, conversion factors, or manual data mappings should be loaded from a .csv file `data/manual` rather than stored in a dictionary or variable in the code.