Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates to facility metrics and dashboard summaries #52

Merged
merged 24 commits into from
Sep 25, 2024
Merged
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
a04268c
ci: refactor actions
jacobdadams Aug 16, 2024
655b79c
ci: weird name liveshare bug
jacobdadams Aug 16, 2024
54a0279
ci: fix topic name
jacobdadams Aug 16, 2024
35aa45a
ci: one more _topic
jacobdadams Aug 16, 2024
cc45c7b
ci: read the actual error message
jacobdadams Aug 16, 2024
4ca4855
ci: topic should be before deploy
jacobdadams Aug 16, 2024
7740ccd
ci: line continuations
jacobdadams Aug 16, 2024
598f8fa
ci: missing close quote
jacobdadams Aug 16, 2024
98d787c
ci: code already checked out by calling action
jacobdadams Aug 16, 2024
9d94ec7
ci: update service_account
jacobdadams Aug 19, 2024
d0c1e43
ci: revert service account change
jacobdadams Aug 19, 2024
b27f99f
chore: add facility classification to validation
jacobdadams Aug 19, 2024
f99fb40
ci: proper trigger type?
jacobdadams Aug 19, 2024
8992ac2
ci: user error
jacobdadams Aug 19, 2024
1c40892
ci: pub/sub is default?
jacobdadams Aug 19, 2024
7e2dd89
ci: nope, default seems to be http
jacobdadams Aug 19, 2024
b660a48
ci: starting from scratch, copying scott's
jacobdadams Aug 19, 2024
e12e0fb
feat: calc total diverted tons for county and state
jacobdadams Aug 20, 2024
3ea9dec
feat: add recycling totals for recycling facilities
jacobdadams Sep 4, 2024
5ba3011
tests: add columns for total diverted
jacobdadams Sep 4, 2024
2f06576
chore: rename BFS materials
jacobdadams Sep 11, 2024
23defe0
ci: set python version in tests
jacobdadams Sep 24, 2024
85eb7d5
ci: run on dev, main
jacobdadams Sep 25, 2024
3b0d872
ci: report test coverage
jacobdadams Sep 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 111 additions & 0 deletions .github/actions/deploy/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
name: Deploy to GCP
description: Deploy to GCP
inputs:
project_id:
description: "The GCP project ID"
required: true
identity_provider:
description: "The identity provider for the workload identity"
required: true
service_account_email:
description: "The service account email"
required: true
storage_bucket:
description: "The GCP storage bucket"
required: true

runs:
using: composite
steps:
- name: Set globals
id: globals
shell: bash
run: |
echo "MAIN_SCHEDULE_NAME=wmrc_main" >> "${GITHUB_OUTPUT}"
echo "MAIN_SCHEDULE_CRON=0 22 * * 6" >> "${GITHUB_OUTPUT}"
echo "MAIN_SCHEDULE_DESCRIPTION=Trigger the wmrc-skid bot every saturday evening at 10pm" >> "${GITHUB_OUTPUT}"
echo "VALIDATOR_SCHEDULE_NAME=validator" >> "${GITHUB_OUTPUT}"
echo "VALIDATOR_SCHEDULE_DESCRIPTION=Trigger the wmrc validation bot every 1st of April, May, and June at 8am" >> "${GITHUB_OUTPUT}"
echo "VALIDATOR_SCHEDULE_CRON=0 8 1 4-6 *" >> "${GITHUB_OUTPUT}"
echo "TOPIC_NAME=wmrc-topic" >> "${GITHUB_OUTPUT}"

- name: 🗝️ Authenticate to Google Cloud
id: auth
uses: google-github-actions/auth@v2
with:
create_credentials_file: true
token_format: access_token
workload_identity_provider: ${{ inputs.identity_provider }}
service_account: ${{ inputs.service_account_email }}

- name: 📥 Create Main PubSub topic
shell: bash
run: |
if [ ! "$(gcloud pubsub topics list | grep ${{ steps.globals.outputs.TOPIC_NAME }})" ]; then
gcloud pubsub topics create ${{ steps.globals.outputs.TOPIC_NAME }} --quiet
fi

- name: 🚀 Deploy Main Cloud Function
id: deploy
uses: google-github-actions/deploy-cloud-functions@v3
with:
name: wmrc-skid
runtime: python311
entry_point: subscribe
source_dir: src/wmrc
service_account: cloud-function-sa@${{ inputs.project_id }}.iam.gserviceaccount.com
event_trigger_type: google.cloud.pubsub.topic.v1.messagePublished
event_trigger_pubsub_topic: projects/${{ inputs.project_id }}/topics/${{ steps.globals.outputs.TOPIC_NAME }}
memory: 1024M
service_timeout: 9m
environment_variables: STORAGE_BUCKET=${{ inputs.storage_bucket }}
secrets: |
/secrets/app/secrets.json=${{ inputs.project_id }}/skid-secrets
max_instance_count: 1
event_trigger_retry: false

- name: 🕰️ Create Main Cloud Scheduler
shell: bash
run: |
if [ ! "$(gcloud scheduler jobs list --location=us-central1 | grep ${{ steps.globals.outputs.MAIN_SCHEDULE_NAME }})" ]; then
gcloud scheduler jobs create pubsub "${{ steps.globals.outputs.MAIN_SCHEDULE_NAME }}" \
--description="${{ steps.globals.outputs.MAIN_SCHEDULE_DESCRIPTION }}" \
--schedule="${{ steps.globals.outputs.MAIN_SCHEDULE_CRON }}" \
--time-zone=America/Denver \
--location=us-central1 \
--topic="${{ steps.globals.outputs.TOPIC_NAME }}" \
--message-body='facility updates' \
--quiet
else
gcloud scheduler jobs update pubsub "${{ steps.globals.outputs.MAIN_SCHEDULE_NAME }}" \
--description="${{ steps.globals.outputs.MAIN_SCHEDULE_DESCRIPTION }}" \
--schedule="${{ steps.globals.outputs.MAIN_SCHEDULE_CRON }}" \
--time-zone=America/Denver \
--location=us-central1 \
--topic="${{ steps.globals.outputs.TOPIC_NAME }}" \
--message-body='facility updates' \
--quiet
fi

- name: 🕰️ Create Validator Cloud Scheduler
shell: bash
run: |
if [ ! "$(gcloud scheduler jobs list --location=us-central1 | grep ${{ steps.globals.outputs.VALIDATOR_SCHEDULE_NAME }})" ]; then
gcloud scheduler jobs create pubsub "${{ steps.globals.outputs.VALIDATOR_SCHEDULE_NAME }}" \
--description="${{ steps.globals.outputs.VALIDATOR_SCHEDULE_DESCRIPTION }}" \
--schedule="${{ steps.globals.outputs.VALIDATOR_SCHEDULE_CRON }}" \
--time-zone=America/Denver \
--location=us-central1 \
--topic="${{ steps.globals.outputs.TOPIC_NAME }}" \
--message-body='validate' \
--quiet
else
gcloud scheduler jobs update pubsub "${{ steps.globals.outputs.VALIDATOR_SCHEDULE_NAME }}" \
--description="${{ steps.globals.outputs.VALIDATOR_SCHEDULE_DESCRIPTION }}" \
--schedule="${{ steps.globals.outputs.VALIDATOR_SCHEDULE_CRON }}" \
--time-zone=America/Denver \
--location=us-central1 \
--topic="${{ steps.globals.outputs.TOPIC_NAME }}" \
--message-body='validate' \
--quiet
fi
40 changes: 40 additions & 0 deletions .github/workflows/pull_request.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
name: Pull Request Events

on:
pull_request:

concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
cancel-in-progress: true

jobs:
test-unit:
name: Unit Tests
runs-on: ubuntu-latest

steps:
- name: ⬇️ Set up code
uses: actions/checkout@v4
with:
show-progress: false

- name: 🐍 Set up Python
uses: actions/setup-python@v5
with:
jacobdadams marked this conversation as resolved.
Show resolved Hide resolved
python-version: 3.11
cache: pip
cache-dependency-path: setup.py

- name: 📥 Install dependencies
run: |
sudo apt-get update
sudo apt-get install -y libkrb5-dev

- name: 🏗 Install module
run: pip install .[tests]

- name: 🧶 Lint
run: ruff check --output-format=github .

- name: 🧪 Run pytest
run: pytest
Loading