SFMS: Daily FFMC #4081

conbrad · 2024-11-12T19:53:25Z

Adds daily FFMC logic for calculating daily FFMC rasters.
Renames bui job to account for ffmc and bui logic
Updates coverage and adds pytest-cov dev dependencies for running coverage in VSCode locally

codecov · 2024-11-12T20:51:02Z

Codecov Report

Attention: Patch coverage is 91.89189% with 3 lines in your changes missing coverage. Please review.

Project coverage is 80.37%. Comparing base (f00bc9f) to head (a9ebd57).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
api/app/jobs/sfms_calculations.py	57.14%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4081      +/-   ##
==========================================
+ Coverage   80.35%   80.37%   +0.02%     
==========================================
  Files         309      309              
  Lines       11828    11845      +17     
  Branches      537      537              
==========================================
+ Hits         9504     9521      +17     
  Misses       2138     2138              
  Partials      186      186

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

brettedw · 2024-11-12T21:36:31Z

api/app/sfms/raster_addresser.py

+        """
+        assert_all_utc(timestamp)
+        iso_date = timestamp.date().isoformat()
+        return f"sfms/uploads/{run_type.value}/{iso_date}/{weather_param.value}{iso_date.replace('-', '')}.tif"


Is this meant to get RDPS weather model data? get_model_data_key does that, but it's stored in a different place in our s3 bucket

Nope, same reasoning as: https://github.com/bcgov/wps/pull/4081/files#r1838823535

brettedw · 2024-11-12T21:46:28Z

api/app/sfms/raster_addresser.py

+        iso_date = timestamp.date().isoformat()
+        return f"sfms/uploads/{run_type.value}/{iso_date}/fine_fuel_moisture_code{iso_date.replace('-', '')}.tif"
+
+    def get_calculated_daily_ffmc(self, timestamp: datetime):


I was using the generic get_calculated_index_key for this

Similar reason to: https://github.com/bcgov/wps/pull/4081/files#r1838825805

brettedw · 2024-11-12T21:47:19Z

api/app/sfms/raster_addresser.py

@@ -81,6 +99,16 @@ def get_calculated_precip_key(self, datetime_to_calculate_utc: datetime):
        calculated_weather_prefix = f"{self.weather_model_prefix}/{datetime_to_calculate_utc.date().isoformat()}/"
        return os.path.join(calculated_weather_prefix, compose_computed_precip_rdps_key(datetime_to_calculate_utc))

+    def get_daily_ffmc(self, timestamp: datetime, run_type: RunTypeEnum):


I was using get_uploaded_index_key for this part, maybe makes sense to split them out though, whatever you think works best

get_uploaded_index_key is based on an enum that contains only the abbreviated FWI names, whereas SFMS gives us the full FWI name for FFMC, so I opted for a specific function.

brettedw · 2024-11-12T21:49:47Z

api/app/sfms/daily_ffmc_processor.py

+                logging.warning(f"No ffmc objects found for key: {yesterday_ffmc_key}")
+                return
+
+            temp_forecast_key = self.addresser.get_daily_model_data_key(current_day, RunType.FORECAST, WeatherParameter.TEMP)


I think if we want to use weather_model data we need to keep track of a prediction_hour because we get model data for so many different hours, but maybe there's a better way to do it

Opted to use all the daily SFMS model data since that's where we get the previous FFMC raster. Not sure if that's correct though. I guess that conflicts with our BUI calculations but the seed FFMC raster will either way.

Gotcha, I could be wrong but I think the intention was to step forward in these calculations with weather model data (RDPS), since the weather data coming from SFMS is the coarsely interpolated forecasts from BC weather stations

dgboss · 2024-11-13T16:10:37Z

api/app/tests/sfms/test_daily_fwi_processor.py

@@ -62,14 +62,15 @@ async def test_bui_date_range_processor(mocker: MockerFixture):
    get_weather_data_key_spy = mocker.spy(mock_key_addresser, "get_weather_data_keys")
    gdal_prefix_keys_spy = mocker.spy(mock_key_addresser, "gdal_prefix_keys")
    get_calculated_index_key_spy = mocker.spy(mock_key_addresser, "get_calculated_index_key")
-    bui_date_range_processor = BUIDateRangeProcessor(TEST_DATETIME, 2, mock_key_addresser)
+    bui_date_range_processor = DailyFWIProcessor(TEST_DATETIME, 2, mock_key_addresser)


Would you mind renaming the bui_date_range_processor variable and this test case (test_bui_date_range_processor)?

dgboss · 2024-11-13T16:17:06Z

api/app/tests/sfms/test_daily_fwi_processor.py

-    calculate_bui_spy = mocker.spy(date_range_processor, "calculate_bui")
+    calculate_dmc_spy = mocker.spy(daily_fwi_processor, "calculate_dmc")
+    calculate_dc_spy = mocker.spy(daily_fwi_processor, "calculate_dc")
+    calculate_bui_spy = mocker.spy(daily_fwi_processor, "calculate_bui")


For consistency we could test calculate_ffmc here as well.

dgboss

Looks great!

brettedw

Looks good!

brettedw · 2024-11-13T16:13:18Z

api/app/sfms/daily_fwi_processor.py

            fwi_keys_exist = await s3_client.all_objects_exist(dc_key, dmc_key)
            if not fwi_keys_exist:
                logging.warning(f"No previous DMC/DC keys found for {previous_fwi_datetime.date().isoformat()}")
                break

-            temp_key, rh_key, precip_key = self.addresser.gdal_prefix_keys(temp_key, rh_key, precip_key)
+            temp_key, rh_key, wind_speed_key, precip_key = self.addresser.gdal_prefix_keys(temp_key, rh_key, wind_speed_key, precip_key)
            dc_key, dmc_key = self.addresser.gdal_prefix_keys(dc_key, dmc_key)


I think ffmc might need to be included here

Great catch!

sonarcloud · 2024-11-13T17:48:34Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

conbrad added 3 commits November 7, 2024 13:29

initial impl

56a9cc9

daily ffmc tests

9c0899d

Add coverage dep and tests

b23f364

conbrad changed the title ~~Task/daily ffmc raster~~ SFMS: Daily FFMC Nov 12, 2024

conbrad added 2 commits November 12, 2024 11:55

Merge branch 'main' into task/daily-ffmc-raster

4f05b04

Update lock content hash

806bb85

conbrad requested review from brettedw and dgboss November 12, 2024 19:56

Fix raster addresser

2cfe911

conbrad added 2 commits November 12, 2024 13:27

Test calculate_ffmc function

fb1b53a

test shape and masking for ffmc

8e648de

brettedw reviewed Nov 12, 2024

View reviewed changes

conbrad requested a review from brettedw November 12, 2024 21:58

conbrad added 4 commits November 12, 2024 16:18

combine ffmc and bui cals

b9a9b95

Cleanup job entry

dc9e45c

More cleanup

2fbf45b

Cleanup imports

da48e99

dgboss reviewed Nov 13, 2024

View reviewed changes

dgboss approved these changes Nov 13, 2024

View reviewed changes

brettedw approved these changes Nov 13, 2024

View reviewed changes

conbrad added 4 commits November 13, 2024 09:11

Naming

ee731b0

prefix ffmc key

ca2c182

Fix test

1c85fb6

Merge branch 'main' into task/daily-ffmc-raster

a9ebd57

conbrad temporarily deployed to production November 13, 2024 18:02 Inactive

conbrad merged commit 6ef7b1c into main Nov 13, 2024
25 checks passed

conbrad deleted the task/daily-ffmc-raster branch November 13, 2024 18:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SFMS: Daily FFMC #4081

SFMS: Daily FFMC #4081

conbrad commented Nov 12, 2024 •

edited

Loading

codecov bot commented Nov 12, 2024 •

edited

Loading

brettedw Nov 12, 2024

conbrad Nov 12, 2024

brettedw Nov 12, 2024

conbrad Nov 12, 2024

brettedw Nov 12, 2024

conbrad Nov 12, 2024

brettedw Nov 12, 2024

conbrad Nov 12, 2024

brettedw Nov 12, 2024

dgboss Nov 13, 2024

dgboss Nov 13, 2024

dgboss left a comment

brettedw left a comment

brettedw Nov 13, 2024

conbrad Nov 13, 2024

sonarcloud bot commented Nov 13, 2024

SFMS: Daily FFMC #4081

SFMS: Daily FFMC #4081

Conversation

conbrad commented Nov 12, 2024 • edited Loading

Test Links:

codecov bot commented Nov 12, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dgboss left a comment

Choose a reason for hiding this comment

brettedw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented Nov 13, 2024

Quality Gate passed

conbrad commented Nov 12, 2024 •

edited

Loading

codecov bot commented Nov 12, 2024 •

edited

Loading