Bias adjusted precip #3202
Bias adjusted precip #3202
Conversation
| stmt = """ | ||
| SELECT day, station_code, sum(precipitation) actual_precip_24h | ||
| FROM | ||
| generate_series(\'{}\', \'{}\', '24 hours'::interval) day |
There was a problem hiding this comment.
I'm not familiar with this syntax, but is this creating a series of days based on 24 hour intervals, then selecting each weather date based on it's hour being less than the subset of the series, then summing up the precip based on the hourly actuals that fit in that window? (There's probably a better way of expressing this in English)...
There was a problem hiding this comment.
Do we have/need an index on hourly_actuals.weather_date for this query?
There was a problem hiding this comment.
Haha, you have the gist of it. I'm glad you asked this question, I think I found an off-by-one day logic error.
I'll add to the comment with a description of what this is doing.
generate_series(\'{}\', \'{}\', '24 hours'::interval)
This gives us a one column table of dates separated by 24 hours between the start and end dates. For example, if start and end dates are 2023-10-31 20:00:00 to 2023-11-03 20:00:00 we would have a table like:
| day |
|---|
| 2023-10-31 20:00:00 |
| 2023-11-01 20:00:00 |
| 2023-11-02 20:00:00 |
| 2023-11-03 20:00:00 |
We then join the HourlyActuals table so that we can sum hourly precip in a 24 hour period. The join is based on the weather_date field in the HourlyActuals table being in a 24 hour range using this odd looking syntax:
weather_date <@ tstzrange(day, day + '24 hours', '(]')
Using 2023-10-31 20:00:00 as an example, rows with the following dates would match. The (] syntax means the lower bound is excluded but the upper bound is included.
2023-10-31 21:00:00
2023-10-31 22:00:00
2023-10-31 23:00:00
2023-11-01 00:00:00
2023-11-01 01:00:00
....
2023-11-01 19:00:00
2023-11-01 20:00:00
My off by one error is that right now I am attributing this accumulated precipitation to October 31, but it should actually be for Nov 1 (we need the preceding 24 hour precip).
There was a problem hiding this comment.
An index on weather_date is a great idea!
Codecov Report
@@ Coverage Diff @@
## main #3202 +/- ##
==========================================
- Coverage 82.16% 82.05% -0.11%
==========================================
Files 272 274 +2
Lines 9240 9301 +61
Branches 413 413
==========================================
+ Hits 7592 7632 +40
- Misses 1535 1557 +22
+ Partials 113 112 -1
... and 3 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
Whitespace
|
Kudos, SonarCloud Quality Gate passed!
|
Co-authored-by: Conor Brady <con.brad@gmail.com>
bias_adjusted_precip_24htoWeatherModelStationPredictionsand the materialized viewPrecipModelclass for handling precip specific linear regression (a uniqueadd_samples()function is requiredTest Links:
Landing Page
MoreCast 2.0
Percentile Calculator
MoreCast
C-Haines
FireBat
FireBat bookmark
Auto Spatial Advisory (ASA)
HFI Calculator