Logic for processing VIIRS snow data #3332
Conversation
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #3332 +/- ##
==========================================
- Coverage 81.53% 81.39% -0.15%
==========================================
Files 277 278 +1
Lines 9560 9681 +121
Branches 450 450
==========================================
+ Hits 7795 7880 +85
- Misses 1646 1682 +36
Partials 119 119 ☔ View full report in Codecov by Sentry. |
| @@ -184,7 +184,7 @@ class ModelRunPrediction(Base): | |||
| # The date and time to which the prediction applies. | |||
| prediction_timestamp = Column(TZTimeStamp, nullable=False, index=True) | |||
| # The station code representing the location (aka weather station). | |||
| station_code = Column(Integer, nullable=True) | |||
| station_code = Column(Integer, nullable=False) | |||
There was a problem hiding this comment.
I'm guessing this should have always been False, correct?
There was a problem hiding this comment.
Yep. I think I originally had it set to True as a column was being added to an existing table so there was the possibility of null values and providing a default didn't make sense. In the end I deleted the contents of the table in the migration and had the migration set nullable=False. This change brings the code into alignment with the current schema of the db.
api/app/jobs/viirs_snow.py
Outdated
| session = requests.session() | ||
| resp = session.get(f"https://n5eil02u.ecs.nsidc.org/egi/capabilities/{SHORT_NAME}.{PRODUCT_VERSION}.xml") | ||
| session.get(resp.url, auth=(config.get("NASA_EARTHDATA_USER"),config.get("NASA_EARTHDATA_PWD"))) | ||
| # Check if request was successful |
There was a problem hiding this comment.
Looks like we do 2 requests here? One to authenticate, then one to get the data. Where are we checking that the auth request was successful?
There was a problem hiding this comment.
Oops, that's a leftover comment at this spot that I'll remove. The check to see if we've made a successful request actually occurs below with request.raise_for_status(). If the request for data returns anything but a 200 this call with raise an error which we subsequently handle.
Co-authored-by: Conor Brady <con.brad@gmail.com>
Co-authored-by: Conor Brady <con.brad@gmail.com>
|
Co-authored-by: Conor Brady <con.brad@gmail.com>
The general flow is:
Occasionally there is a delay in availability of the data. If we encounter this we exit gracefully and try again the next time the cron job runs. This is a pretty specific HTTP status code (501). All other errors get logged and a notification sent to RocketChat.
You can view the output files in S3 at /snow_coverage/{YYYY-mm-dd}/*.tif. The most recent folder should ideally correspond with the most recent available folder at https://n5eil01u.ecs.nsidc.org/VIIRS/VJ110A1.002/.
Test Links:
Landing Page
MoreCast 2.0
Percentile Calculator
MoreCast
C-Haines
FireBat
FireBat bookmark
Auto Spatial Advisory (ASA)
HFI Calculator