What to do about ERA5 and MERRA2?

[williamhobbs]

It looks like #1264 (iotool function for getting ERA5 data) and #1274 (iotools function for getting MERRA2 data) are on hold due to concerns around dependencies, specifically xarray, dask, cdsapi, and pydap. #1264 (comment) has more info. @kanderso-nrel also pointed out to me that this ecmwf/cdsapi#47 has been sitting for about 10 months, which is not a great sign regarding the level of attention that cdsapi is getting.

I don't have much of an opinion about MERRA2, but at least for ERA5 there could be a number of benefits to making that reanalysis dataset readily available to users, specifically:

1. when many historical years of data are needed: it covers almost twice as many years as NSRDB PSM3 (maybe a lot more than that)
2. when recent data are needed: ERA5 preliminary data are available within 5 days of real time [1], unlike PSM3 which often has 6-18 months of lag

Due to item 1, it is used by some grid reliability planners [2] and other energy researchers [3], and it's commonly used in the wind industry (MERRA2 has also been used, but my understanding is that ERA5 is now preferred by most). Item 2 can be addressed with commercial datasets (see #1310), but that's not an option for everyone, particularly for non-commercial research efforts like PV Fleets. Feeding ERA5 data into PVAnalytics to automatically check POA and back of module sensors for drift on a monthly basis would be cool, too.

Ideas:

• Maybe iotools could become a standalone package if the scope/dependencies get too broad?
• Maybe the functionality of cdsapi needs to be replaced with something managed under pvlib (I think this is all of it, https://github.com/ecmwf/cdsapi/blob/master/cdsapi/api.py, not sure how significant that is)?
• Maybe someone reaches out to the ECMWF team to figure out what the long term plan for cdsapi maintenance is?

[1] https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5
[2] https://www.ercot.com/files/docs/2020/05/26/SAWG__Meeting_5-28-2020_Draft_Wind_and_Solar_Profiles.pptx
[3] https://www.epri.com/research/products/000000003002020619

[wholmgren]

I'm not opposed to optional dependencies for xarray and dask. pydap has been largely abandoned for years. There was a flurry of activity a few months ago and then it was abandoned again without actually publishing a new release to pypi. I'd like to think that ECMWF can maintain cdsapi, but yeah, that PR (and the issue board in general) shows that the package is not developed at the level we'd like to see for a dependency.

Is the desired data available through the NCAR/UCAR RDA?

[kandersolar]

This project claims to quickly supply ERA5 data up to 5 days ago: https://open-meteo.com/en/docs/historical-weather-api

Perhaps the ERA5 users in this thread could take a look and comment on whether that API seems like a good option for pvlib.iotools?

[williamhobbs]

That looks promising. I've looked at that project for forecast NWP variables, but not for ERA-5. It requires a paid license for commercial use, which maybe isn't ideal, but that's workable.

Are there existing examples of interfacing with APIs that require a paid license? I know it has been discussed in #1313 and #1310, but I don't think anything has been implemented.

And I think Open-Meteo is different from the services described in the above linked issues: all of those require an API key or some other authentication to get any data, while Open-Meteo appears to be an "honor system" API where users can get data without providing an API key. Does that raise concerns? For example, if someone creates an example that uses Open-Meteo for non-commercial purposes, then someone inadvertently re-uses that code for commercial purposes without even knowing there is a licensing issue? It probably isn't an issue, but I thought I'd raise the question.

[kandersolar]

I tried out accessing ERA5 data through the UCAR RDA. I ran into lots of issues but finally got something working using siphon. The data is available in 15-day chunks, and fetching one chunk of hourly DNI+GHI data for one location took ~15 seconds. From that I expect a full year would take ~6 minutes to retrieve, although I did not try it.

Worth exploring further? It's quite slow compared to something like PVGIS/PSM but fast compared with cdsapi.

Comparison with SURFRAD data:

[williamhobbs]

Seems worth exploring to me. The only downside of UCAR RDA that I know of is (restating my comment from earlier) that it may have ~3 month lag because it only has the quality-assured and not the preliminary data with a ~5 day lag.

[AdamRJensen]

Agree that it seems worthy of further exploration!

[williamhobbs]

@dfregosi gave a presentation at PVSC this year on power loss rate analysis with RdTools. It included ERA5 and Danny and team found (what looked to me looked like) very good performance. The title was "Comparison of Calculated Performance Loss Rates of Large Scale PV Plant using Satellite and Ground Based Sensors", and the abstract is available here https://ieee-pvsc.org/ePVSC/mobile/show_presentation.php?abstractno=167.

Tagging @kperrynrel, since we briefly discussed this.

[williamhobbs]

@kandersolar, you had earlier pointed out that ecmwf/cdsapi#47 had been sitting for a long time, which caused concern about the state of cdsapi. I noticed that the pull request was addressed earlier this year, but also saw this comment on an unrelated issue, ecmwf/cdsapi#44 (comment), including:

Sorry for the lack of a reply. The reason is that User Support don't monitor this repo. The best place to raise queries like this is through the ADS itself: https://ads.atmosphere.copernicus.eu/cdsapp#!/usersupport.

So maybe cdsapi is in ok state of maintenance, it just doesn't follow typical conventions?

[adriesse]

Could this be of interest?

https://github.com/planet-os/notebooks/blob/master/aws/era5-pds.md

[AdamRJensen]

It looks fairly promising actually. Probably would be significantly faster than the official ADS/CDS.

[kandersolar]

I think that's the same as this, right? #1484 (reply in thread)

[williamhobbs]

I previously said it was no longer supported, but now I don't see any indication of that, so I don't know why I said it a year ago...

[williamhobbs]

Aha. A year ago it was deprecated. Wayback Machine to the rescue.

Apparently it has been picked back up.