Use pooch to download data

[adamltyson]

Pooch could be used to download:

• The atlases as part of the API
• The data used to generate atlases

This would provide many benefits, mostly in that we could remove all of our own code to do this, and it would also provide validation that atlases are properly downloaded (with checksums etc).

[adamltyson]

this should hopefully fix - #334

[PolarBean]

I support using pooch to download the atlases in the atlas creation scripts, but one benefit of using a centralised database of atlases is we avoid link rot. So many atlasing papers I have been reviewing recently have dead links to the data files. In many cases these datasets are completely lost. Maybe this is out of scope for the API though.

[adamltyson]

This issue is about using pooch to:

• Fetch the atlas data during packaging
• Fetch the data from GIN when using the API

It only replaces the ad-hoc approach of using different download functions. We will still store our own central copy of all (repackaged) atlases on GIN (and in the future, likely mirrored elsewhere).

[PolarBean]

ah that makes sense then!

[NicoKiaru]

I don't know if it's really part of this issue, but why not storing the atlas raw data on Zenodo ? I experience a very slow download speed with brainglobe (less than a 1MB/s). Zenodo has a very good infrastructure and allow direct dl links without login, etc. And I have a dl speed of over 50 Mb/s over wifi, so wifi is probably limiting here.

[adamltyson]

It's a different issue, but a good point. We're looking at setting up mirrors to improve download speed, but zenodo does seem like a good option. I'm not sure what happened with GIN, I don't recall it being this slow when we first set it up.