ENH: Allow Natural Earth version to be specified #2351

lgolston · 2024-03-15T02:46:06Z

Rationale

As requested in #2293, it would be useful for reproducibility to be able to specify what Natural Earth version to use, in addition to current default behavior of downloading the latest version.

Implications

This enhancement is added. If a version is specified, it will be downloaded to subfolders within cultural or physical corresponding to the version requested and then used.

Checklist

Testing: Since Natural Earth is already in several places in the test suite, I modified one of them to now use a specific version.

rcomer · 2024-03-15T08:45:17Z

Do we also need something to expose the version option on NaturalEarthFeature, similar to how scale is handled?

lgolston · 2024-03-16T02:11:23Z

Do we also need something to expose the version option on NaturalEarthFeature, similar to how scale is handled?

Yes, thanks. Version should definitely be an option there too but I wasn't tracking that feature - have added it in now.

lgolston · 2024-03-16T02:15:41Z

Related, I have linked there to https://github.com/nvkelso/natural-earth-vector/releases as a good source for version information, because it was confusing that most of the versions shown on https://www.naturalearthdata.com/ (e.g. 4.0.0, 2.0.0, etc.) are not available on the AWS service to download, yet there are many newer versioned releases that are available which are not shown on https://www.naturalearthdata.com/.

lgolston · 2024-03-17T02:16:13Z

CI helped pick up an omission of version in one place in NaturalEarthFeature - a clean version is now pushed with that edit.

kaedonkers

An elegant solution, thank you for implementing this!
I have not yet tested it myself, but it looks like a viable solution to me

kaedonkers · 2024-03-19T18:19:11Z

lib/cartopy/tests/mpl/test_features.py

@@ -37,7 +37,7 @@ def test_natural_earth():
 @pytest.mark.mpl_image_compare(filename='natural_earth_custom.png')
 def test_natural_earth_custom():
    ax = plt.axes(projection=ccrs.PlateCarree())
-    feature = cfeature.NaturalEarthFeature('physical', 'coastline', '50m',
+    feature = cfeature.NaturalEarthFeature('physical', 'coastline', '50m', '5.1.0',


Did you find that version 5.1.0 successfully passed this test?

Thanks! Yes, no diff between the baseline image here of Iceland and when using 5.1.0

kaedonkers · 2024-03-19T18:19:56Z

lib/cartopy/tests/test_shapereader.py

@@ -83,7 +83,8 @@ class TestRivers:
    def setup_class(self):
        RIVERS_PATH = shp.natural_earth(resolution='110m',
                                        category='physical',
-                                        name='rivers_lake_centerlines')
+                                        name='rivers_lake_centerlines',
+                                        version='5.0.0')


Similarly, did this specific version (5.0.0) of rivers pass the image comparison tests?

Yes, it was passing. Note there is no image comparison here, the test just looks at a few very specific features and there was no change.

greglucas

I didn't test this out yet, but would you be able to add some description to the docstrings about where the versions are cached/stored? I think they are stored separately looking at the code, but just want to make sure we aren't getting a mixture of versions in the default namespace. If the latest is 5.1.2 and I explicitly ask for 5.1.2 do we have to download/store both?

greglucas · 2024-06-08T21:03:13Z

lib/cartopy/io/shapereader.py

+        _NE_URL_TEMPLATE = ('https://naturalearth.s3.amazonaws.com/'
+                            '{version}/{resolution}_{category}/'
+                            'ne_{resolution}_{name}.zip')
+
+        default_spec = ('shapefiles', 'natural_earth', '{category}', '{version}',
+                        'ne_{resolution}_{name}.shp')


Would you be able to inject this into the current NE template? It looks like the only difference is {version}/, and I'm wondering if you could put that in and always have a version with something like _version_string = "" if version is None else f"{version}/".

Yes, I couldn't figure out how to inject version without making the second default downloader, but I think this solution works! Only problem is it obscures the template slightly but worth it to avoid duplicating the other code. For the other question, yes, if latest is 5.1.2 and then you specifically ask for 5.1.2 it will download it twice, since version-specified data will go in its own subfolder (within the relevant physical or cultural folder). Unfortunately, the .VERSION file is currently filtered out so it is hard to tell what version you have downloaded in the base folder.

greglucas · 2024-10-23T19:41:13Z

ping @lgolston, I had a few questions on this, but it looks like it may be very close to ready to go?

lgolston · 2024-10-26T03:47:40Z

ping @lgolston, I had a few questions on this, but it looks like it may be very close to ready to go?

Sorry for the long delay. I have made an update based on your review, however there is a test error I need to look into.

greglucas · 2024-10-29T16:47:35Z

lib/cartopy/feature/__init__.py

@@ -251,6 +253,8 @@ def __init__(self, category, name, scale, **kwargs):
            The dataset scale, i.e. one of '10m', '50m', or '110m',
            or Scaler object. Dataset scales correspond to 1:10,000,000,
            1:50,000,000, and 1:110,000,000 respectively.
+        version: optional


Can you also update the above cases to follow numpydoc formatting by adding : str to all of them.

Suggested change

version: optional

version : str, optional

greglucas · 2024-10-29T16:48:39Z

lib/cartopy/io/shapereader.py

@@ -299,10 +301,11 @@ def natural_earth(resolution='110m', category='physical', name='coastline'):
    # get hold of the Downloader (typically a NEShpDownloader instance)
    # which we can then simply call its path method to get the appropriate
    # shapefile (it will download if necessary)
+    _version_string = "" if version is None else f"{version}/"


This doesn't need to be private if it is just used within the method.

Suggested change

_version_string = "" if version is None else f"{version}/"

version_string = "" if version is None else f"{version}/"

greglucas · 2024-10-29T16:52:59Z

lib/cartopy/io/shapereader.py

@@ -386,7 +389,7 @@ def default_downloader():
 ne_{resolution}_{name}.shp

        """
-        default_spec = ('shapefiles', 'natural_earth', '{category}',
+        default_spec = ('shapefiles', 'natural_earth', '{category}', '{version}',


Looks like the doctest failure is related to the docstring right above this. Where it got a /{version}/ added in now. I actually don't think that should be included there now because it is path joining the version, but we don't want that if version is None, we want to leave off the /. I guess the question is does this need to be udpated since it is the default case (no version) or can we just leave it alone for now?

lgolston force-pushed the nat-earth-version branch from b5d5a72 to 1654f0a Compare March 17, 2024 02:05

rcomer requested a review from kaedonkers March 17, 2024 09:16

rcomer linked an issue Mar 17, 2024 that may be closed by this pull request

Natural Earth: Include version in URL #2293

Open

kaedonkers reviewed Mar 19, 2024

View reviewed changes

rcomer self-requested a review April 26, 2024 16:22

greglucas reviewed Jun 8, 2024

View reviewed changes

lgolston force-pushed the nat-earth-version branch from 727e25f to 577729d Compare October 26, 2024 03:17

Allow version specific NE

d3e8f7e

lgolston force-pushed the nat-earth-version branch from 577729d to d3e8f7e Compare October 26, 2024 03:30

greglucas reviewed Oct 29, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Allow Natural Earth version to be specified #2351

ENH: Allow Natural Earth version to be specified #2351

lgolston commented Mar 15, 2024

rcomer commented Mar 15, 2024

lgolston commented Mar 16, 2024

lgolston commented Mar 16, 2024

lgolston commented Mar 17, 2024

kaedonkers left a comment

kaedonkers Mar 19, 2024

lgolston Mar 20, 2024

kaedonkers Mar 19, 2024

lgolston Mar 20, 2024

greglucas left a comment

greglucas Jun 8, 2024 •

edited

Loading

lgolston Oct 26, 2024

greglucas commented Oct 23, 2024

lgolston commented Oct 26, 2024

greglucas Oct 29, 2024

greglucas Oct 29, 2024

greglucas Oct 29, 2024

	_version_string = "" if version is None else f"{version}/"
	version_string = "" if version is None else f"{version}/"

ENH: Allow Natural Earth version to be specified #2351

Are you sure you want to change the base?

ENH: Allow Natural Earth version to be specified #2351

Conversation

lgolston commented Mar 15, 2024

Rationale

Implications

Checklist

rcomer commented Mar 15, 2024

lgolston commented Mar 16, 2024

lgolston commented Mar 16, 2024

lgolston commented Mar 17, 2024

kaedonkers left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

greglucas left a comment

Choose a reason for hiding this comment

greglucas Jun 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

greglucas commented Oct 23, 2024

lgolston commented Oct 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

greglucas Jun 8, 2024 •

edited

Loading