sf_transform_xy() warns a lot

[yutannihilation]

I see this warning when I run tests, and I found sf_transform_xy() emits this.

hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)

I'm yet to figure out what's happening here, but it seems this is the cause: r-spatial/sf#2166 Considering the test doesn't fail, I think this is not a serious problem at least at the moment. But let me report here anyway.

Reprex is below. But, since this seems somehow related to the version of PROJ, you might not be able to reproduce this on your machine (I'm on Windows).

points <- data.frame(
  x = c(-80, -80, -76, -76),
  y = c(35, 40, 35, 40)
)

points_trans <- ggplot2:::sf_transform_xy(points, 3347, 4326)

^{Created on 2023-06-17 with reprex v2.0.2}

Standard output and standard error

✖ Install the styler package in order to use `style = TRUE`.
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)
hgridshift: could not find required grid(s).
pipeline: Pipeline: Bad step definition: inv (File not found or invalid)

[teunbrand]

Can reproduce this on Windows 11 with R 4.3.0 and {sf} 1.0-13, not on R 4.2.2 with same {sf} version. The suggestion in the linked issue (i.e. set Sys.setenv("PROJ_NETWORK"="ON") before using {sf}) seems to silence these warnings.

[teunbrand]

It appears one can muffle these warnings by setting sf_transform_xy(..., authority_compliant = TRUE), but I have no idea what kind of implications this would have to make this the default.

[yutannihilation]

I noticed the original issue filed on sf's repo is already closed: r-spatial/sf#2166 (comment)

At least, sf_proj_network(TRUE) works now (edit: with the GitHub version).

points <- data.frame(
  x = c(-80, -80, -76, -76),
  y = c(35, 40, 35, 40)
)

sf::sf_proj_network(TRUE) # new!
#> [1] "https://cdn.proj.org"

points_trans <- ggplot2:::sf_transform_xy(points, 3347, 4326)

^{Created on 2023-10-11 with reprex v2.0.2}

But, reading the example code in r-spatial/sf#2166, it seems the proper fix is to create a "coordinate operation pipeline" (I'm not sure what it is!). grid_availability = "DISCARD" seems to work.

points <- data.frame(
  x = c(-80, -80, -76, -76),
  y = c(35, 40, 35, 40)
)

sf_transform_xy <- function(data, target_crs, source_crs, authority_compliant = FALSE) {
  if (identical(target_crs, source_crs) ||
      is.null(target_crs) || is.null(source_crs) || is.null(data) ||
      is.na(target_crs) || is.na(source_crs) ||
      !all(c("x", "y") %in% names(data))) {
    return(data)
  }

  pipelines <- sf::sf_proj_pipelines(
    sf::st_crs(source_crs), sf::st_crs(target_crs),
    grid_availability = "DISCARD"
  )

  sf_data <- cbind(data$x, data$y)
  out <- sf::sf_project(
    pipelines$definition[1],
    NULL,
    sf_data,
    keep = TRUE,
    warn = FALSE
  )
  out <- ifelse(is.finite(out), out, NA) # replace any infinites with NA

  data$x <- out[, 1]
  data$y <- out[, 2]


  data
}

points_trans1 <- ggplot2:::sf_transform_xy(points, 3347, 4326)
points_trans2 <- sf_transform_xy(points, 3347, 4326)

all.equal(points_trans1, points_trans2)
#> [1] TRUE

^{Created on 2023-10-11 with reprex v2.0.2}

authority_compliant

?sf::sf_project says:

logical; TRUE means handle axis order authority compliant (e.g. EPSG:4326 implying x=lat, y=lon), FALSE means use visualisation order (i.e. always x=lon, y=lat)

[teunbrand]

Do you think it is worth bumping the {sf} version and change to the pipeline approach? (I'm also unsure what it is)

[yutannihilation]

Before answering your question, let me share what I've found so far.

PROJ 7 (released in 2020) introduced "significantly improved handling of gridded models." This uses "grid file"s to pursue precision. The RFC says

To get results with the highest precision, a grid file that defines a model that provides dimension shifts is often needed.

But, the amount of such grid files would be too large to bundle in the PROJ library, so they decided to host the grid files on a CDN and implement a functionality to download it on the fly (cf. https://proj.org/en/9.3/usage/network.html). In sf, this feature is not enabled by default, and user needs to turn on by executing sf_proj_network(TRUE) or setting Sys.setenv("PROJ_NETWORK"="ON") before loading sf.

In my incomplete understanding, the "grid file" is for precision, so probably it's fine to operate without downloading these if we don't care much about precision. Actually, there's an option to control the behavior when the grid file is not available.

https://proj.org/en/9.3/development/reference/datatypes.html#c.PROJ_GRID_AVAILABILITY_USE

Since it seems the sf package exposes this option only via grid_availability argument of sf_proj_pipelines(), probably the pipeline approach is the only way to handle the warnings in question when the grid files are unavailable.

The document (?sf::proj_tools) says:

• "USED" (default) = Grid availability is only used for sorting results. Operations where some grids are missing will be sorted last.
• "DISCARD" = Completely discard an operation if a required grid is missing.

but, it seems the results are the same at least in my code above. Probably, even when "USED" is selected, operations fail due to the lack of the grid files anyway...? I have no idea. Anyway, if so, or the difference is so subtle, the pipeline approach above can be an option.

However, to me, a proper fix would be to enable the downloading functionality with asking users for the permission to download. The implementation might require some version switch like below, but I don't think it needs a version bump.

  if (get_proj_version_of_sf() >= "7.0" && packageVersion("sf") >= "??" && isTRUE(!sf::sf_proj_network())) {
    ask_for_permission_and_abort_if_the_user_disagreed()
    sf::sf_proj_network(TRUE)
  }

[teunbrand]

For reasons I don't completely understand, these warnings no longer appear in recent CI runs or locally on my machine.

[yutannihilation]

Thanks. I tried on my local Windows and found,

• with sf installed via the binary package, I don't see the warnings.
• with sf installed via the source package, I still see the warnings.

The difference is that my Rtools is not the latest one and the version of PROJ is 9.2.0, while the binary package seems to use PROJ 9.3.0. I guess some fix was added between these.

sf::sf_extSoftVersion()
#>           GEOS           GDAL         proj.4 GDAL_with_GEOS     USE_PROJ_H           PROJ 
#>       "3.11.2"        "3.6.2"        "9.2.0"         "true"         "true"        "9.2.0" 

sf::sf_extSoftVersion()
#>           GEOS           GDAL         proj.4 GDAL_with_GEOS     USE_PROJ_H           PROJ 
#>       "3.11.2"        "3.7.2"        "9.3.0"         "true"         "true"        "9.3.0" 

Anyway, since this probably means the proper solution of this is to update the PROJ, I'm fine to close this issue.

[teunbrand]

Nice sleuthing! I agree on the solution.

[yutannihilation]

Thanks. Let's close for now and revisit if we find some case that updating won't solve!