Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add page_size argument to arc_select() #146

Merged
merged 2 commits into from
Feb 14, 2024
Merged

Add page_size argument to arc_select() #146

merged 2 commits into from
Feb 14, 2024

Conversation

JosiahParry
Copy link
Collaborator

@JosiahParry JosiahParry commented Feb 14, 2024

Checklist

  • update NEWS.md
  • documentation updated with devtools::document()
  • devtools::check() passes locally

Changes

This PR adds a page_size argument to arc_select() which permits us to have more control over the total number of features that are returned on a per-request basis. This will allow us to overcome the issue of timeouts with dense geometries.

Closes

Follow up tasks

  • Create docs on working with dense geometries. This doc will tell users how to utilize the batch_size argument to increase the number of requests made and reduce the number of features per page.

@JosiahParry JosiahParry changed the title add page_size parameter, tests, and improve docs and n_max behavior Add page_size argument to arc_select() Feb 14, 2024
@JosiahParry
Copy link
Collaborator Author

Relatedly, this will let us close #138 as it gives us a working solution to the problem. It will not be automagical but it will be doable!

@JosiahParry
Copy link
Collaborator Author

@mmachir could you please review and test when you get a chance? To install you can run:

# get the most recent dev version of {arcgisutils} if it hasn't
#  been updated since token overhaul
pak::pak("r-arcgis/arcgisutils")

# install branch:
pak::pak("r-arcgis/arcgislayers@batch_size")

Copy link
Collaborator

@mmachir mmachir left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

page size argument is looking good!

library(arcgislayers)
library(arcgisutils)

# test dense geometries
url <- 'https://ags.esri.co/arcgis/rest/services/DatosAbiertos/ELECCIONES_PRESIDENCIALES_2014_MPIO/MapServer/0'

# original approach
data <- arc_read(url)
#> Warning in parse_esri_json(httr2::resp_body_string(x)): Status code: 500
#> Error: Error performing query operation

# new approach
x <- arc_open(url)
res <- arc_select(x, n_max = 25, page_size = 5)

res1 <- arc_read(url, n_max=1000, page_size = 5)
#> Iterating ■■ 4% | ETA: 24sIterating ■■ 4% | ETA: 28sIterating ■■■ 6% | ETA:
#> 27sIterating ■■■ 8% | ETA: 23sIterating ■■■ 8% | ETA: 24s
...
#> ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 100% | ETA: 0s
res1
#> Simple feature collection with 1000 features and 12 fields
#> Geometry type: MULTIPOLYGON
#> Dimension:     XY
#> Bounding box:  xmin: -9098782 ymin: -471076.2 xmax: -7441413 ymax: 1504842
#> Projected CRS: WGS 84 / Pseudo-Mercator
#> First 10 features:
#>    OBJECTID DPTO_CCDGO MPIO_CCDGO    DEPTO             MUNICIPIO COD_MUNICI
#> 1         1         94        887  GUAINIA             PANA PANA        887
#> 2         2         91        540 AMAZONAS         PUERTO NARIÑO        540
#> 3         3         95        001 GUAVIARE SAN JOSE DEL GUAVIARE        001
#> 4         4         86        749 PUTUMAYO              SIBUNDOY        749
...

Created on 2024-02-14 with reprex v2.1.0

@JosiahParry JosiahParry merged commit 38111cf into main Feb 14, 2024
7 checks passed
@JosiahParry JosiahParry deleted the batch_size branch September 26, 2024 23:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

New argument batch_size
2 participants