Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Let library users handle output however they like, and add flatgeobuf output #51

Merged
merged 4 commits into from
Jul 28, 2023

Conversation

dabreegster
Copy link
Owner

3 functional changes here:

  1. Change the public API to output features through a callback, letting the caller do whatever they like. Closes Revisit library API and consider streaming #50.
  2. Add an option to output the result as flatgeobuf instead of geojson
  3. Huge speedups -- from 58s to 22s for one large benchmark -- just from upgrading to latest geo packages!

I used the input data from #5: time cargo run --release -- disaggregate --od-csv-path ~/Downloads/od_sp_center.csv --zones-path ~/Downloads/zones_sp_center.geojson --output-path result.fgb --output-fgb

Before this PR, output to geojson took 58s. After this PR, down to 22s!

FGB output takes much longer, 50s. Not sure why yet, though I have a few ideas. The resulting file is much smaller -- 735MB FGB, 1.4GB GeoJSON. I haven't checked downstream yet to see which is faster to read, or if reading is even the bottleneck in a larger pipeline.

This is a step towards #49 (comment) improving the API for language bindings, but the input is still a bunch of files.

Copy link
Owner Author

@dabreegster dabreegster left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I considered using generators to wind up with an iterator-like API, but

  1. Generators aren't in stable Rust yet, and the 3 crates I looked at were a bit unmaintained
  2. Generators wouldn't make it easy for us to plumb back errors or halt execution
  3. The callback is not hard at all to use

@dabreegster dabreegster merged commit a756bd4 into main Jul 28, 2023
1 check passed
@dabreegster dabreegster deleted the callback_api branch July 28, 2023 12:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Revisit library API and consider streaming
1 participant