Supporting locations_on_ways format extension?

[mmd-osm]

Since libosmium has support for locations on ways, I was wondering if this format extension might be relevant for osm2pgsql as well, at least for an initial load.

The toy example below was the bare minimum needed to import such a PBF file. I'm sure, a more sophisticated approach is possible by evaluating the PBF header pbf_optional_feature, which should include an LocationsOnWays in that case.

Calling osm2pgsql using: -r pbf,locations_on_ways=true

middle-ram.cpp:

Added in set_requirements (for testing purposes only)
m_store_options.way_nodes = false;

line 200, nodes_get_list

if (m_store_options.locations) {
    for (auto &nr : *nodes) {
        if (nr.location().is_undefined())     <<< added check. locations_on_ways pbf can skip set_location
          nr.set_location(m_node_locations.get(nr.ref()));

[joto]

I looked into this a while back and it isn't as easy to implement as one might think, mostly due to the way this would interact with the update mechanism. It isn't even clear how updates would work when locations on ways are used. We could probably implement something that only works in non-slim mode though.

In the end I suspect that this really isn't that complicated if we limit the scope, but it needs to be thought through well, we don't want to complicate our code for a niche use case. For now the priority is very low though. If the OSMF goes forward with the data model changes this will probably need some looking into.

[mmd-osm]

Thanks for your insights, creating an issue wasn't really meant to "escalate" anything.

I was curious if your work on data model changes would at least include a high level estimate on the impact, feasibility, and maybe even effort of different proposals on various tools. Since the blog post explicitly mentioned the location on ways topic and you're very familiar with the osm2pgsql internals, this seemed like a natural fit. Maybe my understanding of what you had in scope for the study is just a bit different.

[lonvia]

The file format requires fundamental changes to how we process the data even for the most simple case of non-slim mode without any second stage processing: When processing relations, osm2pgsql currently collects way node IDs from the middle way storage and then coordinates for each ID from the middle node storage. This is kind of baked in into the design, there are currently no data structures in place to save coordinates of ways with the way itself.

I'm not saying that it is a stupid idea to have the coordinates with the ways. On the contrary, it might be a good idea in the long run performance-wise, no matter what kind of PBF is used as input. I'm just saying that support for the location_on_ways format requires writing a middle storage that functions very differently from the existing ones. It's not just an add-on.