A plan for future-proof tile URLs

[migurski]

We are starting to see versioned/dated URLs in sharedstreets/sharedstreets-builder#10 (comment). @kpwebb and I talked by phone to see if there was a sensible URL structure that SharedStreets could commit to for tiles going forward. It would provide immutability so that tile consumers could be reassured of stable identifiers, links to the source OpenStreetMap data so OSM IDs could be unambiguously referenced, and an index format so that clients could choose when to move forward. As with OSM Planet, latest would be a special date to indicate the newest copy of the data.

Here’s what a sample JSON index with URI templates might look like:

[
  {
    "url_template": "https://tiles.sharedstreets.io/planet-latest/{z}-{x}-{y}.{layer}.pbf",
    "planet_url": "http://planet.openstreetmap.org/pbf/planet-180319.osm.pbf",
    "planet_date": "latest",
    "version": "0.2.0"
  },
  {
    "url_template": "https://tiles.sharedstreets.io/planet-180319/{z}-{x}-{y}.{layer}.pbf",
    "planet_url": "http://planet.openstreetmap.org/pbf/planet-180319.osm.pbf",
    "planet_date": "2018-03-19",
    "version": "0.2.0"
  },
  {
    "url_template": "https://tiles.sharedstreets.io/planet-180312/{z}-{x}-{y}.{layer}.pbf",
    "planet_url": "http://planet.openstreetmap.org/pbf/planet-180312.osm.pbf",
    "planet_date": "2018-03-12",
    "version": "0.1.0"
  }
]

We also talked a bit about different build levels for tiles based on road class. Right now, the SharedStreets tiles include all roads down to small service alleys and parking lots. A tile URL might include a minimum road class to slice the data:

• All roads: https://tiles.sharedstreets.io/planet-latest/{z}-{x}-{y}.{layer}.pbf
• Roads at and above a specific class: https://tiles.sharedstreets.io/planet-latest/{z}-{x}-{y}.{layer}.{roadClass}.pbf
  • Residential streets and larger: https://tiles.sharedstreets.io/planet-latest/{z}-{x}-{y}.{layer}.6.pbf
  • …
  • Motorways only: https://tiles.sharedstreets.io/planet-latest/{z}-{x}-{y}.{layer}.0.pbf

Feedback on this proposal welcome!

/cc @hassoncs

[DenisCarriere]

@migurski 👍 Looks rock solid!

I really like the fact that you could extract only certain layers, that would benefit low bandwidth connectivity and only deliver the data that's needed.

It could be interesting to play with.

👍 I like the URI Template sample

[migurski]

Thanks @DenisCarriere! I’m excited to see this move forward, and it’ll be possible to do a lot of cool stuff wth an internally-consistent planet. URLs are currently a blocker for one PR: sharedstreets/sharedstreets-python#7

[kpwebb]

@migurski and @DenisCarriere this looks great from my side. I can start working on the layer implementation now -- we can already generate these using the builder app.

Two additional ideas:

1. what do you all think about adding a tile source to the URL schema to allow for tiles from OSM and other data providers (e.g. public authorities, proprietary maps willing to share their references, etc.). Something like this:

https://tiles.sharedstreets.io/{source}/{tileset-version}/{z}-{x}-{y}.{layer}.0.pbf

So OSM would look like:

https://tiles.sharedstreets.io/osm/planet-latest/{z}-{x}-{y}.{layer}.0.pbf

2. I'm also curious about adding a really simple header line to each tile that says where it comes from. Could be as simple as:

{ source: "osm/planet-latest" last_modified: {timestamp} }

I think everything else is in the file name but could make the z/x/y +layer part of the header to improve legibility.

[DenisCarriere]

👍 I like including {source} to easily expand to other data sources.

I'm thinking maybe the {roadClass} might a bit too much customization, that's going to be a lot of maintenance to host new datasets if we also have support an additional 6+ road classes pbf datasets.

I'd stick with the essentials at first, and if we see the need to include {roadClass} because the file sizes are getting too big, then we can include it later on.

URL Template

https://tiles.sharedstreets.io/{source}/{tileset-version}/{z}-{x}-{y}.{layer}.pbf

I'm also curious about adding a really simple header line to each tile that says where it comes from. Could be as simple as:

👍 Sounds like a good idea, would that be stored in the PBF? Or stored as a *.json file?

We could host two files per tile:

• *.pbf (tile data)
• *.json (metadata)

[josiekre]

I've been following along as you carry out this discussion. I thought I would throw my two cents in.

I think @migurski's proposal to handle {roadClass} layers is elegant. My development team struggles to handle displaying, for example, volume data on all streets when zoomed out on a map. We have brainstormed in the past and thought splitting road classes into different layers would solve the problem so that data-driven styling is snappier no matter the zoom.

I think it would make more sense from a user perspective too. When zoomed out far, I see only traffic volumes (traffic counts) on motorways and trunks, for example. As I zoom in, I see traffic volumes on local roads.

This was actually what caught my attention and had me following this issue.

[migurski]

+1 to {source}, looks good.

I’ll say a bit more about {roadClass} to explain where the idea is coming from. Right now in our Remix work, we're finding that tiny non-roads like parking lot aisles, alleys, and driveways are interfering with our use of the graph. We’ve looked at filtering them out, but it’s not always possible to do this correctly near tile boundaries. It’d be easier to have multiple levels of road class cutoff, allowing us to pick and show all driveable roads.

[kpwebb]

Just to second @josiekre and @migurski's comments about {roadClass} We need to be able to build complete SharedStreets references for various levels in the road hierarchy for a variety of reasons, and because of the tile boundaries it's not easy to do filtering on the client. We can automatically generate these alternative tiles during the build stage.

A few clarifying questions:

1. @migurski can you share a bit more about how you imagine the {layer} attribute working? I get the {roadClass} (driven by the OSM tag classification) but not sure I understand how the layer attribute works

2. How should we handle segments with multiple OSM classifications within the segment? Right now anything that has two or more mergeable classifications we're combining them and labeling it as unclassified. It's a bit of a hack but perhaps we can get away with it because it's just an edge case.

[highway=primary]     [highway=secondary]
===================*======================
         [roadClass=unclassified]

As we start removing roads with excluded classes and merging things that were previously real intersections, I suspect that this will pop up frequently:

              [excluded road]                 
                    | 
[highway=primary]   |  [highway=secondary]
====================*=====================
        [roadClass=unclassified]

Should we change the underlying "unclassified" concept to something more robust? Perhaps we categorize as the highest class road in the segment but flag that theres more than one in the segment?

3. As an FYI, we also have this "form of way" concept that we're including in the references. This borrows from OpenLR and is included primarily for compatibility with the OpenLR spec, however, I'm curious if it is useful as a layer of abstraction that sits on top of the OSM tags?

[migurski]

{layer} comes directly from the current tile URLs with one of the four possible tile name extensions: “reference“, “intersection”, “geometry“, and “metadata”.

The way I was imagining classification, it would happen prior to conversion from OSM via a filter. In the example above, there might be a tile level with the highway=primary included, and the highway=secondary excluded. That might lead to a missed intersection, which I’m unsure is important. It’s one of the many cases where some graph awareness is necessary to merge different SharedStreets data sources together.

[kpwebb]

@migurski doh! of course! sorry for being dense about the layer thing.

This approach to creating the roadClass tiles works great - the builder already roadClass filter ways during the OSM ingest and we can just iterate through the list of possible filter options:

https://github.com/sharedstreets/sharedstreets-builder/blob/master/src/main/java/io/sharedstreets/tools/builder/osm/OSMDataStream.java#L244

I guess my main question is if the you're thinking of the roadClass filter as being something like highway=primary or greater (how it currently works) or if it should filter exclusively filter out one class per tile?

If it's, "{roadClass} or greater" I'm curious to get your take on how we should merge connecting segments of different classes (per point 2 above)?

Currently we merge different classes together, but we could switch to keeping each class discreet. We're merging aggressively, and creating mixed roadClass references to prevent poorly mapped OSM ways getting split up when they should be a contiguous reference.

I don't have a strong opinion about how do this, and am curious to get other takes on how to handle

[migurski]

I’m thinking of it as highway=primary or greater. If you got residential streets, you’d also get primary roads and motorways in the same tile. The reverse might not be true to traverse just graphs of highways and major roads.

Skipping a few levels for brevity, it could look like this:

• Minor roads, major roads, and motorways: https://tiles.sharedstreets.io/planet-latest/{z}-{x}-{y}.{layer}.6.pbf
• Major roads and motorways: https://tiles.sharedstreets.io/planet-latest/{z}-{x}-{y}.{layer}.3.pbf
• Motorways only: https://tiles.sharedstreets.io/planet-latest/{z}-{x}-{y}.{layer}.0.pbf

[josiekre]

+1 @migurski

cc: @gregmacfarlane (my colleague who might want to weigh in)

[kpwebb]

@migurski @josiekre Fantastic -- this works for me!

Any thoughts on the question about combining different roadClass segments together per:

              [excluded road]                 
                    | 
[highway=primary]   |  [highway=secondary]
====================*=====================
               [roadClass=??]

[migurski]

I’d choose either the higher or the lower value for this case — unsure if that would be roadClass=2 (primary) or roadClass=3 (secondary). Probably the latter, since it’s the most limiting class along the segment.

So,

              [excluded road]                 
                    | 
[highway=primary]   |  [highway=secondary]
====================*=====================
               [roadClass=3]

[kpwebb]

@migurski Thanks for all the feedback -- I've taken a stab at implementing this as described above. For testing purposes I'm producing tiles for each classification breakpoint.

Just from experimenting with SF data it looks like a lot of streets are "highway=unclassified" which means that they only show up in level 6 or below.

There are also a lot of surprises in the higher levels (e.g. Market is mapped a tertiary route through most of SOMA...), so we may end up just sticking with level 6 as the primary layer until we can use this is drive improvements in the OSM QA process.

Screenshot of level 6 tiles for SF:

I'm attaching a zip of all the tiles (level 0-7) for SF. Once you have a chance to take a look, let me know and I'll kick off the process for a global tile build.

sf_test_tiles.zip

[josiekre]

@gregmacfarlane and I will be in the same workspace next week and plan to dedicate an hour or two to dive into SharedStreets repos/documentation for CityCast. We'll be brainstorming then about this topic, and the SF tiles will be helpful.

Greg and I both agree that the OSM data with regards to road class are still messy enough that maybe this discussion is putting the cart before the horse, but it's true that developing this could improve the OSM QA process in a really good way.

[migurski]

Looking good Kevin! I’ve downloaded the tiles and we’ll be able to test in SF. Level 6 as the primary use case sounds right to me as well.

[kpwebb]

Just reflecting on this further: I wonder if the main change here is just moving the default SharedStreets threshold from 7 to 6 (excluding "highway=service" streets).

We can go ahead and generate the higher threshold tiles (e.g. primary/motorway only tiles), but given the inconsistencies in OSM I'm thinking that for simplicity we should get folks to use the level 6 tiles as the core reference. Thoughts?

We'll also need to figure out how to improve mapping of the "highway=service" streets -- the big challenge with that data is it combines private driveways with public alleys and some actual public "service" roads. I don't see any quick solutions, but I suspect lot of this comes down to improving OSM QA which I think we can help with over the medium/long-term.

[migurski]

I gave the class=6 tiles a try, and they look great. I’d love to see these become the default.

[kpwebb]

@migurski Fantastic! I'll go ahead and kick off the tile build with 6>, 4> and 2> (Uncategorized>, Tertiary>, and Primary>).

[migurski]

Great! Any chance of an ETA on tiles for this? They’ll be immediately useful to us if we can try them by Friday.

[migurski]

Visual preview:

[kpwebb]

@migurski Super fun!

Global tiles are building (usually takes about 24 hrs) so I expect them to come off the press this evening. I'll let you know when we kick off the copy to S3.

[kpwebb]

@migurski the S3 copy is still running but should complete in the next few hours.

As the copy completes tiles should become visible at:

https://tiles.sharedstreets.io/osm/planet-180312/12-2445-1223.geometry.6.pbf

FYI, this tile only has roadClass=6 tiles. We're building another set with a more recent PBF file that includes 6, 4, and 2. I'll post those tomorrow evening.

[migurski]

Thought I’d bump this issue since it’s been a while since we’ve talked about long-term stable IDs. We’re still really interested to see a time-based index of static tile sets, the first part of the thread above. The 180312 planet has been working well, but eventually it’ll be necessary to migrate forward and determine a cadence for updates.

[josiekre]

Visual preview:

Is there hints/docs/bread crumbs anywhere in the repos to create this experience using these tiles? I'm very familiar with Mapbox GL technology, but I don't see how ShSt pbf tiles come in. I'm circling the repos for bread crumbs of UI examples, but coming up short.

Specifically I'm wondering how to avoid passing geojson to MapboxGL. In my experience this can really slow things down with big networks. Currently we are prebuilding vector tiles using tippecanoe from geojson files (at specific zoom levels by functional class), but is there a way to skip that? This gif seems to suggest yes.