Unexpected GET Requests Count with web-optimized COG

[tebben]

I'm trying to optimize a COG for our Cesium Terrain on Demand service where we request tiles using rio-tiler with the WGS1984Quad tiling schema. I warp a tiff to 4326 and run the the create command with -w to create a COG with internal tiles aligned to the TMS.

gdalwarp -t_srs EPSG:4326 R_09BN2.TIF R_09BN2_4326.TIF

rio cogeo create R_09BN2_4326.TIF R_09BN2_4326_cog.tif --cog-profile deflate --blocksize 256 --overview-blocksize 256 --tms ./WGS1984Quad.json -w

The output works fine in our service but if I profile getting a tile with tilebench (changed to work with WGS1984Quad) it reports 3 GET requests while i would expect 2: head and tile data.

tilebench profile http://localhost:3000/R_09BN2_4326_cog.tif --tile 17-134586-26855 --add-kernels | jq

{
  "HEAD": {
    "count": 1
  },
  "GET": {
    "count": 3,
    "bytes": 212992,
    "ranges": [
      "0-65535",
      "287064064-287211519"
    ]
  },
  "WarpKernels": [],
  "Timing": 0.02144479751586914
}

Running this for a random tile on zoom 16 even reports Get->Count: 4

tilebench profile http://localhost:3000/R_09BN2_4326_cog.tif --tile 16-67300-13416 --add-kernels | jq

{
  "HEAD": {
    "count": 1
  },
  "GET": {
    "count": 4,
    "bytes": 792654,
    "ranges": [
      "0-65535",
      "41233096-41604119",
      "45029981-45386074"
    ]
  },
  "WarpKernels": [],
  "Timing": 0.024923324584960938
}

rio cogeo info reports the following info.

Driver: GTiff
File: R_09BN2_4326_cog.tif
COG: True
Compression: DEFLATE
ColorSpace: None

Profile
    Width:            14336
    Height:           10752
    Bands:            1
    Tiled:            True
    Dtype:            float32
    NoData:           3.4028234663852886e+38
    Alpha Band:       False
    Internal Mask:    False
    Interleave:       BAND
    ColorMap:         False
    ColorInterp:      ('gray',)
    Scales:           (1.0,)
    Offsets:          (0.0,)

Geo
    Crs:              EPSG:4326
    Origin:           (4.788665771484176, 53.16146850585942)
    Resolution:       (5.36441802978515e-06, -5.36441802978515e-06)
    BoundingBox:      (4.788665771484176, 53.10379028320317, 4.865570068359176, 53.16146850585942)
    MinZoom:          12
    MaxZoom:          18

Image Metadata
    AREA_OR_POINT: Area
    OVR_RESAMPLING_ALG: NEAREST
    TILING_SCHEME_NAME: WorldCRS84Quad
    TILING_SCHEME_ZOOM_LEVEL: 17

Image Structure
    COMPRESSION: DEFLATE
    INTERLEAVE: BAND
    LAYOUT: COG

Band 1
    ColorInterp: gray

IFD
    Id      Size           BlockSize     Decimation           
    0       14336x10752    256x256       0
    1       7168x5376      256x256       2
    2       3584x2688      256x256       4
    3       1792x1344      256x256       8
    4       896x672        256x256       16
    5       448x336        256x256       32
    6       224x168        256x256       64

I also tried updating rio-tiler in tilebench and adding align_bounds_with_dataset to the tile request but that didn't make any difference.

Am i doing something wrong? is the GET count as expected? Or is there something else going on?

[vincentsarago]

Hi @tebben thanks for starting this discussion

zoom 17

{
  "HEAD": {
    "count": 1
  },
  "GET": {
    "count": 3,
    "bytes": 212992,
    "ranges": [
      "0-65535",
      "287064064-287211519"
    ]
  },
}

You get 3 GET counts but only 2 Ranges, so I think what's going on is that your COG header is greater than the default GDAL_INGESTED_BYTES_AT_OPEN in you environment.

can you do

tilebench profile http://localhost:3000/R_09BN2_4326_cog.tif --tile 17-134586-26855 --add-kernels --config GDAL_INGESTED_BYTES_AT_OPEN=80000 | jq 

You can also get the size of the COG header using https://github.com/blacha/cogeotiff#command-line-interface

Zoom 16

When creating the cog, the overviews are not all aligned with the TMS, by default we only align the highest resolution (17)

Image Metadata
    AREA_OR_POINT: Area
    OVR_RESAMPLING_ALG: NEAREST
    TILING_SCHEME_NAME: WorldCRS84Quad
    TILING_SCHEME_ZOOM_LEVEL: 17

to fix this you can use the --aligned-levels option.

⚠️ the more level you align, the more ghost block you'll create (bigger COG) (https://github.com/cogeotiff/rio-cogeo/blob/main/tests/test_web.py#L240-L299)

from GDAL documentation:

ALIGNED_LEVELS=<integer>: Number of resolution levels for which GeoTIFF tile and tiles defined in the tiling scheme match. When specifying this option, padding tiles will be added to the left and top sides of the target raster, when needed, so that a GeoTIFF tile matches with a tile of the tiling scheme. Only taken into account if [TILING_SCHEME](https://gdal.org/drivers/raster/cog.html#drivers/raster/cog-co-TILING_SCHEME) is different from CUSTOM. Effect of this option is only visible when setting it at 2 or more, since the full resolution level is by default aligned with the tiling scheme. For a tiling scheme whose consecutive zoom level resolutions differ by a factor of 2, care must be taken in setting this value to a high number of levels, as up to 2^(ALIGNED_LEVELS-1) tiles can be added in each dimension. The driver enforces a hard limit of 10.

To remove the HEAD request there are other optimization you can do but to know more I would need to know what are the GDAL env set in your system.

[vincentsarago]

aligned overviews from 16 to 12

$ morecantile tms --identifier WGS1984Quad > WGS1984Quad.json

$ rio cogeo create R_09BN2.TIF R_09BN2_4326_cog.tif --cog-profile deflate --blocksize 256 --overview-blocksize 256 --tms ./WGS1984Quad.json -w --aligned-levels 4

$ rio cogeo info R_09BN2_4326_cog.tif
Driver: GTiff
File: /Users/vincentsarago/Downloads/R_09BN2_4326_cog.tif
COG: True
Compression: DEFLATE
ColorSpace: None

Profile
    Width:            20480
    Height:           16384
    Bands:            1
    Tiled:            True
    Dtype:            float32
    NoData:           3.4028234663852886e+38
    Alpha Band:       False
    Internal Mask:    False
    Interleave:       BAND
    ColorMap:         False
    ColorInterp:      ('gray',)
    Scales:           (1.0,)
    Offsets:          (0.0,)

Geo
    Crs:              EPSG:4326
    Origin:           (4.76806640625, 53.173828125)
    Resolution:       (5.36441802978515e-06, -5.36441802978515e-06)
    BoundingBox:      (4.76806640625, 53.0859375, 4.8779296875, 53.173828125)
    MinZoom:          12
    MaxZoom:          18

Image Metadata
    AREA_OR_POINT: Area
    OVR_RESAMPLING_ALG: NEAREST
    TILING_SCHEME_ALIGNED_LEVELS: 4
    TILING_SCHEME_NAME: WorldCRS84Quad
    TILING_SCHEME_ZOOM_LEVEL: 17

Image Structure
    COMPRESSION: DEFLATE
    INTERLEAVE: BAND
    LAYOUT: COG

Band 1
    ColorInterp: gray

IFD
    Id      Size           BlockSize     Decimation           
    0       20480x16384    256x256       0
    1       10240x8192     256x256       2
    2       5120x4096      256x256       4
    3       2560x2048      256x256       8
    4       1280x1024      256x256       16
    5       640x512        256x256       32
    6       320x256        256x256       64

[vincentsarago]

Fyi I've just updated tilebench to add the --tms option

$ env | grep "GDAL"
GDAL_CACHEMAX=75%
GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR
GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES
GDAL_HTTP_MULTIPLEX=YES
GDAL_INGESTED_BYTES_AT_OPEN=32768

zoom 17

tilebench profile http://localhost:8000/R_09BN2_4326_cog.tif --tile 17-134586-26855 --add-kernels --tms WGS1984Quad.json | jq

{
  "HEAD": {
    "count": 1
  },
  "GET": {
    "count": 2,
    "bytes": 196608,
    "ranges": [
      "0-32767",
      "302710784-302874623"
    ]
  },
  "WarpKernels": [],
  "Timing": 0.03943824768066406
}

Zoom 16

here you can see that GDAL needs an additional GET request for the header

tilebench profile http://localhost:8000/R_09BN2_4326_cog.tif --tile 16-67300-13416 --add-kernels --tms WGS1984Quad.json | jq
{
  "HEAD": {
    "count": 1
  },
  "GET": {
    "count": 3,
    "bytes": 294912,
    "ranges": [
      "0-32767",
      "32768-98303",
      "43614208-43810815"
    ]
  },
  "WarpKernels": [],
  "Timing": 0.06371212005615234
}

Solved by ingesting more data at open (⚠️ this will impact the performance)

tilebench profile http://localhost:8000/R_09BN2_4326_cog.tif --tile 16-67300-13416 --add-kernels --tms WGS1984Quad.json --config GDAL_INGESTED_BYTES_AT_OPEN=100000 | jq
{
  "HEAD": {
    "count": 1
  },
  "GET": {
    "count": 2,
    "bytes": 327680,
    "ranges": [
      "0-131071",
      "43614208-43810815"
    ]
  },
  "WarpKernels": [],
  "Timing": 0.06105399131774902
}

[vincentsarago]

Unrelated but, just realized that the TMS id in morecantile for WGS1984Quad is set to WorldCRS84Quad developmentseed/morecantile#143 😓

[tebben]

Oh wow, thanks for the quick reply and adding the --tms option to tilebench.

On my system no GDAL environment variables were set, I just set them using your settings:

GDAL_CACHEMAX=75%
GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR
GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES
GDAL_HTTP_MULTIPLEX=YES
GDAL_INGESTED_BYTES_AT_OPEN=32768

I reran create with --aligned-levels 4 and cogeo info has the same output as you. Running profile for zoom 16 and 17 has exactly the same output as you.

I tested setting GDAL_DISABLE_READDIR_ON_OPEN to FALSE and I can spot 3 counts for GET on zoom 17 instead of 2.

Increasing GDAL_INGESTED_BYTES_AT_OPEN to 100000 indeed lowers the GET count to 2 but results in slightly more bytes ingested.

So using --aligned-levels prevents more tile request at lower zooms and the GDAL settings can prevent multiple GET requests for the header.

When rio-cogeo creates the overviews automatically, is there a way to also automatically align all levels instead of passing --aligned-levels?

[vincentsarago]

When rio-cogeo creates the overviews automatically, is there a way to also automatically align all levels instead of passing --aligned-levels?

Sadly no 🙁, this could have been a good idea but we tried to match GDAL options. I would also note that doing so might not be a good idea (except if you have small COGs) because you'll end up creating bigger COG so bigger header. Sometime it's better to have less aligned overview level and pay the price of one or two additional GET request.

[tebben]

Ok clear, thanks for the clarification!