Load testing new MapKnitter exporter! Help wanted!

[jywarren]

We've just published the new MapKnitter cloud exporter for the first time, in a beta mode. We're very excited to try it out before putting some final touches on it!

The design and user interface is not complete (no worries, it'll improve!), but we'd like to ask people to try running exports on some big maps to test out the exporter on some tough maps.

Edit the map

To run the export, click Edit this Map:

Select images

Then shift-drag or shift-click images you'd like to export:

Start the export

Look in the upper left of the window and click the down-arrow export button:

Choose a resolution

Wait for the completion dialog box

Copy that message and paste it into a comment below, along with anything that went wrong! Thanks a lot!

---

Example export

Please include:

1. a link to the map
2. how many images you exported
3. what resolution you chose
4. a link to the output

Map: https://mapknitter.org/maps/east-providence-pond-view-map/

• 35 images
• 28 cm/px in about 3 minutes
• Output JPG:** https://mapknitter-exports-warps.storage.googleapis.com/1585682284/1585682284.jpg

---

List of big maps you can test out (from this list):

• https://mapknitter.org/maps/lemon-fair-burn - ~30 images
• https://mapknitter.org/maps/lincoln-peak-vineyard has 48 images ~3mb each!!!
• https://mapknitter.org/maps/scott-test-bienvenue is quite large
• https://mapknitter.org/maps/otter-view-park-west also huge
• This is a tough one with 2 map areas far apart. It'll probably fail due to being too big a TIFF: https://mapknitter.org/maps/ile-perrot-1947-49

[jywarren]

https://mapknitter.org/maps/lincoln-peak-vineyard/

• 24 cm/px
• 48 images, 3mb each
• https://mapknitter-exports-warps.storage.googleapis.com/1585683543/1585683543.jpg (~3000x2000)

[jywarren]

Running last one again - https://mapknitter.org/maps/lincoln-peak-vineyard/ - at 10cm/px, in same open window...

IT WORKED! ~7000x5000 px output, @ ~5mb:

• 10cm/px
• 48 images, 3mb each
• https://mapknitter-exports-warps.storage.googleapis.com/1585684810/1585684810.jpg

[jywarren]

Now trying at 5cm/px 😅

This breaks my browser, but... it worked @ 15mb: https://mapknitter-exports-warps.storage.googleapis.com/1585685256/1585685256.jpg (14000x10000px)

it also generates a 543mb tif at: https://mapknitter-exports-warps.storage.googleapis.com/1585685256/1585685256.tif

[jywarren]

OK, so, noting that while exporting works, there seem to be some inconsistencies on local saving of export status:

• https://mapknitter.org/maps/the-mews/edit gave an alert upon completion but did not save status of most recent export within the Rails app (stuck at 2 of 22)
• i think this may be because we are fetching the status differently in these two blocks of code? We should be relying on the remote status.json file:

  mapknitter/app/controllers/export_controller.rb

  Lines 51 to 82 in b085f65

  	def progress
  	map = Map.find_by(id: params[:id])
  	export = map.export
  	output = if export.present?
  	if export.status == 'complete'
  	'complete'
  	elsif export.status == 'none'
  	'export not running'
  	elsif export.status == 'failed'
  	'export failed'
  	else
  	export.status
  	end
  	else
  	'export has not been run'
  	end
  	render plain: output, layout: false
  	end
  	def status
  	map = Map.find_by(id: params[:id])
  	if export = map.export
  	if export.export_url.present?
  	status_response = ExporterClient.new(export.export_url).status
  	render json: status_response
  	else
  	render json: export.to_json
  	end
  	else
  	render json: { status: 'export has not been run' }
  	end
  	end

• AHA! the interface shows a link to https://export.mapknitter.org/id/1586280820/status.json, but the ajax request shows https://mapknitter-exports-warps.storage.googleapis.com/1586280820/status.json -- and these show different values!!! We probably need to be flushing a cache, maybe? or updating the address once we get the first response, to be the direct to https://mapknitter-exports-warps.storage.googleapis.com version?

[jywarren]

Wow, @icarito doesn't see the discrepancy but noting here i saw a discrepancy on both browser downloads and wget

[jywarren]

OK so we can bypass by:

you can bust the cache by adding a GET parameter
as in https://export.mapknitter.org/id/1586280820/status.json?a=<timestamp>

let's do this

[jywarren]

mapknitter/app/views/maps/_cloud_exports.html.erb

Line 15 in b085f65

$.ajax(export_url).done(function displayExportStatus(response) {

[icarito]

I suppose this ought to do it: export_url + "?cachebust=#{rand(1000)}"

[jywarren]

8e20d25 - made the change here! I did it in JS

[jywarren]

Confirmed it worked... not 100% sure it busted the cache but it ought've -- we'll publish to production to confirm in a sec...

[jywarren]

Confirmed!

[jywarren]

OK, haha i think i reached the limit: https://mapknitter.org/maps/2011-08-06-brooklyn-newtowncreek-north

13960x20698 px

1.15589GB

2020-05-07 22:18:28 +0000

stalled at making a JPG from that 1.15GB GeoTiff 😂

[icarito]

Oh! What can we do? do we know where it failed? Logs should be at <https://console.cloud.google.com/logs/viewer?project=public-lab&resource=gae_app%2Fmodule_id%2Fdefault> El vie, 8 de may de 2020 a las 10:45, Jeffrey Warren <notifications@github.com> escribió:

…

OK, haha i think i reached the limit: <https://mapknitter.org/maps/2011-08-06-brooklyn-newtowncreek-north> 13960x20698 px 1.15589GB 2020-05-07 22:18:28 +0000 stalled at making a JPG from that 1.15GB GeoTiff 😂 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1244 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABQYSZWLSAJHUSN6RQWZTTRQRANLANCNFSM4LX5ER5A>.

[icarito]

Oh wait that is for the app itself not the exporter...

[jywarren]

It probably ran out of memory? I mean, imagemagick will crash with a gigantic enough image. I think not super high priority as it's probably beyond the limit of what we'd want to support. 13k x 20k pixels is a TON.