Planning a new Exporter UI - tracking export processes

[jywarren]

The current Exporter UI is OK, but we could substantially improve it and will have to reconfigure it while implementing the new Exporter project in #298 -- currently, this is what it looks like:

Once the export is started, we see a progress bar:

Try it out here: https://mapknitter.org/maps/cranston-test

Subprojects

• Exporter JS UI to initiate, track progress (in progress bar) and report completion - see #
• list of exports run in this area and by whom
• ability to set intended export resolution with a slider
• ability to calculate a range of resolutions to offer based on selected images (see multiple image collection actions (like export) Leaflet.DistortableImage#29 for multiple image selection in LDI),
• think about where a function like averageResolution(images) should live
• fetch and display list of recent exports, time of export, duration, author, # of images (this might not need to be in a separate lib, but a view from the MapKnitter Rails app?)

Explore stand-alone JS library to manage exporting

Perhaps we could have an interface which lets you select a set of images, fetch their corner coordinates, and submit a Cloud Exporter (#298) job, then track its progress. The exporter UI need not have any other connection to the Rails app, and we could isolate its code in a pure JavaScript library, even independent from this repository -- maybe a mapknitter-exporter-ui node module?

@sashadev-sky check it out!

[jywarren]

[jywarren]

Thinking about removing the Exporter model code, which really just tracks the state of the export. We are going to have to convert it somehow to a system where we fetch the status from a remote URL, since we'll be running everything off-app.

So we'd have to change this file: https://github.com/publiclab/mapknitter/blob/master/app/views/export/formats.html.erb

to not rely on the Export model columns.

I'm tempted to start by simply refactoring the status we display to check for the presence of specific files, and to add a step to the exporting process to generate a static file indicating the current status, like status.json which we just generate with something like system("''status': 'complete'' > status.json") or something like that.

Then the formats.html.erb file would be replaced by a JavaScript call to some known locations like /warps/map-slug/format/status.json -- where format could be jpg, for example. Then when these start being remote, we can just similarly generate those status.json files remotely and the interface doesn't care.

[jywarren]

I think we'd still want to keep the Export model, to track things like who initiated the exports, where the resources for each type are, etc. But we'd want to not use it to report export progress for long-running processes.

https://github.com/publiclab/mapknitter/blob/master/app/models/export.rb

[jywarren]

Noting here to @sashadev-sky -- regarding potential React use, I'm interested in how the image manager you've proposed would either be completely self-contained (using React or not) so that it can be mixed with other applications and projects as a general-purpose image collection manager plugin, OR if it could be built without React, but in such a way as to allow us to use it within a React component. I'm kind of thinking this through and please offer your thoughts on how to approach this so that we don't /require/ people to be using React to use this plugin. Love to hear your thoughts!

[jywarren]

Also copying in notes from #364 -- where the test dataset will help us model how the export process changes will go.

Basically in the first version, we'd pass the content (or URL) of a file like https://mapknitter.org/maps/ceres--2/warpables.json to one of our export scripts, and let it run;

We'd get a URL back to check on the status of the export -- something like https://remote-exporter.com/unique-id/status.json - whether it's JS or Ruby based! Finally, as it completes, that JS file would report the locations of the export outputs. Our Exporter UI would handle the generation of the initial export request, and display the status as it runs, finally offering links to the downloads. It will also report the output URLs and the initial unique URL for the whole export, back to the Export model.

[sashadev-sky]

@jywarren starting the refactoring with a JSON file sounds good to me! I need to catch up on the most recent changes to the exporting models because I saw you began working on it already.

React - for that we can wrap it to use it as a regular node component if it's built in React and vice versa. Still need to weigh the pros and cons for each approach. I definitely agree we should do it in a way that doesn't add the React DOM as a requirement for running this app. First we should get all other frontend / DOM manipulation dependencies we plan to use implemented and working together efficiently.

For the test dataset - I need to implement the ability to calculate the bounds of selection of multiple image select (currently the code I wrote to start off just returns every image on the layer) so I agree this will eventually be interconnected.

Just mapping out some initial thoughts. Let me know if I am missing anything!

[jywarren]

OK @icarito had a real breakthrough idea on Friday - that the remote export process would, as soon as it begins, create a cloud-storage object for a status.json file! And then as it proceeds, it'd continuously overwrite that file with updated status. Then we actually don't need to handle requests for progress in the exporter itself; we'd just report back the location of the status.json file as soon as it's generated, and that resolves the initial export request call. Then our local client would keep polling the status.json file (which costs us very little vs. having to handle those requests from the export process) and as export products are generated, they appear in the status.json file.

For starters, we can probably use a non-React UI just for the prototype implementation. It could show a Bootstrap progress bar maybe, and the links to the export products as they become available... something really simple. And then build out from there.

Sound good?

[jywarren]

I was thinking of this again today and thought of a good workflow - so we have to store records of the exports, and I had thought we'd do so in the existing Export model. We can add a few columns, like the URL of status.json, and we can use one of these libraries to fetch the properties inside it and report things like status:

• https://github.com/DmitryTsepelev/store_model
• https://github.com/remiprev/her

Opening a new issue to break this out.

[sashadev-sky]

@jywarren Ok sounds good! I am finishing up with multiple image select so we can discuss perhaps after the open call what I should jump into next? I will review above comments and links. Thank you!

[jywarren]

Yes, i guess we should look to complete multiple image select, and then develop a minimal interface to send a request to a remote exporter service and track the results. 1. What we could do short-term is to create a new "status" action in the export controller of MapKnitter ( https://github.com/publiclab/mapknitter/blob/main/app/controllers/export_controller.rb#L51) to translate the current exporter records into the standard status.json format, so that we can point this new multiple-select UI at the existing export system? 2. This might require also allowing parameters to override the list of images it runs on... here, i gave this a try: #415/ -- you might need to build on this to make it work? What do you think?

…

On Tue, Mar 19, 2019 at 8:51 AM Sasha Boginsky ***@***.***> wrote: @jywarren <https://github.com/jywarren> Ok sounds good! I am finishing up with multiple image select so we can discuss perhaps after the open call what I should jump into next? I will review above comments and links. Thank you! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#326 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ6uHcdyB3d9zfyUeU7Xr6gyKs3u0ks5vYN07gaJpZM4an6FJ> .

[sashadev-sky]

@jywarren right that sounds like a good next step. I will also open up the UI for handling multiple images issue shortly.

For the multiple image select I will make a GIF now so you can tell me what you like or want improved. Only thing left to do is connect the zoom box selector implementation in case the person has a lot of images on the screen and wants to select them quickly.

I saw a couple issues open suggesting using shift + drag for other purposes, though, did you want me to use it for this or leave it for something else? Either way I can leave the class there and it'll be easier to implement for anything else because its default functionality is already overriden

[jywarren]

Shift drag, and also click, then shift-click to add a 2nd to the selection, both sound pretty good! How does one initiate the zoom box drag? I'll go look at your gifs, now, sorry!

…

On Tue, Mar 19, 2019 at 2:27 PM Sasha Boginsky ***@***.***> wrote: @jywarren <https://github.com/jywarren> right that sounds like a good next step. I will also open up the UI for handling multiple images issue shortly. For the multiple image select I will make a GIF now so you can tell me what you like or want improved. Only thing left to do is connect the zoom box selector implementation in case the person has a lot of images on the screen and wants to select them quickly. I saw a couple issues open suggesting using shift + drag for other purposes, though, did you want me to use it for this or leave it for something else? Either way I can leave the class there and it'll be easier to implement for anything else because its default functionality is already overriden — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#326 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ66Coao8Ql1T1E5mVb7Sgz0EMTYFks5vYSvcgaJpZM4an6FJ> .

[sashadev-sky]

@jywarren I just set the PR ready for review but I thought I would mention here too - I kept shift drag in the PR but I commented out the event listener for now and updated the README to remove what I had mentioned about it. There is a bug with event propagation that got pretty complicated for me to solve and I don't think merging without this feature takes away from core functionality too much. Please let me know your thoughts!

[icarito]

Hi @jywarren @sashadev-sky I've prepared a Sinatra endpoint to POST a export metadata json, made a webform too:

We then need to pass this info to https://github.com/publiclab/mapknitter-exporter and integrate it into the Sinatra app.

I hope this interface is a good idea to keep the api simple!

[icarito]

^^ when it says "such as this" it is a link to this json file.

[jywarren]

Awesome!

• https://github.com/publiclab/mapknitter-exporter-sinatra - Ruby app!
• https://github.com/publiclab/mapknitter-exporter - Ruby gem!

[jywarren]

Hi @sashadev-sky -- recapping our conversation today:

Part 1: initiating an export

Fetching image URLs, corner coordinates, and scale, and sending them in JSON format to the exporter following the format in publiclab/mapknitter-exporter-sinatra#1 (comment)

Part 2: reading progress from an export

Currently, MapKnitter shows the export progress bar in the export pane, and the outputs in this dropdown:

The dropdown is driven from this template: https://github.com/publiclab/mapknitter/blob/main/app/views/export/formats.html.erb

And progress bar and status text are fetched from this method:

mapknitter/app/controllers/export_controller.rb

Lines 51 to 67 in 784bfbc

	def progress
	map = Map.find params[:id]
	if export = map.export
	if export.status == 'complete'
	output = 'complete'
	elsif export.status == 'none'
	output = 'export not running'
	elsif export.status == 'failed'
	output = 'export failed'
	else
	output = export.status
	end
	else
	output = 'export has not been run'
	end
	render :text => output, :layout => false
	end

To transition to the new system, we should make a new export controller method called status, which returns JSON in the following format, where false means it's not been generated yet:

{
  status: "distorting",
  jpg: false,
  zip: false,
  geotiff: "url/to/geo.tiff",
  tms: "url/to/tms"
}

Let's get the UI displaying this in the new format, and we'll be better prepared to integrate it with a remote exporter when the time comes. Then we can think about storing the URL of the status.json file within the export model itself.

Does this make sense, @sashadev-sky ? Thank you!

[jywarren]

Actually it could be worth just trying to run:

  def status
    @export = Export.find(params[:id])
    render json: @export.to_json
  end

[jywarren]

And hmm, do exports have usernames? They should!

[sashadev-sky]

@jywarren looking this over tonight / more in depth tomorrow

[sashadev-sky]

@jywarren ok so to test how this is working do I need to clone the sinatra app? Sorry just need a little help getting started honestly

[icarito]

Hi @sashadev-sky - I will work on integrating publiclab/mapknitter-exporter-sinatra#1 the exporter code into a very simple JSON parsing Sinatra app (my first Ruby, based on "hello world"). The idea is to POST json with metadata and obtain a status.json location.

[jywarren]

I think you could use a static `status.json` file in this format, stored locally, and point your code at it. Then manually change the different values and save the file, and your code should re-fetch it -- that can simulate everything until we're ready! And, that is probably a good idea to build tests around, too, does that make sense?

…

On Tue, Apr 2, 2019 at 11:46 PM Sasha Boginsky ***@***.***> wrote: @jywarren <https://github.com/jywarren> ok so to test how this is working do I need to clone the sinatra app? Sorry just need a little help getting started honestly — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#326 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ7kfBF-cQrZ87510a84sSIsSbgSQks5vdCPbgaJpZM4an6FJ> .

[sashadev-sky]

@jywarren Yes I think so I am adding a PR for this shortly :)

[jywarren]

Hi, @divyabaid16 is this useful to look over? Thanks!

[divyabaid16]

Yes, I will look into this. Thanks!

[jywarren]

We're doing really well here now! We still don't have:

• slider for resolution
• name of who started the export
• export duration
• exports near here... is this helpful?