Fix access to remote data sources in WebAssembly builds

[wonder-sk]

QGIS Enhancement: Fix access to remote data sources in WebAssembly builds

Date 2024/03/21

Author Martin Dobias (@wonder-sk)

Contact wonder dot sk at gmail dot com

Summary

QGIS core library compiled to WebAssembly is showing a lot of promise, enabling QGIS code to be run in web browsers. At this point, it is possible to load a QGIS project and render the map entirely in the client's browser, without any server-side map rendering - see the demo website at https://qgis.github.io/qgis-js/ with some sample projects.

While many engineering challenges towards production-ready state have been solved already, there are still issues that need addressing. Among those issues are problems with data sources that require network requests from QGIS and GDAL code in order to work, such as:

• Cloud-optimized GeoTIFFs accessed remotely through HTTP(S)
• vector tiles and XYZ raster tiles
• WMS or WMTS.

Currently, QGIS projects with any of these data sources will simply fail to load. All of these data sources are commonly used in QGIS projects, so not having access to them is a significant limitation. The purpose of this QEP is to solve the problems related to network requests so that QGIS projects involving these load and render correctly.

Proposed Solution

The issues with network requests have several causes, with at least three different problems to solve:

• vector tiles and XYZ raster tiles (Vector / raster tiles layer throws error when rendering qgis-js#3) - the problem is in the QgsNetworkAccessManager code and it will need further investigation. In any case the network access manager code for WebAssembly builds should be simplified and there are various configuration bits that do not make sense in the browser environment (e.g. setting up disk cache, effectively duplicating browser's cache). By cleaning up some of the network access manager code, we are expecting that the failures to lock an internal mutex can be solved.

• WMS and WMTS (Loading a WMS/WMTS layer causes infinite loop qgis-js#2) - the problem with WMS/WMTS data provider is that it does a GetCapabilities network request in the data provider's constructor, and blocking the main thread while the request is in progress. This is a no-no in browser environment and the main thread must not block for extended period of time. This problem should get fixed by allowing WMS provider to be created in a worker thread (using ParallelCreateProvider flag) and by inspecting the code for possible problems the flag may introduce. A nice side effect will be that QGIS projects with WMS/WMTS also should load faster in the desktop builds.

• Cloud-optimized GeoTIFFs (Raster layer with a remote COG fails to load qgis-js#4) - for the purpose of QGIS in WebAssembly, the vsicurl handler (that is responsible for remote fetching) has been patched to use emscripten's fetch API (because curl library is not available for emscripten). While in theory things should work fine (gdal provider already supports ParallelCreateProvider flag, allowing its creation in worker threads), the requests crash and this will need further investigation.

Deliverables

Deliverables should be pull requests in qgis-js repository fixing the three issues outlined above. The work is successful when QGIS projects with vector tiles, raster tiles, WMS, WMTS or remote COGs can be loaded and visualized without errors.

Risks

This is slightly more risky project - as it can be seen from the proposed solution, some of the problems may need deeper investigation. Working with WebAssembly builds is not as smooth and polished as with native builds, and problems may be difficult to diagnose at times.

Nevertheless, it seems to be a problem worth tackling - especially as more and more traditional desktop apps are getting ported to web browser environment with the help of WebAssembly (Google Earth, Adobe Photoshop, Autocad, ...) to provide the functionality with less friction - no need to install anything!

Performance Implications

No negative performance implications are expected. On the contrary, loading of QGIS projects with WMS/WMTS layers may get faster.

Issue Tracking ID(s)

• Loading a WMS/WMTS layer causes infinite loop qgis-js#2
• Vector / raster tiles layer throws error when rendering qgis-js#3
• Raster layer with a remote COG fails to load qgis-js#4

[rouault]

Deliverables should be pull requests fixing the three issues outlined above

In which repositories ? I'm in particular interested regarding the GDAL issues. Is the https://github.com/wonder-sk/wasm-gdal-sandbox fork a long term or interim solution? Not saying that upstreaming must be necessary a result of this work (it might be additional work to have a "web-assembly friendly curl emulation/replacement" in a state ready for upstream inclusion), but just to clarify the intended scope.

[wonder-sk]

In which repositories ?

I should have been clear about that, sorry - I meant pull requests in https://github.com/qgis/qgis-js repo.

Is the https://github.com/wonder-sk/wasm-gdal-sandbox fork a long term or interim solution?

I see it as an interim solution, a proof of concept, just to get things off the ground. There are many shortcomings of the current approach, so my preference would be to have curl re-implemented for WebAssembly/Emscripten, to avoid having to patch vsicurl in gdal. But of course that would be a larger undertaking (but it could benefit other projects).

[vpicavet]

While the topic is interesting, I do not see how it corresponds to the criteria for a QGIS Grant ?

[wonder-sk]

While the topic is interesting, I do not see how it corresponds to the criteria for a QGIS Grant ?

@vpicavet Could you please elaborate on why this would not correspond to the grant criteria?

The "Admissible project topics" section of grant proposal form states the following: "Proposals may be on any topic that you think is relevant and beneficial to the greater QGIS community. However, the grant programme does not accept proposals for the development of new features. Therefore, proposals should focus on improving the QGIS project infrastructure and polishing existing features."

To my best knowledge, this proposal is relevant and beneficial to QGIS community and it does not suggest development of new features...

[andreasneumann]

@vpicavet - I see QGIS wasm as a part of the QGIS ecosystem, just like QGIS Desktop, Server, Web Client or other parts of the (Q)GIS infrastructure. It is still in it's early days, but has a good potential and - once the basic problems are solved - it will trigger quite some investments in feature development - but it needs a bit of a "jump start" (initial aid) until it can be seen as a viable solution for future Online-GIS development. Once that is solved, I am pretty sure, that other QGIS users will chime (financially) in the future.

[vpicavet]

Adding COG/WMS/... support to QGIS WASM sounds like a feature implementation to me, that's why I am wondering if this would fit to a grant application.

[andreasneumann]

Adding COG/WMS/... support to QGIS WASM sounds like a feature implementation to me, that's why I am wondering if this would fit to a grant application.

No - the functionality is already there (from QGIS desktop). It is about fixing the issues and dead-locks and threading-issues to get this running inside wasm, and hence in the web browser, without installing it locally.

[vpicavet]

We probably have different views of what would be called a fix and a feature. This would need sitting around a drink to discuss more thoroughly though.

[andreasneumann]

I think that quite a few of the QEP proposals are sort of "edge cases" in this respect.

[anitagraser]

Thank you for submitting your proposal to the 2024 QGIS Grant Programme. The 2 week discussion period starts today. At the end of the discussion, the proposal author has to provide a 3-line pitch of their proposal for the voter information material. (For an example from last year check qgis/PSC#58 (comment))

[nyalldawson]

I'd argue that investment in stablising webassembly QGIS is a sensible risk management approach for the project. I'd personally say that there's a good chance that WebAssembly based builds of QGIS will become the dominant form of QGIS over the next decade, and I'd rather we proactively keep exploring this instead of having to scramble to catch up when the demand rises... 👍

[pathmapper]

Here's a nice presentation from last week. Although it is in German, it also gives an impressive visual impression of the current status of WebAssembly QGIS:

https://media.ccc.de/v/fossgis2024-38328-qgis-js-qgis-im-browser-dank-webassembly

[jjimenezshaw]

There is a detail in PROJ: https://proj.org/en/9.4/apps/cs2cs.html#use-of-remote-grids
PROJ can use remote grids via CURL (similar but different problem as in GDAL).
I would like to develop proj in web assembly (standalone), but first I have to learn WA.

[wonder-sk]

There is a detail in PROJ: https://proj.org/en/9.4/apps/cs2cs.html#use-of-remote-grids
PROJ can use remote grids via CURL (similar but different problem as in GDAL).

The automatic fetching of data from remote grids in PROJ is explicitly disabled in QGIS, because this could have negative performance impacts on rendering, and users are instead guided to get the whole grid downloaded.

Good point though about PROJ using CURL as well, as this could be another project to benefit from CURL port to WebAssembly... The ticket for CURL implementation in Emscripten does not show much progress so far: emscripten-core/emscripten#3270