Using fsspec in flask-socketio views

[rob-ashdown-monolith]

I am having problems using fsspec-based filesystems (e.g. adlfs, s3fs) within flask-socketio using gevent.
I'm posting here in the hope that someone has managed to get this working and can offer some guidance.

Whenever I try and call an fspec filesystem method (e.g. ls), within a flask/flask-socketio view (sync) it raises the following error:

NotImplementedError: Calling sync() from within a running loop
This is to with the way that fsspec wraps its async methods in order that they can be used in sync methods (my view in this case)

minimal example:

server.py

from gevent import monkey

monkey.patch_all()

import os

from adlfs import AzureBlobFileSystem
from azure.identity.aio import DefaultAzureCredential
from flask import Flask
from flask_socketio import SocketIO


def create_app():
    app = Flask(__name__)

    @app.route("/")
    def hello_world():
        return "<p>Hello, World!</p>"

    @app.route("/ls")
    def listing():
        credentials = DefaultAzureCredential()
        account_name = os.environ.get("AZURE_ACCOUNT_NAME")
        azure_path = os.environ.get("AZURE_PATH")
        new_fs = AzureBlobFileSystem(account_name=account_name, credential=credentials)
        res = new_fs.ls(azure_path)
        list_str = [f"<p>{l}</p>" for l in res]
        return f"<p>Hello, Listing!</p>{''.join(list_str)}"

    return app

python server.py

curl 127.0.0.1:5000/ls

results in

[2024-10-02 16:57:35,406] ERROR in app: Exception on /ls [GET]
Traceback (most recent call last):
  File "<path>/site-packages/flask/app.py", line 2073, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<path>/site-packages/flask/app.py", line 1519, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<path>/site-packages/flask/app.py", line 1517, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "<path>/site-packages/flask/app.py", line 1503, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "server.py", line 26, in listing
    res = new_fs.ls(azure_path)
          ^^^^^^^^^^^^^^^^^^^^^
  File "<path>/site-packages/fsspec/asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<patch>/site-packages/fsspec/asyn.py", line 80, in sync
    raise NotImplementedError("Calling sync() from within a running loop")
NotImplementedError: Calling sync() from within a running loop

requirements.txt

adlfs==2024.7.0
Flask==2.1.3
Flask-SocketIO==5.3.6
fsspec==2024.6.1
gevent==24.2.1
azure-identity==1.17.1
werkzeug=2.2.2

Info

I have tried various alternatives, to no avail.
e.g.

• passing a loop to the filesystem init, then calling _ls (the underlying fsspec async method)
• calling asyncio.run(fs._ls)

The issue goes away if I monkeypatch everything but threads, but that seems unwise.
It is possible to pass a loop to AzureBlobFileSystem() but I can't currently get that working either.

I can bypass the issue by telling flask-socketio to use threading and not gevent (and remove the patching), however this has limitations (i.e. it uses werkzeug server, which is dev only) which means it can't be used in production. I am working on using gunicorn with simple-web-socket in the meantime, but would prefer not to go that rout, if I can in fact get the original usage working.

[miguelgrinberg]

This is likely an incompatiblity between the fsspec package and gevent. But I hope you understand that using gevent with Flask-SocketIO is entirely optional. If this combination gives you issues, then I'd suggest you drop gevent and use the threading async mode, which should be fully compatible with the Python ecosystem, unlike gevent which is sometimes tricky.

[rob-ashdown-monolith]

thanks Miguel - I had a suspicion that might be the case. I've raised an issue with fsspec in case there is a workaround they know about.

Regarding using threading - that is what I am working on now.
I'm serving via gunicorn (as opposed to SocketIO.run, since we can't use werkzeug in prod). For flask-socketio to work with gunicorn, the docs mention using simple-websocket.
Question:
Our views are currently all wired up via socketio.on, emit etc. Will those need to be refactored if simple-websocket is there, or does flask-socketio just adapt to its new environment automatically (and use that under-the-hood)?

[miguelgrinberg]

Switching async modes does not require any code changes.