how to use cpu_bound, io_bound?

[WolfgangFahl]

Question

To implement the nicegui widgets issue refactor to use run module for background tasks #33 i need to understand how to use cpu_bound and io_bound.

Currently there is a background module https://github.com/WolfgangFahl/nicegui_widgets/blob/main/ngwidgets/background.py

with function execute_in_background:

def execute_in_background(self, 
                              blocking_function: Callable[..., Any], 
                              *args: Any, 
                              use_process_pool: bool = False,
                              **kwargs: Any) -> (concurrent.futures.Future, Callable[[], Any]):
        """
        Executes a function in the background.
    
        Args:
            blocking_function (Callable[..., Any]): The function to execute.
            *args (Any): Positional arguments to pass to the function.
            use_process_pool (bool, optional): Whether to use a process pool executor. 
                Defaults to False, which means a thread executor will be used.
            **kwargs (Any): Keyword arguments to pass to the function.
    
        Returns:
            concurrent.futures.Future: The future representing the background task.
            Callable[[], Any]: A coroutine to get the result.
        """
        loop = asyncio.get_running_loop()
    
        # The lambda here ensures that both args and kwargs are passed to blocking_function
        func_with_args = lambda: blocking_function(*args, **kwargs)
    
        if use_process_pool:
            future = loop.run_in_executor(self.process_pool_executor, func_with_args)
        else:
            future = loop.run_in_executor(None, func_with_args)
    
        async def get_result():
            return await future
    
        return future, get_result

the idea here is that the call immediately returns and with get_result the actual result can be retrieved if need be.
I'd like to add a progress bar functionality here so that generally such tasks are callable with an interface to a progress bar (which might be nicegui ui element or a tqdm progress bar) - this is what https://github.com/WolfgangFahl/nicegui_widgets/blob/main/ngwidgets/progress.py is for.

Since cpu_bound and io_bound don't return a future i don't understand the intended usage of these function or how they'd be integrated in the idea of simply calling something like:

background_execute(function,progress,bound_style)

with function being a callable, progress being a progress bar interface, and bound_style specifying io or cpu_bound.

How would this idea be implementable?

[natankeddem]

I am interested in where this discussion goes. I find executing arbitrary async functions/methods one of the more challenging aspects of working with NiceGUI currently, doubly so if you are doing any ui.notify calls.

[WolfgangFahl]

@natankeddem - the scan2wiki upload form currently has two long running actions:

• OCR
• uploading
• Both are currently handled with the Backgroundtaskhandler and the progress bar handling is done with a workaround by estimating the progress based on the number of lines being logged by the subtasks. I'ts not perfect but shows where this is supposed to be going to. I assume there will be quite a few tasks where "statistical progress bars" will be necessary - that is measuring the task performance based on some inputs (e.g. PDF file size or pages in this case) and then estimating how long it will take. I personally think the user experience is better if there is a proper progress bar that is somewhat close to the real timing. Also the progess steps might be adapted "on the fly" in some cases if the progress formular is known and performance can be measured with the first few steps. Wrapping such functionality to nice components with a simple API is what i intended with the discussion NiceGUI Component Bazaar #1618

[natankeddem]

I'll play around with that function. Currently my goto for running things in the background is ui.timer which is ofcourse not optimal.

[falkoschindler]

run.io_bound and run.cpu_bound let you run a synchronous function in a separate thread or process and make the call awaitable.

• If the function basically waits for IO, e.g. a server request or a database query, it would be bad to block the main UI thread while waiting. Therefore run.io_bound moves the call to another thread and allows processing other tasks from the event loop while waiting for the function to complete.
• If the function needs significant CPU time, it would be bad to block the main UI thread while the computation is running. But running on a separate thread doesn't help due to Python's global interpreter lock. Therefore run.cpu_bound moves the execution to another process. It can also be awaited so that other tasks in the event loop can be processed in the meantime.
• Because @natankeddem mentioned ui.timer: This is basically an async task running on the event loop. You can move any coroutine to the background using background_tasks.create. This way you don't have to await a coroutine.

I hope this explains the intended use of these functions. In our FAQs we also refer to several examples.

Since cpu_bound and io_bound don't return a future

I don't quite understand. Both functions return an Awaitable. Why do you need a Future, @WolfgangFahl? Python's documentation says: "The rule of thumb is to never expose Future objects in user-facing APIs".

How to implement a progress bar for a long-running background task? I don't know. We usually try to break up the task into small chunks which can be awaited in a loop, while the progress is updated. If you put a long computation in another process (via run.cpu_bound), you would probably need some kind of inter-process queue since they don't share memory.

[ed2050]

@moi90
Nice. Having worked with Managers and Queues, that code can sometimes fail. You should use mp.get_context () to create manager. Like this:

mp_context = multiprocessing.get_context('fork')  # or spawn or forkserver ...
manager = mp_context.Manager()

Among the many potential issues with not using context, from pydocs:

• instantiating Manager () without a context sets the default context. Default context can only be set once. If user code tries to set the default context later, an error will be raised.
• "A library which wants to use a particular start method should probably use get_context() to avoid interfering with the choice of the library user."
• fork is the default start method on POSIX except macOS. At least until 3.14 when it changes. Current defaults: windows and mac use spawn, others use fork.
• "Note that objects related to one context may not be compatible with processes for a different context. In particular, locks created using the fork context cannot be passed to processes started using the spawn or forkserver start methods."
• Warning The 'spawn' and 'forkserver' start methods generally cannot be used with “frozen” executables (i.e., binaries produced by packages like PyInstaller and cx_Freeze) on POSIX systems. The 'fork' start method may work if code does not use threads.

If this seems like a mess - it is. Python multiprocessing is full of strange incompatibilities and weird errors (for instance, mac has a long-standing bug with fork). It's hard enough to use on a single platform. Getting it to work smoothly across many platforms without triggering obscure bugs is nigh impossible.

[ed2050]

Also note that fork and threads are incompatible. Likely to cause intermittent, non-reproducible bugs. It's better to get a context manager and start the queue Manager before nicegui starts running it's event loop. Which means moving your Manager() instantiation outside @ui.page def main ().

[Alyxion]

If the function needs significant CPU time, it would be bad to block the main UI thread while the computation is running. But running on a separate thread doesn't help due to Python's global interpreter lock.

Such statements regarding the GIL are in my opinion potentially quite irritating or misleading for Python newbies though. From my experience multi-threading also for CPU heavy tasks works like a charm in Python in 2024, given one knows how to lock data, avoid race conditions, dead locks etc. If you process. e.g. images or dataframes on a 16 core system with 16 threads in a single process in Python it will nearly be 16x as fast.

Nowdays nearly all Python base libraries and most importantly all really cpu heavy libraries such as Pandas, NumPy, OpenCV, Scikit, Numba etc., so everything that in daily practice really needs significant CPU time do not care about the global interpreted lock anymore at all. Once you call one of theirs functions the GIL is released and available for the other threads until its finished.

Very app and use case specific out of quesiton but I would claim if someone uses Python "right" in terms of vectorization, using the right libraries if they provide a function for a task rather than editing every pixel of an image in a nested for-loop by hand, then the GIL should be nearly neglegible nowdays, may be low one digit percent.

A simple example allowing the user to access processing state "live" in a single process.

from typing import Self
import time
from threading import RLock, Event
from nicegui import ui, run, background_tasks

class MtBackgroundTask:
    # Important: Only use static data as input in the task, ensure its deep copied rather than referenced or
    # remove other references to the data to avoid side effects

    def __init__(self):
        self._data_lock = RLock()
        self._progress = 0
        self._data = []
        self._done = Event()
        self._cancelled = Event()

    def _execute(self):
        self.run()
        self._done.set()

    def run(self):
        """The actual task to be executed, override this method"""
        raise NotImplementedError

    def cancel(self):
        self._cancelled.set()

    @property
    def progress(self):
        with self._data_lock:
            return self._progress

    @progress.setter
    def progress(self, value):
        with self._data_lock:
            self._progress = value
    
    @property
    def done(self):
        return self._done.is_set()

    @property
    def data(self):
        if not self._done.is_set():
            raise ValueError('Task is not done')
        with self._data_lock:
            return self._data
    
    @data.setter
    def data(self, value):
        with self._data_lock:
            self._data = value
    
    def start(self) -> Self:
        background_tasks.create(run.io_bound(self._execute))
        return self

class SquareTask(MtBackgroundTask):
    def __init__(self, numbers:list[int]):
        super().__init__()
        self._numbers = numbers
        
    def run(self):
        data = []
        for i in self._numbers:
            if self._cancelled.is_set():
                break
            time.sleep(0.04)     # Simulate work, actual computation goes here
            data.append(i * i)  # Collect resuls
            self.progress = i + 1
        self.data = data  # store results

@ui.page('/')
async def index():
    task: MtBackgroundTask | None = None
    timer: ui.timer | None = None
    def start_task():    
        results.clear()
        nonlocal task, timer        
        if task:
            ui.notify('Task is already running')
            return
        sample_numbers = [i for i in range(1, 100)]
        progress._props['max'] = len(sample_numbers)
        task = SquareTask(numbers=sample_numbers).start()        
        timer = ui.timer(interval=0.05, callback=timer_handler, active=True)

    def cancel_task():
        nonlocal task, timer
        progress.value = 0
        if task:
            task.cancel()
            task = None
            if timer:
                timer.cancel()
                timer = None

    def timer_handler():
        nonlocal task
        if task is not None:
            # update progress bar
            progress.value = task.progress
            if task.done:
                ui.notify('Task is done')
                results.clear()
                with results:
                    for element in task.data:
                        ui.label(str(element))
                task = None

    ui.button('Start', on_click=start_task)
    ui.button('Cancel', on_click=cancel_task)
    progress = ui.circular_progress(0, min=0, max=10)
    results = ui.grid(columns=10)

ui.run()

[rodja]

From my experience multi-threading also for CPU heavy tasks works like a charm in Python in 2024

Are you sure this statement is right? To my knowledge Python multi-threading is always using the GIL.

all really cpu heavy libraries such as Pandas, NumPy, OpenCV, Scikit, Numba etc.,

These libraries are doing the cpu intensive tasks in C++, and of course the GIL does not apply.

so everything that in daily practice really needs significant CPU time do not care about the global interpreted lock anymore

I disagree. What about networx and shapely for example?

[Alyxion]

From my experience multi-threading also for CPU heavy tasks works like a charm in Python in 2024

Are you sure this statement is right? To my knowledge Python multi-threading is always using the GIL.

Thats the point of how one defines "Python multithreading".

By my definition it is if: I have a task currently single thread in Python, split this task up into sub tasks and parallize them using a Python thread.
And then doing the overall execution time is usually divided more or less linearly by the number of threads, given they don't exceed the CPU core count.

We do large scale image and database crunching on them in the cloud, not rarely with 128 threads running on a 128 core machine in a single process and easily bring them close to 100% CPU usage, see simple example here: https://github.com/Alyxion/nicegui-samples/tree/main/nicegui_samples/multithreading Not sure if absolutely every function in those libraries profit, likely not, but at least the time intensive ones seem to do.

all really cpu heavy libraries such as Pandas, NumPy, OpenCV, Scikit, Numba etc.,

These libraries are doing the cpu intensive tasks in C++, and of course the GIL does not apply.

That's the point.

so everything that in daily practice really needs significant CPU time do not care about the global interpreted lock anymore

I disagree. What about networx and shapely for example?

Well obviously one can always find exceptions and say "Hah, didnt I tell you?" :-). My point is how the average person will inteprete the statement and could believe multithreading in Python actually does not help in CPU bound scenarios.

Take 100 random Python applications here on GitHub which have general potential for doing some stuff in parallel.

And then enhance them with "multithreading in Python".

The average user does not know or need to know wether the function he/she invokes does C++ or Python under the hood, why should he?

Then I'm pretty sure the statement If the function needs significant CPU time ... But running on a separate thread doesn't help due to Python's global interpreter lock. will fail for the majority of these 100 random samples because it will very likely help most of them.

People do use pandas, they do use numpy, they use PIL, pytorch, tensorflow, Numba, they do use OpenCV etc. and so YES, it does help them running on a separate thread ;-). And next to all of those it obviously also makes everything "faster" when waiting for a database, files etc though of course just because of parallel waiting for i/o delays.

My 2 cents, have a happy new year 🚀

[WolfgangFahl]

see #4053 and #2729