Lower-level TaskGroup API

[davidbrochart]

Say I have an async context manager MyResource that launches async tasks internally on enter, and cleans them up on exit. If I want to use AnyIO for managing these tasks, this wouldn't work:

import asyncio
from anyio import create_task_group


async def my_task():
    await asyncio.Future()


class MyResource:
    async def __aenter__(self):
        async with create_task_group() as self.task_group:
            self.task_group.start_soon(my_task)

    async def __aexit__(self, exc_type, exc_val, exc_tb):
        self.task_group.cancel_scope.cancel()


async def main():
    async with MyResource():
        pass

asyncio.run(main())

MyResource.__aenter__ never completes because it's waiting for my task to be done, but I want to run it in the background. I could pass a task group to MyResource and start the task on it, but I want this resource to be used in a general asyncio environment, so I can't assume people use AnyIO outside MyResource.
A way to achieve this would be to create an asyncio-specific TaskGroup in MyResource, and manually call __aenter__ and __aexit__ on it:

import asyncio
from anyio import create_task_group
from anyio._backends._asyncio import TaskGroup


async def my_task():
    await asyncio.Future()


class MyResource:
    async def __aenter__(self):
        self.task_group = TaskGroup()
        await self.task_group.__aenter__()
        self.task_group.start_soon(my_task)

    async def __aexit__(self, exc_type, exc_val, exc_tb):
        self.task_group.cancel_scope.cancel()
        await self.task_group.__aexit__(exc_type, exc_val, exc_tb)


async def main():
    async with MyResource():
        pass

asyncio.run(main())

But this is relying on a bunch of private methods.
Would it make sense to be able to use a TaskGroup without the context manager API? It's a common practice for libraries to offer both a context manager and a "raw object", the latter allowing for lower-level management, but this might go against the general philosophy of AnyIO.
Do you see a better way to solve my issue? Am I wrong in wanting such feature in the first place?

[Fuyukai]

The standard pattern for this is:

@contextlib.asynccontextmanager
async def open_my_object(...):
	async with anyio.create_task_group() as nursery:
		yield MyObject(nursery)

Would it make sense to be able to use a TaskGroup without the context manager API?

No, because then you could easily violate structured concurrency. The whole point of making it a context manager is to ban you from keeping tasks around after your object exits.

[agronholm]

But this is relying on a bunch of private methods.

What private methods? It's totally kosher to call __aenter__() and __aexit__() from other such methods. I do that all the time.

[smurfix]

It may be kosher in simple situations (this one qualifies), but unwrapping __aexit__ calls isn't exactly trivial. Just look at the implementation of [async]contextmanager.

Given that it's fairly easy to code yourself into corners (e.g. you extend your __aexit__ wrapper next week … and next month … and suddenly it might raise an exception before (i.e. instead of) closing the nursery … oops), I tend to agree with @Fuyukai and always use the asynccontextmanager+yield solution.

[agronholm]

It may be kosher in simple situations (this one qualifies), but unwrapping aexit calls isn't exactly trivial. Just look at the implementation of [async]contextmanager.

In nontrivial cases I just use an async exit stack:

async def __aenter__(self):
    async with AsyncExitStack() as exit_stack:
        self._nursery = await exit_stack.enter_async_context(open_nursery())
        self._exit_stack = exit_stack.pop_all()

async def __aexit__(self, exc_type, exc_value, exc_tb):
    return await self._exit_stack.__aexit__(exc_type, exc_value, exc_tb)

What's nice about this approach is that if bad things happen in __aenter__() already, it still closes all the resources that have already been put into the exit stack.

[njsmith]

Pretty much the only reasonable way to implement a custom __aenter__/__aexit__ while wrapping another context manager is to make your __aenter__/__aexit__ a trivial wrapper around something from contextlib -- either @asynccontextmanager or AsyncExitStack. In particular, __aexit__'s exception-handling semantics are gnarly, and TaskGroup really depends on them being implemented properly.

[davidbrochart]

Thanks for all the feedback, that's very helpful!