Skip to content

Commit

Permalink
add with_async_env (#268)
Browse files Browse the repository at this point in the history
This PR helps the users stuck in a sync context. They had problems before my changes, had real trouble through my changes and now have an easy and performant logic through `with_async_env` so are first-class citizen again.

Changes:

- simplify the async env creation to the `with_async_env` method
- document the new `with_async_env` method
  • Loading branch information
devkral authored Jan 23, 2025
1 parent 200791a commit 50f4fdb
Show file tree
Hide file tree
Showing 10 changed files with 223 additions and 15 deletions.
34 changes: 34 additions & 0 deletions docs/connection.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,12 @@ Or doing it manually (that applies to every framework):
{!> ../docs_src/connections/simple.py !}
```

Or just as an async contexmanager

```python
{!> ../docs_src/connections/asynccontextmanager.py !}
```

And that is pretty much this. Once the connection is hooked into your application lifecycle.
Otherwise you will get warnings about decreased performance because the databasez backend is not connected and will be
reininitialized for each operation.
Expand Down Expand Up @@ -80,11 +86,39 @@ This warning appears, when an unconnected Database object is used for an operati

Despite bailing out the warning `DatabaseNotConnectedWarning` is raised.
You should connect correctly like shown above.
In sync environments it is a bit trickier.

!!! Note
When passing Database objects via using, make sure they are connected. They are not necessarily connected
when not in extra.

## Integration in sync environments

When the framework is sync by default and no async loop is active we can fallback to `run_sync`.
It is required to build an async evnironment via the `with_async_env` method of registry. Otherwise
we run in bad performance problems and have `DatabaseNotConnectedWarning` warnings.
`run_sync` calls **must** happen within the scope of `with_async_env`. `with_async_env` is reentrant and has an optional loop parameter.

```python
{!> ../docs_src/connections/contextmanager.py !}
```
To keep the loop alive for performance reasons we can either wrap the server worker loop or in case of
a single-threaded server the server loop which runs the application. As an alternative you can also keep the asyncio eventloop alive.
This is easier for sync first frameworks like flask.
Here an example which is even multithreading save.

```python
{!> ../docs_src/connections/contextmanager_with_loop.py !}
```

That was complicated, huh? Let's unroll it in a simpler example with explicit loop cleanup.


```python
{!> ../docs_src/connections/contextmanager_with_loop_and_cleanup.py !}
```


## Querying other schemas

Edgy supports that as well. Have a look at the [tenancy](./tenancy/edgy.md) section for more details.
Expand Down
5 changes: 2 additions & 3 deletions docs/queries/queries.md
Original file line number Diff line number Diff line change
Expand Up @@ -1053,14 +1053,13 @@ await User.query.create(name="Edgy")
**With run_sync**

```python
from edgy import run_sync

run_sync(User.query.all())
run_sync(User.query.filter(name__icontains="example"))
run_sync(User.query.create(name="Edgy"))
```

And that is it! You can now run all queries synchronously within any framework, literally.
And that is it! You can now run all queries synchronously within any framework. You still have to connect
the Registry with its dbs first via `async with registry: ...` or in sync-frameworks with `with registry.with_async_env(): ...`.

## Cross database queries

Expand Down
4 changes: 4 additions & 0 deletions docs/release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ hide:

- Add `exclude_autoincrement` parameter/class attribute to ModelFactory.
- Add `build_values` method to ModelFactory. It can be used to extract the values without a model.
- Make Registry initialization compatible with sync contexts via `with_async_env(loop=None)` method.
- `run_sync` has now an optional loop parameter.

### Changed

Expand All @@ -24,6 +26,8 @@ hide:
- `to_list_factory_field` honors the min and max parameter specified by parameters.
It defaults however to the provided min and max parameters.
- RefForeignKey has now an extra subclass of BaseField. This way the exclusion of works reliable.
- `run_sync` reuses idling loops.
- `run_sync` uses the loop set by the Registry contextmanager `with_async_env`.

### Fixed

Expand Down
12 changes: 12 additions & 0 deletions docs_src/connections/asynccontextmanager.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
from edgy import Registry, Instance, monkay

models = Registry(database="sqlite:///db.sqlite", echo=True)


async def main():
# check if settings are loaded
monkay.evaluate_settings_once(ignore_import_errors=False)
# monkey-patch so you can use edgy shell
monkay.set_instance(Instance(registry=registry))
async with models:
...
14 changes: 14 additions & 0 deletions docs_src/connections/contextmanager.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
from edgy import Registry, Instance, monkay

models = Registry(database="sqlite:///db.sqlite", echo=True)


# check if settings are loaded
monkay.evaluate_settings_once(ignore_import_errors=False)
# monkey-patch app so you can use edgy shell
monkay.set_instance(Instance(registry=registry))


def main():
with models.with_async_env():
edgy.run_sync(User.query.all())
31 changes: 31 additions & 0 deletions docs_src/connections/contextmanager_with_loop.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
import asyncio
from contextvars import ContextVar
from edgy import Registry, Instance, monkay, run_sync

models = Registry(database="sqlite:///db.sqlite", echo=True)


# multithreading safe
event_loop = ContextVar("event_loop", default=None)


def handle_request():
loop = event_loop.get()
if loop is None:
# eventloops die by default with the thread
loop = asyncio.new_event_loop()
event_loop.set(loop)
with models.with_loop(loop):
edgy.run_sync(User.query.all())


def get_application():
app = ...
# check if settings are loaded
monkay.evaluate_settings_once(ignore_import_errors=False)
# monkey-patch app so you can use edgy shell
monkay.set_instance(Instance(registry=registry, app=app))
return app


app = get_application()
21 changes: 21 additions & 0 deletions docs_src/connections/contextmanager_with_loop_and_cleanup.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
import asyncio
from edgy import Registry, Instance, monkay, run_sync

models = Registry(database="sqlite:///db.sqlite", echo=True)

# check if settings are loaded
monkay.evaluate_settings_once(ignore_import_errors=False)
# monkey-patch app so you can use edgy shell
monkay.set_instance(Instance(registry=registry))

loop = asyncio.new_event_loop()
with models.with_loop(loop):
edgy.run_sync(User.query.all())

# uses the same loop
with models.with_loop(loop):
edgy.run_sync(User.query.all())


loop.run_until_complete(loop.shutdown_asyncgens())
loop.close()
36 changes: 31 additions & 5 deletions edgy/core/connection/registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import re
import warnings
from collections import defaultdict
from collections.abc import Container, Sequence
from collections.abc import Container, Generator, Sequence
from copy import copy as shallow_copy
from functools import cached_property, partial
from types import TracebackType
Expand All @@ -18,7 +18,7 @@
from edgy.conf import evaluate_settings_once_ready
from edgy.core.connection.database import Database, DatabaseURL
from edgy.core.connection.schemas import Schema
from edgy.core.utils.sync import run_sync
from edgy.core.utils.sync import current_eventloop, run_sync
from edgy.types import Undefined

from .asgi import ASGIApp, ASGIHelper
Expand Down Expand Up @@ -249,9 +249,9 @@ def callback(model_class: type["BaseModelType"]) -> None:
if "content_type" in model_class.meta.fields:
return
related_name = f"reverse_{model_class.__name__.lower()}"
assert (
related_name not in real_content_type.meta.fields
), f"duplicate model name: {model_class.__name__}"
assert related_name not in real_content_type.meta.fields, (
f"duplicate model name: {model_class.__name__}"
)

field_args: dict[str, Any] = {
"name": "content_type",
Expand Down Expand Up @@ -578,6 +578,32 @@ async def __aexit__(
ops.append(value.disconnect())
await asyncio.gather(*ops)

@contextlib.contextmanager
def with_async_env(
self, loop: Optional[asyncio.AbstractEventLoop] = None
) -> Generator["Registry", None, None]:
close: bool = False
if loop is None:
try:
loop = asyncio.get_running_loop()
# when in async context we don't create a loop
except RuntimeError:
# also when called recursively and current_eventloop is available
loop = current_eventloop.get()
if loop is None:
loop = asyncio.new_event_loop()
close = True

token = current_eventloop.set(loop)
try:
yield run_sync(self.__aenter__(), loop=loop)
finally:
run_sync(self.__aexit__(), loop=loop)
current_eventloop.reset(token)
if close:
loop.run_until_complete(loop.shutdown_asyncgens())
loop.close()

@overload
def asgi(
self,
Expand Down
28 changes: 21 additions & 7 deletions edgy/core/utils/sync.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,15 @@
import asyncio
import weakref
from collections.abc import Awaitable
from contextvars import copy_context
from contextvars import ContextVar, copy_context
from threading import Event, Thread
from typing import Any, Optional

# for registry with
current_eventloop: ContextVar[Optional[asyncio.AbstractEventLoop]] = ContextVar(
"current_eventloop", default=None
)


async def _coro_helper(awaitable: Awaitable, timeout: Optional[float]) -> Any:
if timeout is not None and timeout > 0:
Expand Down Expand Up @@ -54,18 +59,27 @@ def get_subloop(loop: asyncio.AbstractEventLoop) -> asyncio.AbstractEventLoop:
return sub_loop


def run_sync(awaitable: Awaitable, timeout: Optional[float] = None) -> Any:
def run_sync(
awaitable: Awaitable,
timeout: Optional[float] = None,
*,
loop: Optional[asyncio.AbstractEventLoop] = None,
) -> Any:
"""
Runs the queries in sync mode
"""
loop = None
try:
loop = asyncio.get_running_loop()
except RuntimeError:
loop = None
if loop is None:
try:
loop = asyncio.get_running_loop()
except RuntimeError:
# in sync contexts there is no asyncio.get_running_loop()
loop = current_eventloop.get()

if loop is None:
return asyncio.run(_coro_helper(awaitable, timeout))
elif not loop.is_closed() and not loop.is_running():
# re-use an idling loop
return loop.run_until_complete(_coro_helper(awaitable, timeout))
else:
ctx = copy_context()
# the context of the coro seems to be switched correctly
Expand Down
53 changes: 53 additions & 0 deletions tests/registry/test_registry_run_sync.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
import asyncio

import pytest

import edgy
from edgy.testing.client import DatabaseTestClient
from tests.settings import DATABASE_URL

database = DatabaseTestClient(
DATABASE_URL,
full_isolation=False,
use_existing=False,
drop_database=True,
force_rollback=False,
)
models = edgy.Registry(database=database)


class User(edgy.StrictModel):
name: str = edgy.fields.CharField(max_length=100)

class Meta:
registry = models


def test_run_sync_lifecyle():
with models.with_async_env():
edgy.run_sync(models.create_all())
user = edgy.run_sync(User(name="edgy").save())
assert user
assert edgy.run_sync(User.query.get()) == user


def test_run_sync_lifecyle_sub():
with models.with_async_env(), models.with_async_env():
edgy.run_sync(models.create_all())
user = edgy.run_sync(User(name="edgy").save())
assert user
assert edgy.run_sync(User.query.get()) == user


def test_run_sync_lifecyle_with_idle_loop():
with pytest.raises(RuntimeError):
asyncio.get_running_loop()
loop = asyncio.new_event_loop()
with models.with_async_env(loop=loop):
edgy.run_sync(models.create_all())
user = edgy.run_sync(User(name="edgy").save())
assert user
assert edgy.run_sync(User.query.get()) == user
loop.close()
with pytest.raises(RuntimeError):
asyncio.get_running_loop()

0 comments on commit 50f4fdb

Please sign in to comment.