fix bugs and add tests for OperationsAPI/data, fix retry_index for cl…

…ient plugin handler, raise error in Client for no status match when fail_silently=False

docs/en/guide/schema-query.md

@@ -155,6 +155,11 @@ In addition to the default behavior based on the primary key, you can adjust the
 
 * `must_create`: If set to True, the method is forced to be processed as a creation method, although an error is thrown if the data contains a primary key that has already been created.
 * `must_update`: If set to True, the method is forced to be processed as an update method, and an error is thrown when the update cannot be completed, such as when the primary key is missing or does not exist.
+* `transaction`: you can set to True or a database connection name to enable the **transaction** for that database, default is False, set to True will enable transaction for the default database of the corresponding model
+* `using`: Set a database connection name to specify the database to save, by default will be the default database of the corresponding model (`default` in Django ORM)
+
+!!! tip
+	The name of a database connection is the dict key defined in the `DatabaseConnections` configuration, like `'default'`
 
 The following example shows `save` the use of the method in the interface by writing and creating the article API.
 
@@ -211,8 +216,8 @@ The following is an example of an API for creating users in bulk
 
 	class UserAPI(api.API):
 	    @api.post
-	    async def bulk(self, data: List[UserSchema] = request.Body):
-	        await UserSchema.abulk_save(data)
+	    async def bulk(self, data: List[UserSchema] = request.Body) -> List[UserSchema]:
+	         return await UserSchema.abulk_save(data)
 	```
 === "Sync API"
 	```python   hl_lines="10"
@@ -224,14 +229,20 @@ The following is an example of an API for creating users in bulk
 
 	class UserAPI(api.API):
 	    @api.post
-	    def bulk(self, data: List[UserSchema] = request.Body):
-	        UserSchema.bulk_save(data)
+	    def bulk(self, data: List[UserSchema] = request.Body) -> List[UserSchema]:
+	         return UserSchema.bulk_save(data)
 	```
 
 The method in the example uses `List[UserSchema]` to annotate the body of the request, indicating that it accepts a list of JSON data. The API will automatically parse and convert it into a list of `UserSchema` instances. You only need to call the `UserSchema.bulk_save` method to create or update the data in this list in batches.
 
-!!! tip
-	whether an object of the list be updated or created depends on the existence of the primary key field, you can still use `must_create` or `must_update` to restrain the behaviour
+`bulk_save` will returns the saved Schema instances list, set the primary key value for the newly created instance. In the above example, we directly returned the result of `bulk_save` as the API response data.
+
+Apart from the first parameter that takes a list to save, `bulk_save` provided other parameters to control its save behaviour:
+
+* `must_create`: If set to True, all the items of the list are forced to be created, although an error is thrown if the data contains a primary key that has already been created.
+* `must_update`: If set to True, all the items of the list are forced to be updated, and an error is thrown when the update cannot be completed, such as when the primary key is missing or does not exist.
+* `transaction`: you can set to True or a database connection name to enable the **transaction** for that database, default is False, set to True will enable transaction for the default database of the corresponding model
+* `using`: Set a database connection name to specify the database to save, by default will be the default database of the corresponding model (`default` in Django ORM)
 
 ### `commit` - Update the queryset
 

docs/zh/community/release.md

@@ -1,4 +1,16 @@
 # 版本发布记录
+## v2.7.2
+
+发布时间：2024/12/25
+
+### 优化项
+
+* `Client` 类对于响应码不匹配任何响应模板的响应，在 `fail_silently=False` 时会抛出错误
+
+### 问题修复
+
+* UtilMeta 平台的 Data 板块的数据查询问题修复
+
 
 ## v2.7.1
 
@@ -11,7 +23,7 @@
 ### 优化项
 
 * 对 `orm.Schema` 查询的无限循环嵌套问题解决方式进行优化，增加了 `Perference.orm_schema_query_max_depth` 设置限制 Schema 查询深度，默认为 100
-* 对 FastAPI / Starlette 应用优化服务端报错的处理逻辑，能够降异常调用栈记录到 Operations 系统的日志存储中并提供查看
+* 对 FastAPI / Starlette 应用优化服务端报错的处理逻辑，能够在日志中记录 FastAPI 接口抛出的异常调用栈信息
 
 ### 问题修复
 
@@ -31,7 +43,7 @@
 
 * 支持连接到 **代理节点** 进行内网服务集群的管理
 * 支持解析与同步 **Django Ninja** 的 OpenAPI 文档
-* 支持 `meta.ini` 中指定 `pidfile` 存储当前服务进程，同时支持 `restart` 重启服务命令和 `down` 停止服务命令 
+* 支持 `meta.ini` 中指定 `pidfile` 存储当前服务进程 PID，同时支持 `restart` 重启服务命令和 `down` 停止服务命令 
 
 ### 优化项
 

docs/zh/guide/ops.md

@@ -280,7 +280,9 @@ please visit [URL] to view and manage your APIs'
 UtilMeta 已经提供了一个开源的代理服务 [utilmeta-proxy](https://github.com/utilmeta/utilmeta-proxy)
 
 在 UtilMeta 平台中也有连接和管理内网集群的操作指引，点击【Add Cluster】即可进入添加集群功能
+
 <img src="https://utilmeta.com/assets/image/add-cluster-hint.png" href="https://ops.utilmeta.com" target="_blank" width="300"/>
+
 沿着其中的步骤指引进行操作将会自动搭建好一个集群的内网代理节点并连接到 UtilMeta 平台，之后集群中的服务只需要配置连接到代理节点即可，无需再手动连接到平台，代理节点会作为内网集群的服务注册中心向 UtilMeta 平台进行同步
 
 在 Operations 配置中，可以使用 `proxy` 参数配置代理服务节点的地址与设置，主要的参数有

docs/zh/guide/schema-query.md

@@ -150,10 +150,15 @@ class ArticleSchema(orm.Schema[Article]):
 
 由一个 `orm.Schema` **实例**调用，将其中的数据保存到模型对应的数据表中，如果 Schema 实例中包含着存在于数据表中的主键，那么会更新对应的表记录，否则会创建一个新的表记录
 
-除了根据主键判断的默认行为外，你还可以通过两个参数调节 `save` 方法的行为
+你可以通过 `save` 方法的参数调节它的行为
 
 * `must_create`：如果设为 True，则该方法会被强制处理为创建方法，当然如果数据中包含了已经被创建的主键，则会抛出错误
 * `must_update`：如果设为 True，则该方法会被强制处理为更新方法，当无法完成更新（如缺少主键或者主键不存在时）会抛出错误
+* `transaction`：设为 True 或者数据库连接的名称来开启对应数据库连接的 **事务**，默认不开启，设为 True 开启的是模型默认连接数据库的事务
+* `using`：可以传入一个数据库连接名称字符串来指定保存到的数据库，默认将沿用对应模型的数据库配置（在 Django ORM 中默认为 `default` 数据库）
+
+!!! tip
+	数据库连接的名字就是在 `DatabaseConnections` 中定义的数据库字典的键，如 `'default'`
 
 下面以编写创建文章 API 为例展示了 `save` 方法在接口中的使用
 
@@ -210,8 +215,8 @@ class ArticleSchema(orm.Schema[Article]):
 
 	class UserAPI(api.API):
 	    @api.post
-	    async def bulk(self, data: List[UserSchema] = request.Body):
-	        await UserSchema.abulk_save(data)
+	    async def bulk(self, data: List[UserSchema] = request.Body) -> List[UserSchema]:
+	         return await UserSchema.abulk_save(data)
 	```
 === "同步 API"
 	```python   hl_lines="10"
@@ -223,14 +228,20 @@ class ArticleSchema(orm.Schema[Article]):
 
 	class UserAPI(api.API):
 	    @api.post
-	    def bulk(self, data: List[UserSchema] = request.Body):
-	        UserSchema.bulk_save(data)
+	    def bulk(self, data: List[UserSchema] = request.Body) -> List[UserSchema]:
+	         return UserSchema.bulk_save(data)
 	```
 
 例子中的方法使用 `List[UserSchema]` 作为请求体的类型声明，表示接受一个列表 JSON 数据，接口将自动解析转化为 UserSchema 实例的列表，你只需要调用 `UserSchema.bulk_save` 方法即可将这个列表中的数据批量创建或更新
 
-!!! tip
-	决定列表中的元素是创建还是更新依然是根据其中是否包含着数据表中存在的主键，你依然可以使用 `must_create` 和 `must_update` 来调节其中的行为
+`bulk_save` 方法会返回保存好的 Schema 数据实例列表，为新创建的数据实例设置主键值，在上面的例子中把这个结果直接作为 API 函数的响应数据返回
+
+除了第一个接受列表数据的参数外，`bulk_save` 方法还提供一些参数来调控其中的行为
+
+* `must_create`：如果设为 True，则列表中的元素会被强制创建，当然如果数据中包含了已经被创建的主键，则会抛出错误
+* `must_update`：如果设为 True，则列表中的元素会被强制更新，当无法完成更新（如缺少主键或者主键不存在时）会抛出错误
+* `transaction`：设为 True 或者数据库连接的名称来开启对应数据库连接的 **事务**，默认不开启，设为 True 开启的是模型默认连接数据库的事务
+* `using`：可以传入一个数据库连接名称字符串来指定保存到的数据库，默认将沿用对应模型的数据库配置（在 Django ORM 中默认为 `default` 数据库）
 
 ### `commit` - 更新查询集
 

tests/server/app/api.py

@@ -88,15 +88,16 @@ def patch(self, data: UserBase[orm.WP], user: User = session_user) -> UserBase:
         data.save()
         return UserBase.init(data.id)
 
-    from utilmeta.utils import print_time
-
-    @api.get
-    @print_time
-    async def test(self, using='postgresql'):
-        from app.schema import UserSchema
-        from app.models import User
-        return await UserSchema.ainit(
-            User.objects.filter(
-                username='alice',
-            ).using(using),
-        )
+    # @api.get
+    # async def test(self, using='postgresql'):
+    #     from app.schema import ArticleSchema
+    #     article = ArticleSchema[orm.A](
+    #         title='My new async article 2',
+    #         content='my async content',
+    #         creatable_field='a',
+    #         # test ignore on mode 'a'
+    #         author_id=1,
+    #         views=10
+    #     )
+    #     await article.asave(using=using)
+    #     return await article.aget_instance(fresh=True, using=using)

tests/server/app/migrations/0002_article_tags.py

@@ -0,0 +1,18 @@
+# Generated by Django 5.1.4 on 2024-12-26 05:23
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('app', '0001_initial'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='article',
+            name='tags',
+            field=models.JSONField(default=list),
+        ),
+    ]

tests/server/app/models.py

@@ -1,4 +1,3 @@
-from django.db.models import Manager
 from django.db import models
 from utilmeta.core.orm.backends.django.models import PasswordField, AwaitableModel, AbstractSession
 
@@ -62,7 +61,7 @@ class Article(BaseContent):
     description = models.TextField(default="")
     slug = models.SlugField(db_index=True, unique=True)
     views = models.PositiveIntegerField(default=0)
-
+    tags = models.JSONField(default=list)
     # in Windows under 3.9, JSON field cannot be processed by SQLite
     # if sys.version_info >= (3, 9):
     #     tags = models.JSONField(default=list)

tests/server/app/schema.py

@@ -59,7 +59,7 @@ class ArticleSchema(ContentSchema[Article]):
         # 'w': orm.Relate('author'),
         'r': auth.Require()
     })
-
+    tags: List[str] = orm.Field(default_factory=list, defer_default=True)
     author: UserBase = Field(description='author field')
     author_name: str = orm.Field("author.username", alias="author.name")
     author__signup_time: datetime = orm.Field(no_output=True)  # test __ in field name

tests/server/mock.py

@@ -0,0 +1,89 @@
+from utilmeta.core import api, request
+from utype.types import *
+from utilmeta.ops.schema import ResourcesSchema, SupervisorData
+from utilmeta.ops.client import OperationsClient
+from utilmeta.ops.config import Operations
+from utilmeta.ops import __spec_version__
+from utilmeta.utils import exceptions
+import time
+import utype
+
+
+class SupervisorMockAPI(api.API):
+    node_id: Optional[str] = request.HeaderParam('X-Node-ID', default=None)
+    service_id: Optional[str] = request.HeaderParam('X-Service-ID', default=None)
+    node_key: Optional[str] = request.HeaderParam('X-Node-Key', default=None)
+    access_key: Optional[str] = request.HeaderParam('X-Access-Key', default=None)
+
+    cluster_id: Optional[str] = request.HeaderParam('X-Cluster-ID', default=None)
+    cluster_key: Optional[str] = request.HeaderParam('X-Cluster-Key', default=None)
+
+    @api.get
+    def get(self):
+        """
+        get supervisor status
+        """
+        return dict(
+            utilmeta=__spec_version__,
+            supervisor='test',
+            timestamp=int(time.time() * 1000),
+        )
+
+    @api.get
+    def list(self):
+        return [dict(ident='test', base_url='http://127.0.0.1:8000/api/spv', connected=True)]
+
+    @api.post
+    def resources(self, data: ResourcesSchema = request.Body):
+        pass
+
+    class NodeData(utype.Schema):
+        ops_api: str
+        name: str
+        base_url: str
+        title: Optional[str] = None
+        description: str = ''
+
+        version: Optional[str] = None
+        spec_version: str = None
+        production: bool = False
+
+        language: Optional[str] = utype.Field(default=None)
+        language_version: Optional[str] = utype.Field(default=None)
+        utilmeta_version: Optional[str] = utype.Field(default=None)
+
+    def post(self, data: NodeData = request.Body) -> SupervisorData:
+        if not self.access_key and not self.cluster_key:
+            raise exceptions.BadRequest('key required')
+        from utilmeta import service
+        config = service.get_config(Operations)
+        node_id = 'TEST_NODE_ID'
+        supervisor_data = SupervisorData(
+            node_id=node_id,
+            url=config.connect_url,
+            public_key='TEST_PUBLIC_KEY',
+            ops_api=config.ops_api,
+            ident='test',
+            base_url=data.base_url,
+            backup_urls=[],
+            init_key=self.cluster_key or self.access_key or 'TESt_INIT_KEY'
+        )
+        with OperationsClient(
+            base_url=config.ops_api,
+            node_id=node_id,
+        ) as client:
+            client.add_supervisor(supervisor_data)
+        return supervisor_data
+
+    def delete(self):
+        if not self.access_key:
+            raise exceptions.BadRequest('key required')
+        from utilmeta import service
+        config = service.get_config(Operations)
+        node_id = 'TEST_NODE_ID'
+        with OperationsClient(
+            base_url=config.ops_api,
+            node_id=node_id,
+        ) as client:
+            client.delete_supervisor()
+        return node_id

tests/test_1_orm/test_prepare_data.py

@@ -5,7 +5,7 @@
 
 def test_prepare_data(service, db_using):
     # from utilmeta.utils import exceptions as exc
-    from django.db.utils import OperationalError, ProgrammingError
+    # from django.db.utils import OperationalError, ProgrammingError
     # from server import service
     # setup service
     # service.setup()
@@ -15,12 +15,12 @@ def test_prepare_data(service, db_using):
     # from domain.blog.module import UserMain, ArticleMain, CommentMain, ArticleAdmin
     from utilmeta.core import orm
 
-    try:
-        User.objects.using(db_using).exists()
-    except (OperationalError, ProgrammingError):
-        from django.core.management import execute_from_command_line
-        execute_from_command_line([__name__, 'migrate', f'--database={db_using}'])
-        # os.system("python -m utilmeta migrate")
+    # try:
+    #     User.objects.using(db_using).exists()
+    # except (OperationalError, ProgrammingError):
+    from django.core.management import execute_from_command_line
+    execute_from_command_line([__name__, 'migrate', f'--database={db_using}'])
+    # os.system("python -m utilmeta migrate")
 
     # delete all data
     User.objects.all().using(db_using).delete()
@@ -75,6 +75,9 @@ def test_prepare_data(service, db_using):
 
     assert Follow.objects.using(db_using).count() == 5
 
+    from django.db.models import JSONField
+    print('JSON FIELD:', JSONField)
+
     ArticleSchema[orm.A].bulk_save(
         [
             dict(
@@ -84,7 +87,7 @@ def test_prepare_data(service, db_using):
                 # slug='big-shot',
                 content="big shot content",
                 views=10,
-                tags=["shock", "head"],
+                tags=["shock", "head", db_using],
             ),
             dict(
                 id=2,
@@ -93,6 +96,7 @@ def test_prepare_data(service, db_using):
                 content="news content",
                 # slug='some-news',
                 views=3,
+                tags=[db_using, "news"],
             ),
             dict(
                 id=3,
@@ -101,6 +105,7 @@ def test_prepare_data(service, db_using):
                 # slug='huge-one',
                 content="huge one",
                 views=0,
+                tags=[db_using],
             ),
             dict(
                 id=4,
@@ -138,8 +143,9 @@ def test_prepare_data(service, db_using):
             with connections[db_using].cursor() as cursor:
                 table_name = model._meta.db_table
                 max_id = model.objects.count()
-                sql = f"SELECT setval(pg_get_serial_sequence('{table_name}', 'id'), {max_id});"
-                cursor.execute(sql)
+                if max_id:
+                    sql = f"SELECT setval(pg_get_serial_sequence('{table_name}', 'id'), {max_id});"
+                    cursor.execute(sql)
 
     assert sorted([val.pk for val in Article.objects.all().using(db_using)]) == [1, 2, 3, 4, 5]
     assert sorted([val.pk for val in Comment.objects.all().using(db_using)]) == [6, 7, 8, 9, 10]

tests/test_1_orm/test_schema_query.py

@@ -102,6 +102,7 @@ def test_init_articles(self, service, db_using):
         assert article.author_avg_articles_views == 6.5
         assert len(article.comments) == 2
         assert article.author_tag['name'] == 'bob'
+        assert db_using in article.tags
 
         # test sub relation
         content = ContentBase.init(1, context=orm.QueryContext(using=db_using))
@@ -275,6 +276,7 @@ async def test_async_serialize_articles(self, service, db_using):
         assert articles[0].pk == 1
         assert articles[0].author_avg_articles_views == 6.5
         assert articles[0].author.followers_num == 2
+        assert db_using in articles[0].tags
 
     def test_save(self, service, db_using):
         from app.schema import ArticleSchema