feat: first poc implementation #2

gersmann · 2024-01-21T16:36:55Z

Still pretty basic implementation of Django database datapter for MongoDB.

Reference material on the Django ORM architecture:
https://simpleisbetterthancomplex.com/media/2016/11/db.pdf

This is working:

Select/Insert/Update/Delete type queries.
First set of 'lookup' operations (easily extensible)
MongoDB search support (for lookups and wildcard queries)
'Aliasing' columns / mapping fields to nested mongo paths
Projections of queried fields and selective updates
Count Aggregation and Distinct Query
'Single Table Inheritance'

Notable omissions

Almost all aggregation queries
Joins (did some prototyping, pretty doable, but performance could be bad depending on database setup and schema)
Nested sub-queries (doable, needs some research on how it would perform on large collections).
Migrations are skipped, we are using a schema-less database, but could implement operations such as field renames, at the moment due to the size of our collections not really useful though
Plenty of other topics which need to be implemented / checked to achive full coverage of the ORM functionality

…ns, relations, single-table inheritance and single-table OneToOne relations. Not implemented yet are joins, aggregations (other than count), and embedded document arrays.

…tion

gersmann · 2024-01-21T16:38:42Z

Makefile

@@ -0,0 +1,9 @@
+setup_local_db:
+	@echo "Creating local database..."
+	@atlas deployments setup local --type local --port 3307 --force


atlas cli local deployments support mongo search

gersmann · 2024-01-21T16:40:17Z

django_mongodb/base.py

+        return self.settings_dict
+
+    def get_new_connection(self, conn_params):
+        name = conn_params.get("NAME") or "test"


Django instantiates the connection in db-less mode (without db name) for operations such as create / drop database.

gersmann · 2024-01-21T16:41:04Z

django_mongodb/client.py

+
+
+class DatabaseClient(BaseDatabaseClient):
+    executable_name = "django_mongodb"


this class implements the django sqlshell command, which I personally don't use, so this is optional, but should be fairly easy

gersmann · 2024-01-21T16:43:55Z

django_mongodb/compiler.py

+        self.extr = None
+
+    def build_mongo_filter(self, filter_expr):
+        referenced_tables = set()


A bare predicate filter, which is used as $match in query pipelines or as filter in update/delete.

gersmann · 2024-01-21T16:45:41Z

django_mongodb/compiler.py

+            {"$replaceRoot": {"newRoot": "$_id"}},
+        ]
+
+    def as_operation(self, with_limits=True, with_col_aliases=False):


MongoDB implementation of as_sql of the base sql compiler

gersmann · 2024-01-21T21:57:39Z

django_mongodb/base.py

+
+class DatabaseWrapper(BaseDatabaseWrapper):
+    vendor = "django_mongodb"
+    data_types = {


This is the base configuration Django, but adapted where needed (notably objectid). Those are currently logical names only, used in prep operations, as we are not generating / updating schematas..

gersmann · 2024-01-21T22:00:37Z

django_mongodb/base.py

+            with self.wrap_database_errors:
+                self.mongo_client.close()
+
+    def _set_autocommit(self, autocommit):


Does not currently implement transactions, we are also not using any in the backend as of now. But should be possible actually without too much effort to do so, using client_session, and doesn't look that complicated at a first glance, but would need some investigation. Biggest benefit at the moment would be in testing, but for the few models we are using a db cleaning auto-fixture would do.

gersmann · 2024-01-21T22:02:32Z

django_mongodb/compiler.py

+
+    @cached_property
+    def mongo_meta(self):
+        if hasattr(self.query.model, "MongoMeta"):


some custom meta class / props, at the moment only for defining what search indexes are set up on the collections, so that when doing queries via search, we can push the right lookups to the search index.

gersmann · 2024-01-21T22:05:13Z

django_mongodb/compiler.py

+
+    def execute_sql(
+        self, result_type=MULTI, chunked_fetch=False, chunk_size=GET_ITERATOR_CHUNK_SIZE
+    ):


This is the entry point into the compiler for QuerySet methods such as fetch_all, udpate, delete.

The implementation is analoguous, but adapted where needed, from the implementation in the base SQLCompiler.

gersmann · 2024-01-21T22:08:22Z

django_mongodb/compiler.py

+            cursor.close()
+            raise
+
+        if result_type == CURSOR:


this is used by raw queries, which rely on cursor methods such as rowcount for info about query results

gersmann · 2024-01-23T08:25:02Z

django_mongodb/query.py

+
+class MongoNothingNode(Node):
+    def get_mongo_query(self, compiler, connection, is_search=...) -> dict:
+        return {"$expr": {"$eq": [True, False]}}


needed for Queryset#none

gersmann · 2024-01-23T08:25:45Z

django_mongodb/query.py

+        self.alias = alias
+
+    def get_mongo(self):
+        return {"$project": {(self.alias or self.col.target.attname): f"${self.col.target.column}"}}


This maps the MongoDB field path to the attribute names we are defining in the MongoDB model.

gersmann · 2024-01-23T08:26:38Z

django_mongodb/query.py

+        self.query = query
+        self.order = query.order_by
+
+    def get_mongo_order(self, attname_as_key=False):


in aggregation queries, we already have the document with Django attribute names as keys and are setting attname_as_key=True

gersmann · 2024-01-23T08:29:04Z

django_mongodb/schema.py

+from django.db.backends.base.schema import BaseDatabaseSchemaEditor
+
+
+class DatabaseSchemaEditor(BaseDatabaseSchemaEditor):


Does not at the moment implement and schema migrations. We could add support for field renames and type conversions here, if we'd have use for that.

gersmann · 2024-01-23T08:31:49Z

testapp/models.py

+    extended = models.CharField(max_length=100)
+
+    class Meta:
+        db_table = "testapp_foomodel"


This child model is persistent with it's and it's parents fields in the same collection.

gersmann and others added 27 commits January 21, 2024 17:37

Implements a MongoDB backend for Django, which supports CRUD operatio…

0ce142f

…ns, relations, single-table inheritance and single-table OneToOne relations. Not implemented yet are joins, aggregations (other than count), and embedded document arrays.

chore: ruff + black + pre-commit

3da313b

fix: bug in row item lookup

dce303e

fix: getter for null values

9b02409

fix: get prep value

50c75bd

chore: guard rhs sub-query expressions / fix aggregation result

ef64e75

fix: use proper pk field if order by pk

9703122

chore: reset migrations, mongodb as secondary, better distinct aggrea…

6db5161

…tion

fix: post-filter search results + more docs

f67603b

chore: search improvements

47bc345

chore: search improvements

e5c5a0c

chore: update version tag

31711ed

chore: github actions

6cba184

Change version to 0.19

7334437

Change version to 0.20

c754f69

chore: search query bug

f709594

chore: fmt

45d03f6

Change version to 0.21

4cb8e61

chore: tests for mongodb search

1d26501

fix: wrong init file name

84c65cd

fix: wrong init file name

c78e0f4

fix: unused fixture

11c56b7

fix: implements NothingNode

ec577b6

Change version to 0.22

1f2159b

chore: refactoring query nodes for reuse

e682df9

chore: db prep

700b6ce

fix: operator rename

fb4cd97

gersmann commented Jan 21, 2024

View reviewed changes

gersmann added 2 commits January 21, 2024 21:56

chore: publish

409c503

chore: publish (fix)

8c6ebbb

gersmann changed the title ~~feat: first poc implementaiton~~ feat: first poc implementation Jan 21, 2024

gersmann commented Jan 23, 2024

View reviewed changes

Change version to 0.19

871b985

gersmann force-pushed the review-branch branch from dbdc837 to 1b66fca Compare January 23, 2024 11:07

chore: small cleanup

0a708d8

gersmann force-pushed the review-branch branch from 1b66fca to 0a708d8 Compare January 23, 2024 12:39

gersmann marked this pull request as ready for review January 23, 2024 12:58

gersmann added 3 commits January 25, 2024 18:20

chore: skip empty rhs search filters

be3fd5b

chore: add support for autocomplete search

4983c2b

chore: add shorthand to retrieve collection from cursor

93f9ac2

gersmann force-pushed the review-branch branch from 31f2b60 to e8757a8 Compare February 5, 2024 21:19

feat: implements raw mongo query

3dc452f

gersmann force-pushed the review-branch branch from e8757a8 to 3dc452f Compare February 5, 2024 21:22

gersmann added 2 commits February 6, 2024 20:50

feat: search fixes + test update

eb68891

chore: update pyproject.toml + django 5.0 support

83a8452

gersmann merged commit 55c3faf into main-empty Apr 7, 2024
1 check failed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: first poc implementation #2

feat: first poc implementation #2

gersmann commented Jan 21, 2024 •

edited

Loading

gersmann Jan 21, 2024

gersmann Jan 21, 2024

gersmann Jan 21, 2024

gersmann Jan 21, 2024

gersmann Jan 21, 2024

gersmann Jan 21, 2024

gersmann Jan 21, 2024

gersmann Jan 21, 2024

gersmann Jan 21, 2024

gersmann Jan 21, 2024

gersmann Jan 23, 2024

gersmann Jan 23, 2024

gersmann Jan 23, 2024

gersmann Jan 23, 2024

gersmann Jan 23, 2024



		class DatabaseClient(BaseDatabaseClient):
		executable_name = "django_mongodb"

		from django.db.backends.base.schema import BaseDatabaseSchemaEditor


		class DatabaseSchemaEditor(BaseDatabaseSchemaEditor):

feat: first poc implementation #2

feat: first poc implementation #2

Conversation

gersmann commented Jan 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gersmann commented Jan 21, 2024 •

edited

Loading