Persistence by SQLite

[ArmanMirzaei]

Greetings to you all,

I was thinking that if the respected developers have been dedicated and generous enough to have invested the time and devotion to bring the project up to this point;
Then why not adding an SQLite backend to it, in order to make things a little bit more serious and take them to the next level?
I mean the hardest part is already done and the project is very elegant, it just ignores the local machine scalability ability.

SQLite is embedded in python, and by doing this we would make the first embedded-in-python graph database just like the java people easily have, and things get to somewhat more professional for this masterpiece handcrafted project.
All it take is to store and index things through the very ergonomic python interface for the very stable SQLite backend.
I'm just suggesting the possibility and I wish the debate on it goes on ;)

High Regards
Arman
on 2022-8-9 4:30 UTC

[ArmanMirzaei]

It is not just that, I mean we should be purpose oriented with the work we put, the jobs we do, and the time we invest.
For example, What are we going to get at the end, where are we gonna lead, what are the achievements gonna be.

What are the jobs that can be done using a graph database? - well, many ai algorithms being one.
Is there any other person interested for leading towards a full-scale graph db on the top of this elegant project?

The question is that, is there enough qualitative sum of intent among us, so that we can have one more last development phase, leading to a python but application ready graph database? meaning developing at least all those basic features provided in (say) orientdb, within our pythonic platform?

Anyone in?

[arun1729]

hey @ArmanMirzaei thanks for the interest in this project! As for using not using SQLite (or anything else) as a storage layer was a deliberate choice. In the case of SQLite, the reason is that SQL (relational model) and graph are fundamentally different ways to model data. To use a relational DB as a storage layer, we would have to transform a graph in to a relational model to store it and then do the same in reverse while querying it. This is quite inefficient, honestly makes things very complicated and limiting. The vision behind CogDB is to create a graph DB that is self sufficient, easy to learn and pleasant to use.

[ArmanMirzaei]

the data models of graph databases are a subset of the possible data models of relational databases, so they are the same thing plus a one level higher layer, not an entirely different thing.

we can merge this project in if we decide to use sqlite:
https://github.com/dpapathanasiou/simple-graph

but ultimately you are right. sqlite has not been designed from scratch with the prospect of cogdb being built on top of it.
alternatively we can build on top of the unix file system as well, making things as efficient and accessible as python gets. some more work, some more algorithms development, but more portability and pythonicity ; )

but i was wondering that isn't persistency a real bottleneck?
i mean the common civil computers used everywhere come with the assumption of not using the ram for high level data that is not about handling the hardware its self. so while professional and industrial computers may have terabytes of ram, for the rest of us we only have gigabytes.
so there are certainly serious bottlenecks for effectively deploying the cogdb to real world applications, at least on consumer computers.

could i have changed your mind, about the necessity of using storage hardware, even for the data storage of high level knowledge such as knowledge graphs ? i mean everything is files in unix, if some data can not be filed and partially loaded, then it can not be scaled.
please note that i do not suggest network scaling at all, as it may not be suitable for autonomous thinking processes and ai applications which are an important application of the graph data systems. i only suggest scaling with the storage hardware directly connected to the device.

[arun1729]

You mention alternatively we can build on top of the unix file system and isn't persistency a real bottleneck and also talk about limitations of RAM on computers - if persistence is your concern, then let me clarify that Cogdb is a persistent database (all data is stored on disk on a file system such as the unix file system), it is not an in-memory store.