LMDB Store and syncing to disk

[hmottestad]

Does the LMDB Store use MDB_NOSYNC for transactions with NONE as isolation level?

If not then maybe this can be used to improve performance for bulk loading data.

[kenwenzel]

It always uses MDB_NOSYNC if force sync is not activated in the configuration. Maybe we should change this as it might be unsafe for production environments.

@hmottestad Have you noticed performance issues with the LMDB store?

One issue might be that I've separated the values and the triple store into two LMDB databases. If the RAM starts getting füll and the memory pressure grows then it might be the case that both databases influence each other by forcing the OS to swap pages.

[hmottestad]

I was reading a comparison between LMDB and RocksDB. One of the differences was that RocksDB performed writes in the background, while the default behaviour of LMDB was to always sync each write. This made RocksDB seem a lot faster than LMDB.

Isolation levels don't really say anything about durability, so my suggestion isn't really that good. Maybe we could introduce a transaction setting for syncing to disk. For LMDB we could implement it by calling mdb_env_sync.

We should probably also call mdb_env_sync on some regular interval, maybe based on number of triples, transactions and time. That seems recommended when using MDB_NOSYNC.

MDB_NOSYNC Don't flush system buffers to disk when committing a transaction. This optimization means a system crash can corrupt the database or lose the last transactions if buffers are not yet flushed to disk. The risk is governed by how often the system flushes dirty buffers to disk and how often mdb_env_sync() is called. However, if the filesystem preserves write order and the MDB_WRITEMAP flag is not used, transactions exhibit ACI (atomicity, consistency, isolation) properties and only lose D (durability). I.e. database integrity is maintained, but a system crash may undo the final transactions. Note that (MDB_NOSYNC | MDB_WRITEMAP) leaves the system with no hint for when to write transactions to disk, unless mdb_env_sync() is called. (MDB_MAPASYNC | MDB_WRITEMAP) may be preferable. This flag may be changed at any time using mdb_env_set_flags().

Would also be nice to better understand what kind of corruption LMDB can handle, especially since you are using two LMDB databases which I assume would mean that you have to call commit on each of them. I know that the NativeStore is fairly bad at handling corruption. If you pull the plug while a transaction is being committed there is a fairly large chance that the database becomes corrupt and unrecoverable.

[kenwenzel]

The main difference is that RocksDB first writes into a WAL and then adds data to sorted files by using background compaction tasks while LMDB directly modifies memory pages in a copy on write fashion. When starting with a new RDF4J storage backend I also tried the LevelDB Java port and in other projects RocksDB via JNI.
Both showed bad insertion performance in comparison to LMDB although theoretical they should be faster (LSM vs BTree).
We need some benchmarks with a lot of triples (more than 1 billion) to observe the performance degradation of LMDB when the database gets (much) larger than RAM.

This is a short article that summarizes some points about LMDB:
https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8880032