9.0.0.md

9.0.0

So I was working hard the past month to prepare everything for the next major release of RxDB. The last major release was 1,5 years ago by the way.

When I started listing up the planned changes I had big ambitions about basically rewriting everything. But I found out this time has not come yet. There is some work to be done first. So version 9.0.0 was more about fixing all these small things that made improving the codebase difficult. Much has been refactored and moved. Some API-parts have been changed to have a more simple project with a cleaner codebase.

Notice that I use major releases to bundle stuff that breaks the RxDB usage in your project. By having only few major releases you can be sure that you can upgrade in one big block instead of changing stuff each few months. Big features are released in non-major releases because they mostly can be implemented without side effects.

Breaking changes

You have to apply these changes to your codebase when upgrading RxDB.

All default exports have been removed

Using default exports and imports can be helpfull when you want to write code fast. But using them also disabled the tree-shaking of your bundler which means you added much code to your bundle that was not even used. To prevent this common behavior, I removed all default exports and renamed functions so that they are more unlikely to clash with other non-RxDB function names.

Instead of doing

import RxDB from 'rxdb';
RxDB.plugin(/* ... */);
await RxDB.create({/* ... */});

You now do

import { createRxDatabase, addRxPlugin } from 'rxdb';
addRxPlugin(/* ... */);
await createRxDatabase({/* ... */});

Also removeDatabase() is renamed to removeRxDatabase() and plugin() is now addRxPlugin().

Same goes for all previous default exports of the plugins.

Indexes are specified at the top level of the schema definition

related issue

In the past the indexes of a collection had to be specified at the field level of the schema like

{
    "firstName": {
        "type": "string",
        "index": true
    }    
}

This made it complex to list up the index fields which had a bad performance on startup. To fix this the indexes are now specified at the top level of the schema like

{
    "title": "my schema",
    "version": 0,
    "type": "object",
    "properties": {},
    "indexes": [
        "firstName",
        ["compound", "index"]
    ]
}

Encrypted fields at the top level of the schema

Same as the indexes, encrypted fields are now also defined in the top level like

{
    "title": "my schema",
    "version": 0,
    "type": "object",
    "properties": {},
    "encrypted": [
        "password"
    ]
}

New dev-mode plugin

In the past we had stuff that is only wanted for development in the two plugins error-messages and schema-check.

Now we have a single plugin dev-mode that contains all these checks and development helper functions. I also moved many other checks out of the core-module into dev-mode.

import { RxDBDevModePlugin } from 'rxdb/plugins/dev-mode';
addRxPlugin(RxDBDevModePlugin);

New migration plugin

The data migration was not used by all users of this project. Often it was easier to just wipe the local data store and let the client sync everything from the server. Because the migration has so much code, it is now in a seperate plugin so that you do not have to ship this code to the clients if not necessary.

import { RxDBMigrationPlugin } from 'rxdb/plugins/migration';
addRxPlugin(RxDBMigrationPlugin);

Rewritten key-compression

The key-compression logic was fully coded only for RxDB. This was a problem because it was not useable for other stuff and also badly tested. We had a known problem with nested arrays that caused much confusion because some queries did not find the correct documents.

I now created a npm-package jsonschema-key-compression that has cleaner code, better tests and can also be used for non-RxDB stuff.

If you used the key-compression in the past and have clients out there with old data, you have to find a way to migrate that data by using the json-import or other solutions depending on your project.

Rewritten query-change-detection to event-reduce

One big benefit of having a realtime database is that big performance optimizations can be done when the database knows a query is observed and the updated results are needed continuously. In the past this optimization was done by the internal queryChangeDetection which was a big tree of if-else-statements that hopefully worked. This was also the reason why queryChangeDetection was in beta mode and not enabled by default.

After months of research and testing I was able to create Event-Reduce: An algorithm to optimize database queries that run multiple times. This JavaScript module contains an algorithm that is able to optimize realtime queries in a way that was not possible before. The algorithm is not RxDB specific and also heavily tested.

Instead of setting queryChangeDetection when creating a RxDatabase, you now set eventReduce which defaults to true.

find() and findOne() now accepts the full mango query

In the past, only the selector of a query could be passed to find() and findOne() if you wanted to also do sort, skip or limit, you had to call additional functions like

const query = myRxCollection.find({
    age: {
        $gt: 10
    }
}).sort('name').skip(5).limit(10);

Now you can pass the full query to the function call like

const query = myRxCollection.find({
    selector: {
        age: {
            $gt: 10
        }
    },
    sort: [{name: 'asc'}],
    skip: 5,
    limit: 10
});

moved query builder to own plugin

The query builder that allowed to create queries like .where('foo').eq('bar') etc. was not really used by many people. Most of the time it is better to just pass the full query as simple json object. Also the code for the query builder was big and increased the build size much more then its value added. Only some edge-cases where recursive query modification was needed made the query builder usefull. If you still want to use the query builder, you have to import the plugin.

import { RxDBQueryBuilderPlugin } from 'rxdb/plugins/query-builder';
addRxPlugin(RxDBQueryBuilderPlugin);

Refactored RxChangeEvent

The whole data structure of RxChangeEvent was way more complicated than it had to be. I refactored the whole class to be more simple. If you directly use RxChangeEvent in your project you have to adapt to these changes. Also the stream of RxDatabase().$ will no longer emit the COLLECTION event when a new collection is created.

Internal hash() is now using a salt

The internal hash function was used to store hashes of database passwords to compare them and directly throw errors when the wrong password was used with an existing data set. This was dangerous because you could use rainbow tables or even just google the hash to find out the plain password. So now the internal hashing is using a salt to prevent these attacks. If you have used the encryption in the past, you have to migrate your internal schema storage.

Changed default of RxDocument.toJSON()

By default RxDocument.toJSON() always returned also the _rev field and the _attachments. This was confusing behavior which is why I changed the default to RxDocument().toJSON(withRevAndAttachments = false)

Typescript 3.8.0 or newer is required

Because RxDB and some subdependencies extensively use export type ... you now need typescript 3.8.0 or newer.

GraphQL replication will run a schema validation of incoming data

In dev-mode, the GraphQL-replication will run a schema validation of each document that comes from the server before it is saved to the database.

Internal and other changes

I refactored much internal stuff and moved much code out of the core into the specific plugins.

• Renamed RxSchema.jsonID to RxSchema.jsonSchema
• Moved remaining stuff of leader-election from core into the plugin
• Merged multiple internal databases for metadata into one internalStore
• Removed many runtime type checks that now should be covered by typescript in buildtime
• The GraphQL replication is now out of beta mode
• Removed documentation examples for require() CommonJS loading
• Removed RxCollection.docChanges$() because all events are from the docs

Help wanted

RxDB is an open source project an heavily relies on the contribution of its users. There are some things that must be done, but I have no time for them.

Refactor data-migrator

The current implementation has some flaws and should be completely rewritten.

• It does not use pouchdb's bulkDocs which is much faster
• It could have been written without rxjs and with less code that is easier to understand
• It does not migrate the revisions of documents which causes a problem when replication is used

Add e2e tests to the react example

The react example has no end-to-end tests which is why the CI does not ensure that it works all the time. We should add some basic tests like we did for the other example projects.

Fix pouchdb bug so we can upgrade pouchdb-find

There is a bug in pouchdb that prevents the upgrade of pouchdb-find. This is why RxDB relies on an old version of pouchdb-find that also requires different sub-dependencies. This increases the build size a lot because for example we ship multiple version of spark-md5 and others.

About the future of RxDB

At the moment RxDB is a realtime database based on pouchdb. In the future I want RxDB to be a wrapper around pull-based databases that also works with other source-dbs like mongoDB or PostgreSQL. As a start I defined the RxStorage interface and created a RxStoragePouchdb class that implements it and contains all pouchdb-specific logic. I want to move every direct storage usage into that interface so that later we can create other implementations of it for other source databases.