Rxdb #1987

hsmr29 · 2024-05-31T01:04:23Z

Remove unused ThoughtspaceOption properties from init
Efficient observe mechanism for useObserveCol
Key metrics: Load 1k thoughts, Delete 1k thoughts (I will send new sample data)
rxdm-premium deployment strategy
Get CI tests to pass
freeThought/freeLexeme implementation

raineorshine · 2024-06-13T19:44:10Z

src/data-providers/yjs/thoughtspace.ts

@@ -272,140 +96,10 @@ export const init = async (options: ThoughtspaceOptions) => {
 const websocketUrl = await options.websocketUrl
 const cursor = await options.cursor

- // generate docKeys for cursor, otherwise replicateThought will fail


You can remove the following from ThoughtspaceOptions and this file. They were used by the replication controller which has been replaced with RxDB replication:

accessToken

cursor

getItem

setItem

tsid

tsidShared

websocketUrl

We can discuss the migration of the following thoughtspace options:

isThoughtLoaded/isLexemeLoaded - Used to subscribe to remote changes that are visible to the user, i.e. thoughts or lexemes that are loaded in the Redux state.

onError - Dispatches an error action when there is a db error.

onProgress - Updates the UI with replication progress (entire thoughtspace) and save progress (when local changes have been synced to the server).

onThoughtChange - Marks a thought as pending if its parent is pending.

onThoughtReplicated - Triggers repairThoughtActionCreator to run data integrity checks on the thought tree structure.

onThoughtIDBSynced - Used to dismiss the "Connecting..." message as soon as thoughts are loaded from the local db.

onUpdateThoughts - Removes a thought from the Redux state when it has been deleted remotely. Adds a Lexeme when it has been remotely loaded or changed. (Does not check if the Lexeme is currently loaded in Redux state.)

raineorshine

Thanks, this looks good. My only major concern is about race conditions and transaction atomicity, mentioned in this comment. Let's talk about this at our next meeting.

I think we can safely reorganize the file structure now that YJS is completely removed:

Combine src/data-prociders/yjs/thoughtspace.ts with src/data-prociders/rxdb/thoughtspace.ts
Move everything in src/data-providers/yjs and src/data-providers/rxdb up a level to to src/data-providers.

raineorshine · 2024-07-08T21:33:08Z

scripts/rxdb-premium.js

@@ -0,0 +1,142 @@
+import { execSync } from 'child_process'


Please add a summary comment to the top of this script for documentation purposes.

raineorshine · 2024-07-08T21:33:56Z

src/components/__tests__/ColorPicker.ts

-it('Set the text color using the ColorPicker', async () => {
+it.skip('Set the text color using the ColorPicker', async () => {


Please add a comment describing why this test is being skipped.

raineorshine · 2024-07-08T21:41:48Z

src/hooks/useObserveCol.ts

+/**
+ * Subscribes to a rxdb collection, e.g. RxCollection<RxPermission>.
+ */
+const useObserveCol = <T>(rxCol: RxCollection<T>): Document<T>[] => {


Let's call this useRxCollection.

raineorshine · 2024-07-08T21:52:00Z

src/data-providers/rxdb/thoughtspace.ts

+ return createRxDatabase({
+ name: `${testPrefix}${DATABASE_NAME}`,
+ storage: await getRxStorage(),
+ ignoreDuplicate: isTestEnv,


Can you explain ignoreDuplicates and why it is needed when in test mode?

raineorshine · 2024-07-08T22:00:42Z

src/data-providers/rxdb/schemas/lexeme.ts

+ properties: {
+ id: {
+ type: 'string',
+ maxLength: 32,


Let's create a constant for this id type.

Also let me know what the reasoning is behind the value 32.

raineorshine · 2024-07-08T22:16:22Z

src/data-providers/yjs/permissionsModel.ts

+ } catch (error) {
+ return { error }
+ }


If a function can't handle an error, it should throw it up the call stack rather than return it.

raineorshine · 2024-07-08T22:28:43Z

src/data-providers/yjs/thoughtspace.ts

+ console.info(
+ 'TODO_RXDB: thoughtspace.freeThought - The thought is no longer visible and should be removed from memory. Realtime changes to this thought from other clients no longer need to be subscribed to.',
+ { id: docKey },
+ )


You can remove each of these console.info debugging statements. They were just there to assist in the RxDB conversion.

raineorshine · 2024-07-08T22:32:08Z

src/data-providers/yjs/thoughtspace.ts

+
+ return foundThoughtDoc.toJSON() as Thought
+ })
+}

 /** Pauses replication for higher priority network activity, such as push or pull. */
 export const pauseReplication = async () => {


Do we have the ability to pause replication in RxDB? If not, we can remove pauseReplication and startReplication.

Yes, we can pause replication in RxDB.

raineorshine · 2024-07-08T22:52:34Z

src/data-providers/yjs/thoughtspace.ts

@@ -435,876 +93,167 @@ export const init = async (options: ThoughtspaceOptions) => {
 * Methods
 **********************************************************************/

-/** Updates a yjs thought doc. Converts childrenMap to a nested Y.Map for proper children merging. Resolves when transaction is committed and IDB is synced (not when websocket is synced). */
-// NOTE: Ids are added to the thought log in updateThoughts for efficiency. If updateThought is ever called outside of updateThoughts, we will need to push individual thought ids here.
+/** Updates a thought in the db. */
 export const updateThought = async (id: ThoughtId, thought: Thought): Promise<void> => {


I'm concerned about the possibility of race conditions in updateThought due to all the awaits. With YJS, changes are synchronous. Here, between the await getThoughtById(id), await getLexemeById(lexemeKey), and await thoughtCollection.findOne, it seems like there are many opportunities for the data to have changed in between awaited functions, resulting in inconsistencies.

What's preventing the data in the db from changing in between awaits? How to perform the update as an atomic transaction?

raineorshine · 2024-07-08T22:58:07Z

src/data-providers/yjs/thoughtspace.ts

+ ...Object.entries(thoughtUpdates || {}).map(([id, thought]) => updateThought(id as ThoughtId, thought)),
+ ...Object.entries(lexemeUpdates || {}).map(([key, lexeme]) => updateLexeme(key, lexeme)),


Can these updates be batched for better performance?

raineorshine · 2024-07-08T23:06:31Z

Also, what is this? It shows up in the console when I clear the local state and refresh.

…ough's collection

…s when using RxDB

… remove USE_RXDB flag

… to Lexemes

…, Remove docKeys.

- Destroy Rx database when db.clear method is called

…base in the clear function

- replicateThought - replicateChildren - replicateLexeme

… to avoid side effects

… file

- rxdb-server: Create RxDB server to replicate with the WebRTC plugin - rxdb/thoughtspace: Initial implementation for replication with the WebRTC controller, Add a simple custom conflict handler

hsmr29 force-pushed the rxdb branch 4 times, most recently from f6ff14d to 0a8ecf4 Compare June 17, 2024 11:11

raineorshine reviewed Jun 17, 2024

View reviewed changes

hsmr29 force-pushed the rxdb branch 2 times, most recently from 0fe9dd0 to bd04524 Compare June 24, 2024 19:21

hsmr29 force-pushed the rxdb branch 3 times, most recently from 0fc0d3d to c34e43f Compare July 3, 2024 20:01

raineorshine requested changes Jul 8, 2024

View reviewed changes

raineorshine mentioned this pull request Jul 9, 2024

Fix ModalExport tests #2130

Open

hsmr29 force-pushed the rxdb branch from c34e43f to 56d93c8 Compare July 11, 2024 13:46

This was referenced Jul 12, 2024

Fix pullQueue tests #2139

Open

Fix pushQueue tests #2140

Open

Fix deleteThought tests #2141

Open

Fix moveThought shortcut test #2143

Open

Fix puppeteer pull test #2148

Open

raineorshine and others added 11 commits July 22, 2024 17:36

Add TODO_RXDB logging.

d1d3015

Move thoughtspace from web worker back to main thread.

524336b

Initial RxDB implementation, add libraries, create a database, and th…

7e21731

…ough's collection

add RxDB logic into yjs/thoughspace with a USE_RXDB flag

c5c0431

thoughtspace: improve updateThought and replicateChildren for thought…

034a355

…s when using RxDB

thoughtspace: remove Yjs code related to thoughts where possible, and…

c8d881b

… remove USE_RXDB flag

Create RxDB Lexeme schema and collection

fd1e9a7

thoughtspace: Update lexemes to use Rxdb, and remove Yjs code related…

26c5e1f

… to Lexemes

thoughtspcae: Remove Yjs/hocuspocus/y-indexeddb related code

8c03046

thoughtspace: Use findByIds when getting multiple thoughts or lexemes…

bbe27ce

…, Remove docKeys.

Update thoughts schema

db06da8

hsmr29 added 22 commits July 22, 2024 17:37

Add freeThought back to the thoughtspace api

765a349

- Create RxDB test database with RxStorageMemory storage

a1f13d3

- Destroy Rx database when db.clear method is called

thoughtspace: Use collection.remove method instead of destroying data…

e2f8e49

…base in the clear function

Add rxdb-premium script to build the project using the IndexedDB storage

df6df7c

Rename build premium script

0014df9

rxdb-premium script: Check if rxdb-premium library is already installed.

9e2890e

thoughtspace: Simplify and mark as deprecated the following functions:

cecfc19

- replicateThought - replicateChildren - replicateLexeme

thoughtspace: Add "await" to the rxdb calls in updateThought function…

30bdd50

… to avoid side effects

thoughtspace: Update comments and minor refactor to lexeme functions

beca6a4

Use Dexie storage for testing as well

1535357

Fix tests for Dexie Storage, Remove fakeTimer from vitest

8ff2763

Add random number to rxdb database name on test environment

67a0b47

Update RxDB library to latest version

502f4a9

Skip failing test on CI

fcd00ab

rxdb-premium: Add summary comment for documentation purposes

d169490

ColorPicker: Add comment to explain why we disabled the tests in this…

c529d03

… file

useRxCollection: rename useObserveCol hook to useRxCollection

c5b5ad9

thoughtspace: remove unnecessary console.info

443c309

Update RxDB from 15.26.0 to 15.28.0

36f6ad3

thoughspace: bulk update/delete for thoughts and lexemes

1304543

thoughtspace: add Rx observers for thoughts/lexemes

2fe63bb

Add RxDB WebRTC replication:

29a9b95

- rxdb-server: Create RxDB server to replicate with the WebRTC plugin - rxdb/thoughtspace: Initial implementation for replication with the WebRTC controller, Add a simple custom conflict handler

hsmr29 force-pushed the rxdb branch from d2a6890 to 29a9b95 Compare July 22, 2024 21:40

raineorshine force-pushed the main branch 7 times, most recently from 80019f7 to 10b9d7c Compare August 21, 2024 12:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rxdb #1987

Rxdb #1987

hsmr29 commented May 31, 2024 •

edited by raineorshine

Loading

raineorshine Jun 13, 2024

raineorshine left a comment •

edited

Loading

raineorshine Jul 8, 2024

raineorshine Jul 8, 2024

raineorshine Jul 8, 2024

raineorshine Jul 8, 2024

raineorshine Jul 8, 2024

raineorshine Jul 8, 2024

raineorshine Jul 8, 2024

raineorshine Jul 8, 2024

hsmr29 Jul 15, 2024

raineorshine Jul 8, 2024

raineorshine Jul 8, 2024

raineorshine commented Jul 8, 2024

		it('Set the text color using the ColorPicker', async () => {
		it.skip('Set the text color using the ColorPicker', async () => {

		...Object.entries(thoughtUpdates \|\| {}).map(([id, thought]) => updateThought(id as ThoughtId, thought)),
		...Object.entries(lexemeUpdates \|\| {}).map(([key, lexeme]) => updateLexeme(key, lexeme)),

Rxdb #1987

Are you sure you want to change the base?

Rxdb #1987

Conversation

hsmr29 commented May 31, 2024 • edited by raineorshine Loading

Choose a reason for hiding this comment

raineorshine left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raineorshine commented Jul 8, 2024

hsmr29 commented May 31, 2024 •

edited by raineorshine

Loading

raineorshine left a comment •

edited

Loading