enh: Implement PrimaryReadReplicaConnection #41998

juliusknorr · 2023-12-04T09:41:22Z

First approach for #33542 split to only have the read replica support in this PR. Follow up prepared in #42345

Summary

Make use of the PrimaryReadReplicaConnection class that Doctrine offers to be able to configure read replicas for Nextcloud databases.

Library docs: https://github.com/doctrine/dbal/blob/3.7.x/src/Connections/PrimaryReadReplicaConnection.php#L28C1-L76

Checklist

Code is properly formatted
Sign-off message is added to all commits
Tests (unit, integration, api and/or acceptance) are included
Screenshots before/after for front-end changes
Documentation (manuals or wiki) has been updated or is not required
- feat(adminmanual): Add db replica docs documentation#11446
Backports requested where applicable (ex: critical bugfixes)

lib/private/DB/ConnectionFactory.php

lib/private/DB/SetTransactionIsolationLevel.php

mickenordin · 2023-12-12T10:14:32Z

Is this ready for testing, or does it need more work you think?

lib/private/DB/ConnectionFactory.php

lib/private/DB/Middleware/SetAutocommitMiddleware.php

lib/private/DB/Middleware/SetTransactionIsolationLevelMiddleware.php

lib/private/DB/Connection.php

Signed-off-by: Julius Härtl <jus@bitgrid.net>

juliusknorr · 2023-12-18T09:52:09Z

Should be good now for testing @mickenordin :)

nickvergessen

I can see how wrong development can nullify the efforts, but it's a start and can only get better forward

mickenordin · 2023-12-18T10:32:07Z

Is this only for performance, or will there be failover to a replica, if the primary fails?

juliusknorr · 2023-12-18T11:36:22Z

Is this only for performance, or will there be failover to a replica, if the primary fails?

Only for performance and read only replicas.

juliusknorr · 2023-12-18T15:39:54Z

I can see how wrong development can nullify the efforts, but it's a start and can only get better forward

Just to mention, there is some follow up for that already in the queue at #42345

ChristophWurst

Makes sense!!

kesselb · 2023-12-27T20:55:42Z

lib/private/DB/ConnectionFactory.php

@@ -237,7 +233,11 @@ public function createConnectionParams(string $configPrefix = '') {
 			$connectionParams['persistent'] = true;
 		}

-		return $connectionParams;
+		$replica = $this->config->getValue('dbreplica', []) ?: [$connectionParams];


I found that one a bit strange.

Isn't that the same as

$this->config->getValue('dbreplica', [$connectionParams]);

?

kesselb

Cool 👍

Also without configuring dbreplica it still works 🙌

smorimoto · 2023-12-28T05:16:34Z

This is really fantastic! I was really thinking about whether I should set up YugabyteDB or not!

smorimoto · 2023-12-28T05:39:37Z

~~How can I use different port(s)?~~

smorimoto · 2024-01-03T15:13:22Z

@juliushaertl Can you set this as the milestone for 28? It would be great to include this in the next minor release!

juliusknorr · 2024-01-03T15:43:19Z

We only backport bugfixes and this is a larger enhancement so currently only targeting Nextcloud 29.

smorimoto · 2024-01-03T16:03:09Z

That's a sad thing... I'm not familiar with the release cycle of Nextcloud, so when will 29 be released?

juliusknorr · 2024-01-03T16:23:40Z

Current planned schedule is in https://github.com/nextcloud/server/wiki/Maintenance-and-Release-Schedule

juliusknorr mentioned this pull request Dec 4, 2023

Primary/Secondary choosing querybuilder #33542

Closed

juliusknorr commented Dec 4, 2023

View reviewed changes

lib/private/DB/ConnectionFactory.php Outdated Show resolved Hide resolved

juliusknorr commented Dec 4, 2023

View reviewed changes

lib/private/DB/SetTransactionIsolationLevel.php Outdated Show resolved Hide resolved

juliusknorr added enhancement 2. developing Work in progress labels Dec 4, 2023

juliusknorr added this to the Nextcloud 29 milestone Dec 4, 2023

juliusknorr added the performance 🚀 label Dec 4, 2023

github-advanced-security bot found potential problems Dec 4, 2023

View reviewed changes

lib/private/DB/SetTransactionIsolationLevel.php Fixed Show resolved Hide resolved

juliusknorr force-pushed the enh/read-replica branch from 65e3016 to 2d2ef65 Compare December 15, 2023 20:52

github-advanced-security bot found potential problems Dec 15, 2023

View reviewed changes

lib/private/DB/ConnectionFactory.php Fixed Show fixed Hide fixed

lib/private/DB/Middleware/SetAutocommitMiddleware.php Fixed Show fixed Hide fixed

lib/private/DB/Middleware/SetTransactionIsolationLevelMiddleware.php Fixed Show fixed Hide fixed

juliusknorr force-pushed the enh/read-replica branch 2 times, most recently from 65089ee to 55cfca8 Compare December 16, 2023 12:09

github-advanced-security bot found potential problems Dec 16, 2023

View reviewed changes

lib/private/DB/Connection.php Fixed Show fixed Hide fixed

juliusknorr force-pushed the enh/read-replica branch 4 times, most recently from 6882810 to 1a1e6f9 Compare December 16, 2023 19:17

enh: Implement PrimaryReadReplicaConnection

79c4986

Signed-off-by: Julius Härtl <jus@bitgrid.net>

juliusknorr force-pushed the enh/read-replica branch from 1a1e6f9 to 79c4986 Compare December 16, 2023 19:27

juliusknorr mentioned this pull request Dec 18, 2023

feat: Track dirty table writes and long transactions #42345

Merged

2 tasks

juliusknorr marked this pull request as ready for review December 18, 2023 09:51

juliusknorr requested review from nickvergessen, ChristophWurst and kesselb December 18, 2023 09:52

juliusknorr added 3. to review Waiting for reviews pending documentation This pull request needs an associated documentation update and removed 2. developing Work in progress labels Dec 18, 2023

nickvergessen approved these changes Dec 18, 2023

View reviewed changes

ChristophWurst approved these changes Dec 27, 2023

View reviewed changes

kesselb reviewed Dec 27, 2023

View reviewed changes

kesselb merged commit b9b0ca6 into master Dec 27, 2023
52 checks passed

kesselb deleted the enh/read-replica branch December 27, 2023 20:57

juliusknorr mentioned this pull request Dec 28, 2023

fix: Avoid using replicas during setup #42504

Merged

This was referenced Jan 17, 2024

fix(db): Fix replica password example config #42890

Merged

feat(adminmanual): Add db replica docs nextcloud/documentation#11446

Merged

ChristophWurst mentioned this pull request Feb 1, 2024

[stable28] feat(database): Primary/replica split #43261

Closed

10 tasks

blizzz mentioned this pull request Mar 5, 2024

29.0.0 beta 1 #43991

Merged

This was referenced Apr 24, 2024

[Bug]: Two database connections and missing transaction isolation level for read operations #45003

Closed

fix(db): Prevent two connections for single node databases #45013

Merged

joshtrichards mentioned this pull request May 18, 2024

fix(db): Prevent data loss by temporarily disabling db:convert-type #45400

Merged

4 tasks

juliusknorr removed the pending documentation This pull request needs an associated documentation update label Jul 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enh: Implement PrimaryReadReplicaConnection #41998

enh: Implement PrimaryReadReplicaConnection #41998

juliusknorr commented Dec 4, 2023 •

edited by hweihwang

Loading

mickenordin commented Dec 12, 2023

juliusknorr commented Dec 18, 2023

nickvergessen left a comment

mickenordin commented Dec 18, 2023

juliusknorr commented Dec 18, 2023

juliusknorr commented Dec 18, 2023

ChristophWurst left a comment

kesselb Dec 27, 2023

kesselb left a comment

smorimoto commented Dec 28, 2023

smorimoto commented Dec 28, 2023 •

edited

Loading

smorimoto commented Jan 3, 2024

juliusknorr commented Jan 3, 2024

smorimoto commented Jan 3, 2024

juliusknorr commented Jan 3, 2024

enh: Implement PrimaryReadReplicaConnection #41998

enh: Implement PrimaryReadReplicaConnection #41998

Conversation

juliusknorr commented Dec 4, 2023 • edited by hweihwang Loading

Summary

Checklist

mickenordin commented Dec 12, 2023

juliusknorr commented Dec 18, 2023

nickvergessen left a comment

Choose a reason for hiding this comment

mickenordin commented Dec 18, 2023

juliusknorr commented Dec 18, 2023

juliusknorr commented Dec 18, 2023

ChristophWurst left a comment

Choose a reason for hiding this comment

kesselb Dec 27, 2023

Choose a reason for hiding this comment

kesselb left a comment

Choose a reason for hiding this comment

smorimoto commented Dec 28, 2023

smorimoto commented Dec 28, 2023 • edited Loading

smorimoto commented Jan 3, 2024

juliusknorr commented Jan 3, 2024

smorimoto commented Jan 3, 2024

juliusknorr commented Jan 3, 2024

juliusknorr commented Dec 4, 2023 •

edited by hweihwang

Loading

smorimoto commented Dec 28, 2023 •

edited

Loading