Client side routing implementation #205

madchicken · 2024-12-16T10:51:24Z

This PR is a draft implementation of the client-side routing for new4j clusters.
The PR implements the ROUTE message for the unstable version of the protocol (v2).

Testing: I am currently testing the driver on a real cluster but I have no guarantee of the performance and stability of the routing algorithm
~~- [ ] Bookmarks are not supported at the moment (the array is always sent empty)~~
The maximum supported version of the protocol is now 4.4 but we miss some of the features the protocol should expose from this version (the imp_user should be added to the transaction begin message)
Only the round-robin algorithm is currently implemented as a routing strategy. Maybe we want to implement a LeastUsed algo and make it configurable. This is considered acceptable for the moment
There is no handling of the initial routing context: it is always empty. The initial routing context now contains the URL used to connect to the cluster as other drivers do. We need to add support to parameters passed in the URL as query string
The retry mechanism should probably be reviewed with routing: it does not make sense to retry an operation on a failing connection, we should select another server from the cluster instead. - Retry is still in place but we consider as Permanent an error thrown by the pool when getting the connection

Additional Notes:

The lock mechanism for refreshing the routing table is sub-optimal (to use an euphemism). We should try to be smarter and use a different method to check for a new routing table.
~~- Extra fields in the route message are not implemented yet (part of v4.4)~~ Protocol version is now 4.4
I had to modify the way we serialize some fields: in a structure, the skip_field method is now rendering a null value
IMPORTANT: this PR includes a workaround fix to the v2 protocol for date fields deserialization that is crashing the driver. We should find a better solution to this.

knutwalker · 2024-12-18T17:07:57Z

Hey, thanks for this PR, this helps a lot!

I went over the changes and don't have anything to add, but I also did look in detail at the actual routing protocol and algorithms.

I'm ok with merging something before dealing with all limitations. In particular, I think it's fine to not support the routing context, bookmarks, only have partial 4.3 support, only having one routing strategy at the beginning. We don't need to have everything figured out upfront and can do this in steps/multiple PRs.
I would also be ok with only adding it to the new protocol implementation.

I've been thinking about the testing a bit (or a lot) lately; I don't like the amount of integration tests compared to what the unit tests do. Especially when testing bolt protocol details of a non-happy path, it can be either impossible or very difficult to setup.
The current setup is fine to actually run something against a one-and-done database, but it doesn't scale as well as I would like it to be.

I have half an idea to rewrite the core parts with the connection IO and pooling being abstracted out. I would like to be able to do more simulation testing where exact IO operations can be simulated in the tests without having to have an actual db to setup. I'd also like to move async out from the core to some outer layer. Maybe this goes in the sans-io direction, maybe it'll look a bit different.

madchicken · 2024-12-19T08:53:34Z

Thanks for the reply @knutwalker, I will remove the implementation of the client-side routing for the stable protocol and leave it only when the unstable-bolt-protocol-impl-v2 feature is active. Now that I have the workaround for the dates deserialization bug I can do some more testing. One thing I need to understand is this: when selecting the server in the routing table, should I use the bolt protocol or stick with neo4j for the final connection? I think this is very important for the server, isn't it?

knutwalker · 2024-12-19T15:28:47Z

I will remove the implementation of the client-side routing for the stable protocol and leave it only when the unstable-bolt-protocol-impl-v2 feature is active.

great!

One thing I need to understand is this: when selecting the server in the routing table, should I use the bolt protocol or stick with neo4j for the final connection? I think this is very important for the server, isn't it?

There shouldn't be a need for any scheme. The first connection uses the scheme to determine the initial routing and the encryption to use. When connecting to a server from the routing table, you use the same encryption settings (and, when supported, routing context) and directly connect to that socket address, without going through the bolt/neo4j scheme parsing again. That is what the product drivers do, at least. That looks like this requires some changes to the pool creation, essentially using the same ConnectionInfo, but with only the host/port replaced.

madchicken · 2024-12-19T15:41:47Z

There shouldn't be a need for any scheme. The first connection uses the scheme to determine the initial routing and the encryption to use. When connecting to a server from the routing table, you use the same encryption settings (and, when supported, routing context) and directly connect to that socket address, without going through the bolt/neo4j scheme parsing again. That is what the product drivers do, at least. That looks like this requires some changes to the pool creation, essentially using the same ConnectionInfo, but with only the host/port replaced.

Yes, sorry...I realized that too.

madchicken · 2024-12-19T18:36:27Z

@knutwalker please take a look at my last commit where I had to modify the main Graph interface. I had essentially to expose the Operation::Read/Write parameter only for the new protocol implementation. I hope it makes sense.
I also added a note in the description of the PR about the retry mechanism.

madchicken · 2025-01-05T17:23:02Z

@knutwalker I have finished my work on this first implementation of the client-side routing. I tested it in a simple local cluster and I am now running it on a k8s cluster and it seems to work properly. I still don't have any data about the performance impact regarding the locking mechanism implemented on the routing table the driver holds after the initialization.

knutwalker · 2025-01-07T09:10:17Z

4.4 IT failure could be related to #209

knutwalker

This is very good, thanks for adding all this. I do have few comments I'd like to be resolved before merging in, but it isn't anything bigger (I hope) and some are more ideas and thoughts than suggestions.

lib/src/bolt/request/route.rs

lib/src/config.rs

lib/src/routing/mod.rs

lib/src/routing/routed_connection_manager.rs

lib/src/txn.rs

lib/tests/use_default_db.rs

- Use round-robin balancing - Store routing table in a Dashmap to allow fast concurrency - Perform the update of the routing table using a simple mutex on the TTL of the table itself - read/write mode is propagated into the RUN command - Bolt scheme in pools to allow direct connections with no routing to provided servers

knutwalker · 2025-01-09T17:33:23Z

You said you are already planning a follow up for the update in the background, so we can also leave it as is until then.

…

On Thu, Jan 9, 2025, at 5:23 PM, Pierpaolo Follia wrote: ***@***.**** commented on this pull request. In lib/src/routing/connection_registry.rs <#205 (comment)>: > + registry.retain(|k, _| servers.contains(k)); + let _ = self + .ttl + .fetch_update(Ordering::Relaxed, Ordering::Relaxed, |_ttl| { + Some(routing_table.ttl) + }) + .unwrap(); + debug!( + "Registry updated. New size is {} with TTL {}s", + registry.len(), + routing_table.ttl + ); + *guard = now; + } + } else { + debug!("Routing table is not expired"); Blocking the connection is going to slow down all the connections that are coming during the fetch and update process. I used a try_lock to reduce this slow down even tho I understand it's a minimal improvement. The solution that uses a separated thread should actually solve this issue. I can change this with a lock and see how much it impacts on performances in a real environment. — Reply to this email directly, view it on GitHub <#205 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAE2SX7QZR6BSNNQQCYGZHT2J2PAHAVCNFSM6AAAAABTV3KSECVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDKNBQGM4DENRQGI>. You are receiving this because you were mentioned.Message ID: ***@***.***>

- Lock the creation_time

knutwalker · 2025-01-09T17:34:55Z

Direct sounds good, I like it!

…

On Thu, Jan 9, 2025, at 5:13 PM, Pierpaolo Follia wrote: ***@***.**** commented on this pull request. In lib/src/graph.rs <#205 (comment)>: > }; +use backoff::{Error, ExponentialBackoff}; +use std::time::Duration; + +#[derive(Clone)] +enum ConnectionPoolManager { + #[cfg(feature = "unstable-bolt-protocol-impl-v2")] + Routed(RoutedConnectionManager), + Normal(ConnectionPool), What about `Direct`? — Reply to this email directly, view it on GitHub <#205 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAE2SX7GJPZ5IG2QAJSLUSD2J2NY5AVCNFSM6AAAAABTV3KSECVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDKNBQGM2TKNJQHA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

knutwalker

This is great, thanks!

knutwalker · 2025-01-10T11:34:57Z

The 4.4 test failures are persistent enough that they might actually be related to this change 🤔

knutwalker · 2025-01-10T17:30:07Z

lib/src/version.rs

 }

 impl Version {
    pub fn add_supported_versions(bytes: &mut BytesMut) {
        bytes.reserve(16);
+        bytes.put_u32(0x0404); // V4_4
+        bytes.put_u32(0x0304); // V4_3
        bytes.put_u32(0x0104); // V4_1
        bytes.put_u32(0x0004); // V4
        bytes.put_u32(0);


you need to remove the two put_u32(0) calls. The initial message only expected 4 versions. I guess on 5.x, the server drains every additional bytes until the next message, on 4.4 those 0 values are part of the next message, effectively sending a few empty chunks.

madchicken force-pushed the client-side-routing branch from 2184948 to a8904a9 Compare December 19, 2024 11:47

madchicken force-pushed the client-side-routing branch from 5eacd78 to abf4dd8 Compare December 19, 2024 18:34

madchicken force-pushed the client-side-routing branch from abf4dd8 to 98bc0ce Compare December 19, 2024 18:36

madchicken force-pushed the client-side-routing branch from a0c09be to b7b9889 Compare January 5, 2025 17:08

madchicken marked this pull request as ready for review January 5, 2025 17:23

madchicken force-pushed the client-side-routing branch from 36316ae to 2651092 Compare January 6, 2025 16:31

madchicken force-pushed the client-side-routing branch from 8c7afc4 to 9c449c6 Compare January 8, 2025 17:05

knutwalker requested changes Jan 8, 2025

View reviewed changes

knutwalker mentioned this pull request Jan 9, 2025

when I enable the feature [unstable-v1], I am unable to parse properties with certain struct values. #209

Closed

madchicken force-pushed the client-side-routing branch from 9c449c6 to cae0599 Compare January 9, 2025 09:49

Address most of the comments

a84f034

madchicken requested a review from knutwalker January 9, 2025 11:18

madchicken added 2 commits January 9, 2025 17:09

Remove Default implementation for Database structure

4d8e465

Remove debug message

61a6c36

- Use Instant for creation time in the connection registry

5873f35

- Lock the creation_time

madchicken added 4 commits January 9, 2025 18:35

Remove call to ok()

a0c5e45

Use compare for versions >= 4.4

e5ab887

Renaming Normal -> Direct

2475eeb

Remove run_read since it is essentially useless

bc9b2d4

knutwalker approved these changes Jan 10, 2025

View reviewed changes

Formatting

fe80337

knutwalker reviewed Jan 10, 2025

View reviewed changes

Remove zeroes in supported versions

1f4db49

knutwalker merged commit a202b3f into neo4j-labs:main Jan 10, 2025
11 checks passed

madchicken deleted the client-side-routing branch January 11, 2025 17:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Client side routing implementation #205

Client side routing implementation #205

madchicken commented Dec 16, 2024 •

edited

Loading

knutwalker commented Dec 18, 2024

madchicken commented Dec 19, 2024

knutwalker commented Dec 19, 2024

madchicken commented Dec 19, 2024

madchicken commented Dec 19, 2024

madchicken commented Jan 5, 2025

knutwalker commented Jan 7, 2025

knutwalker left a comment

knutwalker commented Jan 9, 2025 via email

knutwalker commented Jan 9, 2025 via email

knutwalker left a comment

knutwalker commented Jan 10, 2025

knutwalker Jan 10, 2025

Client side routing implementation #205

Client side routing implementation #205

Conversation

madchicken commented Dec 16, 2024 • edited Loading

knutwalker commented Dec 18, 2024

madchicken commented Dec 19, 2024

knutwalker commented Dec 19, 2024

madchicken commented Dec 19, 2024

madchicken commented Dec 19, 2024

madchicken commented Jan 5, 2025

knutwalker commented Jan 7, 2025

knutwalker left a comment

Choose a reason for hiding this comment

knutwalker commented Jan 9, 2025 via email

knutwalker commented Jan 9, 2025 via email

knutwalker left a comment

Choose a reason for hiding this comment

knutwalker commented Jan 10, 2025

knutwalker Jan 10, 2025

Choose a reason for hiding this comment

madchicken commented Dec 16, 2024 •

edited

Loading