feat(outbound): Route TLS connections by SNI #3160

zaharidichev · 2024-08-29T10:42:06Z

This is a draft PR that wires together some of the recently introduced changes to the proxy in order to deliver TLS routing functionality. The changes are:

This change adds a TLS routing stack that has the following properties:

it always expects that SNI is present
uses tls routing information provided by the discovery API to perform routing based on SNI
does not concern itself with terminating TLS by simply proxies the encrypted stream

Signed-off-by: Zahari Dichev zaharidichev@gmail.com

linkerd/io/src/sensor.rs

olix0r

From a merge strategy perspective, I think we'll probably want to split out PRs for from the leaves up to the root. That is, we can easily add the tls/route crate in an independent change with its own tests. Then we can add client-policy changes that incorporate those changes, and then we can add the outbound tls stack with tests, and then finally wire it into the outbound protocol router.

We are going to have to wire this into the ingress stack as well, presumably? Or what do we expect to happen in the case where an ingress makes an upstream HTTPS call?

olix0r · 2024-09-09T14:08:22Z

linkerd/tls/route/Cargo.toml

@@ -0,0 +1,14 @@
+[package]
+name = "linkerd-tls-route"


We could probably introduce this crate in a distinct change. It should include module-level tests like the http-route crate.

olix0r · 2024-09-09T14:15:13Z

linkerd/app/outbound/src/tls/detect.rs

+}
+
+#[derive(Clone, Debug, Default)]
+pub struct DetectSniServer<T, N> {


How does this differ from the middleware in linkerd/tls/src/server.rs? Can they be unified? Is there any reason that this middleware can't be defined in the linkerd-tls crate? That should be our default--it's easier to to test/change modules when they are decoupled from the app. That is, if the app isn't compiling, it can still be possible to test and make changes to this stuff.

olix0r · 2024-09-09T14:16:03Z

linkerd/app/outbound/src/tls/detect.rs

+            let (sni, io) = detect.await.map_err(|_| SniDetectionTimeoutError)??;
+            let sni = sni.ok_or(NoSniFoundError)?;
+
+            println!("detected SNI: {:?}", sni);


This should definitely be a debug log and not a println.

linkerd/app/outbound/src/protocol.rs

olix0r · 2024-09-09T14:25:46Z

linkerd/tls/src/server.rs

@@ -192,7 +192,7 @@ where
 }

 /// Peek or buffer the provided stream to determine an SNI value.
-async fn detect_sni<I>(mut io: I) -> io::Result<(Option<ServerName>, DetectIo<I>)>
+pub async fn detect_sni<I>(mut io: I) -> io::Result<(Option<ServerName>, DetectIo<I>)>


As I asked above, is it possible to define the TLS servers within this module so we don't need to make these details public?

olix0r · 2024-09-09T14:29:12Z

linkerd/proxy/client-policy/src/lib.rs

                    p.fill_backends(&mut backends);
                }
+                Protocol::Tls(_) => {
+                    unreachable!("no API support for TLS protocol yet");


Is it not possible to fill the route backends here? Our tls types should include enough information to do that, right?

linkerd/io/src/sensor.rs

linkerd/app/outbound/src/tls/detect.rs

Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

This reverts commit d909be7. Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

This reverts commit 6c07804. Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

* tls: Avoid InsertParam parameter. We don't actually use InsertParam all that much--only in the TLS server (which is obviously why it was included here). This change removes the InsertParam in favor of using a tuple, generally reducing boilerplate. It turns out that the TLS stack already has a map_target to handle turning the tuple-target into a Tls type, so it shouldn't be needed. * tls: Remove ExtractParam from detect_sni Similarly, we don't actually care about extracting a timeout from the target. Using an ExtractParam causes needless boilerplate. This change updates the stack module to simply take a timeout parameter at construction time. * tls: Make the detect_tls module private We now only need to export the NewDetectSni type. The module reexport is not necessary. * tls: Reorganize NewDetectRequiredSni under the server module Because the DetectTls and DetectSni types are so similar -- and implemented in the context of a server inspecting a provided connection (and not a client establishing a TLS connection), this change reorganizes the module: * The DetectSni types are renamed to DetectRequiredSni to better reflect their purpose and difference from the DetectTls type. * The detect_sni module is renamed and moved to server::required_sni. This module is private and the relevant types are reexported from the server module.

cratelyn

i took another read through this work, it's looking very good! i have some nitpicks i noticed, but these are all small details. nice work @zaharidichev!

cratelyn · 2024-10-10T19:44:51Z

linkerd/app/outbound/Cargo.toml

 linkerd-http-route = { path = "../../http/route" }
+linkerd-tls-route = { path = "../../tls/route" }
 linkerd-identity = { path = "../../identity" }


nit: can we sort this?

cratelyn · 2024-10-10T19:51:20Z

linkerd/app/outbound/src/tls/concrete.rs

+
+/// A backend dispatcher explicitly fails all requests.
+#[derive(Debug, thiserror::Error)]
+#[error("{0}")]


nit: we could use

Suggested change

#[error("{0}")]

#[error(transparent)]

if we are delegating directly to the inner string.

cratelyn · 2024-10-10T19:55:31Z

linkerd/app/outbound/src/tls/logical/route.rs

+// === impl Backend ===
+
+impl<T: Clone> Clone for Backend<T> {
+    fn clone(&self) -> Self {
+        Self {
+            route_ref: self.route_ref.clone(),
+            concrete: self.concrete.clone(),
+        }
+    }
+}
+


nit: could we move this below the MatchedRoute<T>, Route<T>, and error structures below, just to match the general convention of // === impl T === blocks coming after struct/type definitions?

cratelyn · 2024-10-10T20:01:01Z

linkerd/tls/src/server/required_sni.rs

+}
+
+impl<N> NewDetectRequiredSni<N> {


Suggested change

}

impl<N> NewDetectRequiredSni<N> {

}

// === impl NewDetectRequiredSni ===

impl<N> NewDetectRequiredSni<N> {

cratelyn · 2024-10-10T20:01:30Z

linkerd/tls/src/server/required_sni.rs

+}
+
+// === impl DetectSni ===
+


Suggested change

}

// === impl DetectSni ===

}

// === impl DetectRequiredSni ===

cratelyn · 2024-10-10T20:10:30Z

linkerd/tls/src/server/required_sni.rs

+#[error("SNI detection timed out")]
+pub struct SniDetectionTimeoutError;


optional idea:

if this was a newtype around https://docs.rs/tokio/latest/tokio/time/error/struct.Elapsed.html, we could be extra-defensive about confirming that we've encountered a timeout. (contd. below... 👇)

cratelyn · 2024-10-10T20:11:41Z

linkerd/tls/src/server/required_sni.rs

+        // Detect the SNI from a ClientHello (or timeout).
+        let detect = time::timeout(self.timeout, detect_sni(io));
+        Box::pin(async move {
+            let (res, io) = detect.await.map_err(|_| SniDetectionTimeoutError)??;


Suggested change

let (res, io) = detect.await.map_err(|_| SniDetectionTimeoutError)??;

let (res, io) = detect.await.map_err(SniDetectionTimeoutError)??;

(contd. from above..) and then this could wrap the timeout error, making sure we don't ever discard some other error we might care about.

cratelyn · 2024-10-10T20:16:39Z

linkerd/app/outbound/src/tls.rs

+// === impl Outbound ===
+
+impl<C> Outbound<C> {
+    /// Builds a stack that proxies tls connections.


Suggested change

/// Builds a stack that proxies tls connections.

/// Builds a stack that proxies TLS connections.

cratelyn · 2024-10-10T20:16:59Z

linkerd/app/outbound/src/tls/concrete.rs

+    Fail { message: Arc<str> },
+}
+
+/// A backend dispatcher explicitly fails all requests.


Suggested change

/// A backend dispatcher explicitly fails all requests.

/// A backend dispatcher that explicitly fails all requests.

that suggestion fixes a typo, i believe, but is this comment affixed to the wrong structure? it seems like this is an error returned by the fail service created below in this statement:

let fail = svc::ArcNewService::new(|message: Arc<str>| { svc::mk(move |_| futures::future::ready(Err(DispatcherFailed(message.clone())))) });

Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

olix0r

LGTM 🎉

Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

zaharidichev force-pushed the zd/tls-stack branch from 0330781 to 6930b90 Compare August 29, 2024 10:43

zaharidichev commented Aug 29, 2024

View reviewed changes

linkerd/io/src/sensor.rs Show resolved Hide resolved

olix0r reviewed Sep 9, 2024

View reviewed changes

linkerd/app/outbound/src/tls/detect.rs Outdated Show resolved Hide resolved

zaharidichev added 6 commits September 11, 2024 10:50

chore: split TLS detection logic

6c07804

Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

remove println

d909be7

Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

Revert "remove println"

cfcfa0a

This reverts commit d909be7. Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

Revert "chore: split TLS detection logic"

d354cc3

This reverts commit 6c07804. Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

add independent DetectSni middleware

22c3a9c

Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

Merge branch 'main' into zd/split-tls-detection

4956e8f

zaharidichev force-pushed the zd/tls-stack branch from 6930b90 to 54b6cc0 Compare September 23, 2024 11:50

zaharidichev changed the base branch from main to zd/split-tls-detection September 23, 2024 11:50

feat(policy): Tls routing stack

dbf6791

Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

zaharidichev force-pushed the zd/tls-stack branch from 54b6cc0 to dbf6791 Compare September 23, 2024 11:54

zaharidichev added 2 commits September 24, 2024 11:39

add SNI routing test

fd10990

Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

lint

ff9b5c1

Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

Base automatically changed from zd/split-tls-detection to main September 25, 2024 15:36

olix0r and others added 2 commits October 10, 2024 18:52

Merge branch 'main' into zd/tls-stack

801051e

zaharidichev marked this pull request as ready for review October 10, 2024 16:05

zaharidichev requested a review from a team as a code owner October 10, 2024 16:05

cratelyn reviewed Oct 10, 2024

View reviewed changes

Merge branch 'main' into zd/tls-stack

513cec2

Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

olix0r changed the title ~~TLS Routing stack~~ feat(outbound): Route TLS connections by SNI Oct 17, 2024

olix0r approved these changes Oct 17, 2024

View reviewed changes

final feedback

660d58f

Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

zaharidichev merged commit 73e96dd into main Oct 18, 2024
15 checks passed

zaharidichev deleted the zd/tls-stack branch October 18, 2024 11:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(outbound): Route TLS connections by SNI #3160

feat(outbound): Route TLS connections by SNI #3160

zaharidichev commented Aug 29, 2024 •

edited

Loading

olix0r left a comment

olix0r Sep 9, 2024

olix0r Sep 9, 2024

olix0r Sep 9, 2024

olix0r Sep 9, 2024

olix0r Sep 9, 2024

cratelyn left a comment

cratelyn Oct 10, 2024

cratelyn Oct 10, 2024

cratelyn Oct 10, 2024

cratelyn Oct 10, 2024

cratelyn Oct 10, 2024

cratelyn Oct 10, 2024

cratelyn Oct 10, 2024

cratelyn Oct 10, 2024

cratelyn Oct 10, 2024

cratelyn Oct 10, 2024

olix0r left a comment

		#[error("SNI detection timed out")]
		pub struct SniDetectionTimeoutError;

	let (res, io) = detect.await.map_err(\|_\| SniDetectionTimeoutError)??;
	let (res, io) = detect.await.map_err(SniDetectionTimeoutError)??;

	/// Builds a stack that proxies tls connections.
	/// Builds a stack that proxies TLS connections.

	/// A backend dispatcher explicitly fails all requests.
	/// A backend dispatcher that explicitly fails all requests.

feat(outbound): Route TLS connections by SNI #3160

feat(outbound): Route TLS connections by SNI #3160

Conversation

zaharidichev commented Aug 29, 2024 • edited Loading

olix0r left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cratelyn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olix0r left a comment

Choose a reason for hiding this comment

zaharidichev commented Aug 29, 2024 •

edited

Loading