Feat: transparent proxy support (linux only) #343

VendettaReborn · 2024-03-27T14:52:17Z

🤔 This is a ...

🔗 Related issue link

💡 Background and solution

on linux, the system provides a native way of handling the global proxy: IP_TRANSPARENT & tproxy.
compared with tun, it has a better performance, since it will not need one more traverse of network stack.
some introductions and documents can be found bellow:

there are 3 main parts of this PR:

support get_orig_dst option for udp socket
utilize the iptables to redirect the socket to tproxy listener
use so_mark and policy routing to avoid the endless loop

i have test this PR on my linux machine, but the compatibility tests with other tools that depend on iptables, like docker, wireguard haven't been done

to be discussed

should we support nftables as well?
should we support IP_RECVTOS option in socket?
can we provide a better experience, to avoid the conflict in iptables' rules?

📝 Changelog

Support Transparent proxy on linux(Ipv4 only)

☑️ Self-Check before Merge

⚠️ Please check all items below before requesting a reviewing. ⚠️

Doc is updated/provided or not needed
Changelog is provided or not needed

ibigbug · 2024-03-28T14:12:25Z

clash_lib/Cargo.toml

@@ -37,6 +37,8 @@ base64 = "0.22"
 uuid = { version = "1.8.0", features = ["v4", "fast-rng", "macro-diagnostics", "serde"] }
 boring = "4.5.0"
 boring-sys = "4.5.0"
+unix-udp-sock = { git = "https://github.com/Watfaq/unix-udp-sock.git", rev = "cd3e4eca43e6f3be82a2703c3d711b7e18fbfd18"}
+cmd_lib = "1.9.3"


not a big fan of executing system commands in a program - at lease should we print out what we are executing so the users know what's happening under the hood?

ibigbug · 2024-03-28T14:12:52Z

clash_lib/src/app/inbound/manager.rs

@@ -88,7 +88,7 @@ impl InboundManager {
        };
        self.network_listeners
            .values()
-            .for_each(|x| match x.listener_type {
+            .for_each(|x: &NetworkInboundListener| match x.listener_type {


is this necessary

ibigbug · 2024-03-28T14:19:48Z

clash_lib/src/app/dispatcher/dispatcher_impl.rs

@@ -181,6 +181,7 @@ impl Dispatcher {

    /// Dispatch a UDP packet to outbound handler
    /// returns the close sender
+    /// will ignore the source and destination in `Session`


is this comment still valid?

ibigbug · 2024-03-28T14:23:04Z

clash_lib/src/proxy/tproxy/inbound.rs

+        let mut buf = vec![0_u8; 1024 * 64];
+        while let Ok(meta) = socket.recv_msg(&mut buf).await {
+            match meta.orig_dst {
+                Some(orig_dst) => {


would tproxy override the dst so we need to patch the socket to get the orig_dst?

i see it now IP_ORIGDSTADDR

ibigbug · 2024-03-28T14:24:34Z

clash_lib/src/proxy/tproxy/iptables.rs

+        if command_exists("iptables") {
+            TProxyStrategy::Iptables
+        } else {
+            TProxyStrategy::None


would it still work if iptables was not installed? should we prompt user?

ibigbug · 2024-03-28T14:28:43Z

clash_lib/src/proxy/tproxy/iptables.rs

+    );
+
+    run_cmd!(ip rule add fwmark $DEFAULT_TPROXY_MARK lookup $POLICY_ROUTING_TABLE_NUM);
+    run_cmd!(ip route add local "0.0.0.0/0" dev lo table $POLICY_ROUTING_TABLE_NUM);


this would be lowest priority, is it necessary/useful to do what openvpn does? https://serverfault.com/a/312977 (i think mainly for having higher priority than the system default route)

and do we need to delete this route in clean up?

ibigbug · 2024-03-28T14:29:58Z

clash_lib/src/proxy/tun/mod.rs

@@ -1,4 +1,3 @@
 pub mod inbound;
-pub use netstack_lwip as netstack;
-mod datagram;


is this intentional?

ibigbug · 2024-03-28T14:30:45Z

clash_lib/src/proxy/vmess/mod.rs

@@ -36,6 +36,7 @@ pub enum VmessTransport {

 pub struct HandlerOptions {
    pub name: String,
+    // TODO: @VendettaReborn, delete this after confirmed


where do you think better to put this?

ibigbug · 2024-03-28T14:32:11Z

clash_lib/src/session.rs

@@ -12,6 +13,13 @@ use tokio::io::{AsyncRead, AsyncReadExt};

 use erased_serde::Serialize as ESerialize;

+// mark of the packet from clash-rs
+static DEFAULT_PACKET_MARK: AtomicU32 = AtomicU32::new(0xff);


isn't this just a fixed u32, why you need an atomic?

ibigbug · 2024-03-28T14:33:18Z

clash_lib/src/proxy/tproxy/transparent_socket.rs

+    ))
+}
+
+// only support v4 now


plans for supporting v6? is it huge amount of work?

and does v4 support mean we can't proxy requests targeting a remote v6 addr or we just can't bind v6 locally

Itsusinn · 2024-03-30T07:03:49Z

clash_lib/src/proxy/mod.rs

I am actually confused here. Shouldn't every connect from clash-rs use the same so_mark?

Itsusinn · 2024-03-30T07:27:00Z

clash_lib/src/session.rs

+pub(crate) fn get_packet_mark() -> u32 {
+    DEFAULT_PACKET_MARK
+}
+


What i thought is:

static GLOBAL_PACKET_MARK: AtomicU32 = AtomicU32::new(0); pub(crate) fn get_packet_mark() -> Option<u32> { let mark = GLOBAL_PACKET_MARK.load(Ordering::Relex) if mark != 0 { return Some(mark) } else { return None } }

if tun auto-route or tproxy auto-route is enabled, tun or tproxy can generate mark itself or read from conf, then set GLOBAL_PACKET_MARK
if tun auto-route or tproxy auto-route is disabled, we can just set this to 0.
Then anywhere we constructing Session, we can(must) use it(get_packet_mark).

Signed-off-by: Yuwei Ba <dev@watfaq.com>

ibigbug · 2024-04-03T14:25:08Z

i find this post is useful https://powerdns.org/tproxydoc/tproxy.md.html

ibigbug · 2024-04-03T14:42:37Z

clash_lib/src/proxy/tproxy/inbound.rs

+        let mut buf = vec![0_u8; 1024 * 64];
+        while let Ok(meta) = socket.recv_msg(&mut buf).await {
+            match meta.orig_dst {
+                Some(orig_dst) => {


i see it now IP_ORIGDSTADDR

ibigbug · 2024-04-03T14:45:03Z

clash_lib/src/proxy/tproxy/transparent_socket.rs

+pub fn new_tcp_listener(addr: SocketAddr) -> io::Result<TcpListener> {
+    let socket = TcpSocket::new_v4()?;
+    set_ip_transparent(socket.as_raw_fd())?;
+    socket.set_reuseaddr(true)?;


why we need to reuseaddr?

ibigbug · 2024-04-08T08:02:57Z

this is the iptables created by the tproxy

-> % sudo iptables -L -t mangle
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
CLASH_TPROXY_PREROUTING  tcp  --  anywhere             anywhere
CLASH_TPROXY_PREROUTING  udp  --  anywhere             anywhere

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
CLASH_TPROXY_OUTPUT  tcp  --  anywhere             anywhere
CLASH_TPROXY_OUTPUT  udp  --  anywhere             anywhere

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination

Chain CLASH_TPROXY_OUTPUT (2 references)
target     prot opt source               destination
RETURN     all  --  anywhere             0.0.0.0/8
RETURN     all  --  anywhere             10.0.0.0/8
RETURN     all  --  anywhere             100.64.0.0/10
RETURN     all  --  anywhere             127.0.0.0/8
RETURN     all  --  anywhere             link-local/16
RETURN     all  --  anywhere             172.16.0.0/12
RETURN     all  --  anywhere             192.0.0.0/24
RETURN     all  --  anywhere             192.0.2.0/24
RETURN     all  --  anywhere             192.88.99.0/24
RETURN     all  --  anywhere             192.168.0.0/16
RETURN     all  --  anywhere             198.18.0.0/15
RETURN     all  --  anywhere             198.51.100.0/24
RETURN     all  --  anywhere             203.0.113.0/24
RETURN     all  --  anywhere             base-address.mcast.net/4
RETURN     all  --  anywhere             240.0.0.0/4
RETURN     all  --  anywhere             255.255.255.255
RETURN     all  --  anywhere             anywhere             mark match 0xff
RETURN     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL
RETURN     all  --  anywhere             anywhere             ADDRTYPE match dst-type BROADCAST
MARK       tcp  --  anywhere             anywhere             MARK set 0x1
MARK       udp  --  anywhere             anywhere             MARK set 0x1

Chain CLASH_TPROXY_OUTPUT_DIVERT (0 references)
target     prot opt source               destination
MARK       all  --  anywhere             anywhere             MARK set 0x1
ACCEPT     all  --  anywhere             anywhere

Chain CLASH_TPROXY_PREROUTING (2 references)
target     prot opt source               destination
RETURN     all  --  anywhere             0.0.0.0/8
RETURN     all  --  anywhere             10.0.0.0/8
RETURN     all  --  anywhere             100.64.0.0/10
RETURN     all  --  anywhere             127.0.0.0/8
RETURN     all  --  anywhere             link-local/16
RETURN     all  --  anywhere             172.16.0.0/12
RETURN     all  --  anywhere             192.0.0.0/24
RETURN     all  --  anywhere             192.0.2.0/24
RETURN     all  --  anywhere             192.88.99.0/24
RETURN     all  --  anywhere             192.168.0.0/16
RETURN     all  --  anywhere             198.18.0.0/15
RETURN     all  --  anywhere             198.51.100.0/24
RETURN     all  --  anywhere             203.0.113.0/24
RETURN     all  --  anywhere             base-address.mcast.net/4
RETURN     all  --  anywhere             240.0.0.0/4
RETURN     all  --  anywhere             255.255.255.255
LOG        all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL LOG level warning prefix ""LOCAL IN""
RETURN     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL
RETURN     all  --  anywhere             anywhere             mark match 0xff
TPROXY     tcp  --  anywhere             anywhere             TPROXY redirect 0.0.0.0:7893 mark 0x1/0x1
TPROXY     udp  --  anywhere             anywhere             TPROXY redirect 0.0.0.0:7893 mark 0x1/0x1

ibigbug · 2024-04-08T08:06:08Z

clash_lib/src/proxy/tproxy/iptables.rs

+    ) {
+        match self {
+            TProxyStrategy::Iptables => {
+                clean_iptables_tproxy(output_chain_name, prerouting_chain_name);


do you need to cleanup everytime before setting up? it also seems a bit counter single responsitiblity to have clean up and set up in setup

ibigbug · 2024-04-08T08:06:28Z

clash_lib/src/proxy/tproxy/iptables.rs

+    run_cmd!(iptables "-t" mangle "-A" $prerouting_chain_name "-m" addrtype "--dst-type" LOCAL "-j" LOG "--log-prefix" "\"LOCAL IN\"");
+    run_cmd!(iptables "-t" mangle "-A" $prerouting_chain_name "-m" addrtype "--dst-type" LOCAL "-j" RETURN);
+    run_cmd!(iptables "-t" mangle "-A" $prerouting_chain_name "-m" mark "--mark" $skip_mark "-j" RETURN);
+    run_cmd!(iptables "-t" mangle "-A" $prerouting_chain_name "-p" tcp "-m" socket "-j" divert_chain_name);


Suggested change

run_cmd!(iptables "-t" mangle "-A" $prerouting_chain_name "-p" tcp "-m" socket "-j" divert_chain_name);

run_cmd!(iptables "-t" mangle "-A" $prerouting_chain_name "-p" tcp "-m" socket "-j" $divert_chain_name);

ibigbug · 2024-04-08T08:17:50Z

clash_lib/src/proxy/tproxy/iptables.rs

+                );
+            }
+            TProxyStrategy::None => {
+                error!("No tproxy command found");


Suggested change

error!("No tproxy command found");

error!("No iptables command found");

ibigbug · 2024-04-08T08:18:16Z

clash_lib/src/proxy/tproxy/iptables.rs

+    for addr in RESERVED_ADDRS {
+        run_cmd!(iptables "-t" mangle "-A" $prerouting_chain_name "-d" $addr "-j" RETURN);
+    }
+    run_cmd!(iptables "-t" mangle "-A" $prerouting_chain_name "-m" addrtype "--dst-type" LOCAL "-j" LOG "--log-prefix" "\"LOCAL IN\"");


is this log useful in any case?

ibigbug · 2024-04-08T08:18:58Z

clash_lib/src/proxy/tproxy/iptables.rs

+    run_cmd!(iptables "-t" mangle "-A" $prerouting_chain_name "-m" addrtype "--dst-type" LOCAL "-j" LOG "--log-prefix" "\"LOCAL IN\"");
+    run_cmd!(iptables "-t" mangle "-A" $prerouting_chain_name "-m" addrtype "--dst-type" LOCAL "-j" RETURN);
+    run_cmd!(iptables "-t" mangle "-A" $prerouting_chain_name "-m" mark "--mark" $skip_mark "-j" RETURN);
+    run_cmd!(iptables "-t" mangle "-A" $prerouting_chain_name "-p" tcp "-m" socket "-j" divert_chain_name);


why you need to redirect it to a "divert" chain rather than just setting mark here inplace?

ibigbug

have tried this out on my box and it's working as expected!

overall looks good to me.

would be great please see the comments and clean up the code a bit esp the #[allow(warning)] dead_code related things.

ibigbug · 2024-04-08T08:26:30Z

clash_lib/src/config/def.rs

@@ -374,6 +374,7 @@ pub struct Config {
    ///   device-id: "dev://utun1989"
    /// ```
    pub tun: Option<HashMap<String, Value>>,
+    pub iptables: Iptables,


not seeing this used anywhere

ibigbug · 2024-04-08T08:34:42Z

to be discussed

1. should we support nftables as well? - let's only do iptables for now, it would be rather straighforward to impl using other commands when anyone requests.
2. should we support IP_RECVTOS option in socket? - is this for us to do advanced routing in the app or there's any other benefits to let the kernel do optimization? not very sure.
3. can we provide a better experience, to avoid the conflict in iptables' rules? - i think managing users' firewall rules can be tricky, but you currently have a name space for each chain which i think should be fine. some edge cases that i can think of is you are doing append here rather than insert which might be a lower match, but still I consider this to be fine until we hit a real issue

VendettaReborn · 2024-04-09T11:46:19Z

have tried this out on my box and it's working as expected!

overall looks good to me.

would be great please see the comments and clean up the code a bit esp the #[allow(warning)] dead_code related things.

i will try my best to resolve some trivial issues and warnings here.

ibigbug · 2024-09-11T13:30:08Z

@VendettaReborn What's still missing for this to merge?

ibigbug · 2024-09-30T15:12:33Z

I created a new on on top of this without the iptables integration since the code base has changed a lot and it takes some effort to rebase #615

VendettaReborn added 6 commits March 24, 2024 23:34

feat:tproxy draft

c232281

fix wrong rules in iptables

9833874

fix logs

c207d67

fix iptables rules

3eb4f40

fix iptables rules, add reserved addrs

00e5047

update unix-udp-sock dep

93838be

VendettaReborn added enhancement New feature or request core labels Mar 27, 2024

VendettaReborn self-assigned this Mar 27, 2024

VendettaReborn added 3 commits March 28, 2024 10:40

update reserved addrs for tproxy

9a2e8e9

override the iface&so_mark in each proxy's config

e065316

cargo fmt

1955376

ibigbug reviewed Mar 28, 2024

View reviewed changes

VendettaReborn added 2 commits March 29, 2024 22:36

run cargo clippy

c0d682e

fix error commented in PR

d58d874

Itsusinn reviewed Mar 30, 2024

View reviewed changes

clash_lib/src/proxy/mod.rs Outdated

Copy link

Member

Itsusinn Mar 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am actually confused here. Shouldn't every connect from clash-rs use the same so_mark?

Itsusinn reviewed Mar 30, 2024

View reviewed changes

Merge branch 'master' into feat/tproxy

651a9df

Signed-off-by: Yuwei Ba <dev@watfaq.com>

ibigbug reviewed Apr 3, 2024

View reviewed changes

ibigbug reviewed Apr 8, 2024

View reviewed changes

ibigbug requested changes Apr 8, 2024

View reviewed changes

ibigbug mentioned this pull request Apr 9, 2024

Refactor/mark #362

Closed

ibigbug mentioned this pull request Sep 30, 2024

feat(tproxy): support tproxy on Linux #615

Merged

12 tasks

ibigbug closed this Oct 1, 2024

ibigbug deleted the feat/tproxy branch October 1, 2024 07:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat: transparent proxy support (linux only) #343

Feat: transparent proxy support (linux only) #343

VendettaReborn commented Mar 27, 2024

ibigbug Mar 28, 2024

ibigbug Mar 28, 2024

ibigbug Mar 28, 2024

ibigbug Mar 28, 2024

ibigbug Apr 3, 2024

ibigbug Mar 28, 2024

ibigbug Mar 28, 2024

ibigbug Mar 28, 2024

ibigbug Mar 28, 2024

ibigbug Mar 28, 2024

ibigbug Mar 28, 2024

Itsusinn Mar 30, 2024

Itsusinn Mar 30, 2024 •

edited

Loading

ibigbug commented Apr 3, 2024

ibigbug Apr 3, 2024

ibigbug Apr 3, 2024

ibigbug commented Apr 8, 2024

ibigbug Apr 8, 2024

ibigbug Apr 8, 2024

ibigbug Apr 8, 2024

ibigbug Apr 8, 2024

ibigbug Apr 8, 2024

ibigbug left a comment

ibigbug Apr 8, 2024

ibigbug commented Apr 8, 2024

VendettaReborn commented Apr 9, 2024

ibigbug commented Sep 11, 2024

ibigbug commented Sep 30, 2024

	run_cmd!(iptables "-t" mangle "-A" $prerouting_chain_name "-p" tcp "-m" socket "-j" divert_chain_name);
	run_cmd!(iptables "-t" mangle "-A" $prerouting_chain_name "-p" tcp "-m" socket "-j" $divert_chain_name);

	error!("No tproxy command found");
	error!("No iptables command found");

Feat: transparent proxy support (linux only) #343

Feat: transparent proxy support (linux only) #343

Conversation

VendettaReborn commented Mar 27, 2024

🤔 This is a ...

🔗 Related issue link

💡 Background and solution

to be discussed

📝 Changelog

☑️ Self-Check before Merge

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Itsusinn Mar 30, 2024 • edited Loading

Choose a reason for hiding this comment

ibigbug commented Apr 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ibigbug commented Apr 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ibigbug left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ibigbug commented Apr 8, 2024

VendettaReborn commented Apr 9, 2024

ibigbug commented Sep 11, 2024

ibigbug commented Sep 30, 2024

Itsusinn Mar 30, 2024 •

edited

Loading