Unbreak building for FreeBSD #103

mateuszkwiatkowski · 2019-12-17T11:47:20Z

I naively copied darwin files to unbreak building FreeBSD binaries. The other thing is that upstream version of water library doesn't support FreeBSD. There is a fork with added FreeBSD support https://github.com/yggdrasil-network/water and work in progress pull request to upstream: songgao/water#37

After these dirty hacks I'm able to start nebula on FreeBSD hosts but no traffic is passed between them:

$ sudo ./nebula -config config.yml
INFO[0000] Firewall rule added                           firewallRule="map[caName: caSha: direction:outgoing endPort:0 groups:[] host:a
ny ip:<nil> proto:0 startPort:0]"
INFO[0000] Firewall rule added                           firewallRule="map[caName: caSha: direction:incoming endPort:0 groups:[] host:a
ny ip:<nil> proto:1 startPort:0]"
INFO[0000] Firewall rule added                           firewallRule="map[caName: caSha: direction:incoming endPort:443 groups:[laptop
 home] host: ip:<nil> proto:6 startPort:443]"
INFO[0000] Firewall started                              firewallHash=853d3005de969aa0cb1100731e983a740ab4218f89c78189edd389ff5e05ae99
INFO[0000] Main HostMap created                          network=192.168.100.2/24 preferredRanges="[192.168.0.0/24]"
INFO[0000] UDP hole punching enabled
command: ifconfig tap0 192.168.100.2/24 192.168.100.2
command: ifconfig tap0 mtu 1300
INFO[0000] Nebula interface is active                    build=dev+20191217111808 interface=tap0 network=192.168.100.2/24
INFO[0000] Handshake message sent                        handshake="map[stage:1 style:ix_psk0]" initiatorIndex=3879127975 remoteIndex=0
 udpAddr="188.116.33.203:4242" vpnIp=192.168.100.1
INFO[0000] Handshake message received                    durationNs=446865780 handshake="map[stage:2 style:ix_psk0]" initiatorIndex=387
9127975 remoteIndex=3879127975 responderIndex=834573217 udpAddr="188.116.33.203:4242" vpnIp=192.168.100.1

tap0 interface is configured correctly:

tap0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1300
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:ff:96
        inet 192.168.100.2 netmask 0xffffff00 broadcast 192.168.100.2
        groups: tap
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 42831

kwiat@monster-1 ~/nebula/build/freebsd (support-freebsd*) $ netstat -rn4
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.0.2        UGS        igb0
127.0.0.1          link#5             UH          lo0
192.168.0.0/24     link#1             U          igb0
192.168.0.11       link#1             UHS         lo0
192.168.100.0/24   link#6             U          tap0
192.168.100.2      link#6             UHS         lo0

There's no response for who-has requests:

kwiat@monster-1 ~/nebula/build/freebsd (support-freebsd*) $ sudo tcpdump -i tap0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:55:38.490465 ARP, Request who-has 192.168.100.1 tell 192.168.100.2, length 28
12:55:39.532137 ARP, Request who-has 192.168.100.1 tell 192.168.100.2, length 28
12:55:40.559399 ARP, Request who-has 192.168.100.1 tell 192.168.100.2, length 28

Dropping it here with hope that someone would be willing to pick-up and continue this effort.
I was testing on few weeks old CURRENT:

FreeBSD monster-1 13.0-CURRENT FreeBSD 13.0-CURRENT #5 1b501770dd3-c264495(master): Wed Nov 27 01:35:34 CET 2019 root@monster-1:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64

nbrownus · 2019-12-17T17:22:08Z

I can likely spin a VM up shortly and push this along. My initial thoughts are that this is pushing ethernet frames (tap) through to nebula and we only reason about ip (tun) frames currently. I'm not super familiar with the BSDs anymore but I'd imagine the linux files would be a better starting point.

mateuszkwiatkowski · 2019-12-17T19:54:56Z

@nbrownus that's great! I'm happy to help if any assistance is needed. :-)

dch · 2019-12-17T20:49:58Z

@mateuszkwiatkowski neat keep me posted on progress!

mateuszkwiatkowski · 2020-01-24T15:02:28Z

@nbrownus Did you have a chance to take a look into this?

nbrownus · 2020-01-25T01:34:40Z

Sorry, I got pulled in another direction. I'll re-review this PR and merge as a stop gap.

nbrownus · 2020-01-25T01:35:01Z

Mind rebasing on master?

mateuszkwiatkowski · 2020-01-31T11:31:36Z

@nbrownus rebased and force-pushed. :-)

wadey · 2020-02-04T01:14:02Z

tun_freebsd.go

+ c.Device = c.Interface.Name()
+
+ // TODO use syscalls instead of exec.Command
+ fmt.Println("command: ifconfig", c.Device, c.Cidr.String(), c.Cidr.IP.String())


remove this fmt.Println and the one below, or change it to use the logger that the rest of Nebula uses (search the repository for l.Debug(…) etc). Debug level would be correct here I believe.

CLAassistant · 2020-03-05T17:48:58Z

All committers have signed the CLA.

cellinix3m · 2020-03-27T22:04:20Z

Does it look like nebula is going to get FreeBSD support? that would be perfect.
i built it and it is running be i am not getting any traffic from one machine to the other.
but it is running, i needed to make the tap device manual but after that it ran fine.

i dont understand much of the code but i would love to help testing it.

wadey · 2020-04-08T01:48:14Z

I set up a FreeBSD VM to mess around with this. I was able to reproduce the behavior you saw, it looks like the issue is that it is creating a tap device instead of a tun device. This is because the water implementation for bsd is a bit confusing. I will try to work on this some more when I have time now that I have the VM set up.

The yggdrasil-network/water patch to make water work on bsd does some confusing things. One thing is that unless you set `Name`, it will create a tap device. Fix this by requiring the user set `tun.dev` in their config file to a name like `tun0` or `tun1` (the naming required by freebsd). We also need to add the `route` command back now that we are using a tun device.

wadey · 2020-04-08T02:34:15Z

ok, I pushed a fix to this PR which makes it work on freebsd. You have to specify tun.dev in your config file, to a name like tun0 or tun1. Can you retest?

mateuszkwiatkowski · 2020-04-08T09:51:38Z

@wadey Many thanks for pushing it further. You rock!

wadey · 2020-04-08T13:02:18Z

Glad to help. When you have a chance can you test again with the latest? Or if anyone else is watching this issue do you mind testing? Thanks!

cellinix3m · 2020-04-08T22:15:47Z

I set up 3 FreeBSD machines.
2 machines from digitalocean.com and my laptop
after loosening up the nebula firewall settings i get ssh connected.
i have not checked the speed yet but it seems to work fine right now.
This is really cool, and i will try and do some speed tests tomorrow.
Is there anything particular you want me to test?
i do need to raise the privilege with sudo to create the tunnel, but this might be necessary to create the tun interface.

I have not tried nebula on Linux so i don`t know that the differences is.

cellinix3m · 2020-04-09T00:39:30Z

Did some tests now.
just had to have some fun with it.

`anders@X1_bsd:~/devel/nebula/build/freebsd-amd64 % iperf -c 192.168.100.9 && iperf -c 10.100.100.12 && iperf -c 134.209.204.190

Client connecting to 192.168.100.9, TCP port 5001
TCP window size: 33.2 KByte (default)

[ 3] local 192.168.100.2 port 36584 connected with 192.168.100.9 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 3.38 MBytes 2.81 Mbits/sec

Client connecting to 10.100.100.12, TCP port 5001
TCP window size: 32.3 KByte (default)

[ 3] local 10.100.100.3 port 56129 connected with 10.100.100.12 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 3.25 MBytes 2.71 Mbits/sec

Client connecting to 134.209.204.190, TCP port 5001
TCP window size: 32.8 KByte (default)

[ 3] local 192.168.43.27 port 15356 connected with 134.209.204.190 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.3 sec 3.75 MBytes 3.04 Mbits/sec
`
Sins it is Easter i am at the cabin so this is running from my laptop tethered to my phone.

the first one is running over nebula, second is running over wireguard and the last one is running directly to the IP address

i ran ping to but sinse the computer is teathered it is so unstable that i dont realy think you can read much from the values. but added them anyway.
anders@X1_bsd:~/devel/nebula/build/freebsd-amd64 % echo nebula: && ping -c 100 192.168.100.9 | grep min/avg && echo wireguard: && ping -c 100 10.100.100.12|grep min/avg &&echo direct: && ping -c 100 134.209.204.190 | grep min/avg nebula: round-trip min/avg/max/stddev = 39.811/74.857/233.090/33.446 ms wireguard: round-trip min/avg/max/stddev = 45.503/71.400/148.723/16.040 ms direct: round-trip min/avg/max/stddev = 36.777/80.377/502.177/64.442 ms

Ran the same tests from a test machine at work to the same digital ocean server.
anders@sundet_bsd:~/nebula/build/freebsd-amd64 % echo nebula: && iperf -c 192.168.100.9 && echo wireguard: && iperf -c 10.100.100.12 && echo direct: && iperf -c 134.209.204.190
nebula:

Client connecting to 192.168.100.9, TCP port 5001
TCP window size: 33.2 KByte (default)

[ 3] local 192.168.100.3 port 22874 connected with 192.168.100.9 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 15.1 MBytes 12.5 Mbits/sec
wireguard:

Client connecting to 10.100.100.12, TCP port 5001
TCP window size: 32.3 KByte (default)

[ 3] local 10.100.100.10 port 62461 connected with 10.100.100.12 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 10.4 MBytes 8.64 Mbits/sec
direct:

Client connecting to 134.209.204.190, TCP port 5001
TCP window size: 32.8 KByte (default)

[ 3] local 192.168.1.139 port 60819 connected with 134.209.204.190 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 386 MBytes 323 Mbits/sec

What i do notice is the cpu values on the iperf server
nebula running:
Nebula: 5-10%
wireguard:
bash 52%
wireguard about 11%
direct:
iperf: 9%

ping from test machine to digital ocean server is as following.
anders@sundet_bsd:~/nebula/build/freebsd-amd64 % echo nebula: && ping -c 100 192.168.100.9 | grep min/avg && echo wireguard: && ping -c 100 10.100.100.12 | grep min/avg && echo direct: && ping -c 100 134.209.204.190 | grep min/avg
nebula:
round-trip min/avg/max/stddev = 29.854/30.150/30.904/0.152 ms
wireguard:
round-trip min/avg/max/stddev = 30.870/31.155/31.600/0.136 ms
direct:
round-trip min/avg/max/stddev = 29.489/29.733/30.173/0.113 ms

Hope this is use-full information :)

The only thing the library is doing for freebsd is calling OpenFile.

wadey · 2020-04-11T14:21:55Z

Ok I fixed this up a bit more and removed the dependency on water. I want to do one more thing before getting a review to merge this, which is make it auto-select a tun device if you don't set tun.dev.

The second pass of this would be to remove the exec calls and use ioctl(2) and route(4) instead, but I think we can do that in a second PR.

cellinix3m

I cant read the code but it builds and runs for me :)

wadey

This works, let's merge it and we can improve it more in the future.

mateuszkwiatkowski mentioned this pull request Dec 17, 2019

Add FreeBSD support #104

Closed

mateuszkwiatkowski added 2 commits January 31, 2020 12:21

Rebase against upstream/master

ce387c5

Make gofmt happy

3f42be1

mateuszkwiatkowski force-pushed the support-freebsd branch from eb43467 to 3f42be1 Compare January 31, 2020 11:25

wadey reviewed Feb 4, 2020

View reviewed changes

nbrownus added this to the v1.2.0 milestone Apr 6, 2020

wadey modified the milestones: v1.2.0, future Apr 8, 2020

wadey added 5 commits April 7, 2020 22:35

Merge branch 'master' into support-freebsd

bf14a31

add release-freebsd to the Makefile

2c24d99

Build freebsd as part of the release

e1d6386

debug logs instead of fmt.Println

f5661da

use absolute paths

7751e9e

we don't even need water

660a455

The only thing the library is doing for freebsd is calling OpenFile.

wadey added 3 commits April 11, 2020 06:55

we don't even need water

70d6f49

The only thing the library is doing for freebsd is calling OpenFile.

we don't even need water

580d177

The only thing the library is doing for freebsd is calling OpenFile.

support unsafe_routes

585dcfe

cellinix3m approved these changes Apr 30, 2020

View reviewed changes

wadey approved these changes May 27, 2020

View reviewed changes

wadey merged commit cc03ff9 into slackhq:master May 27, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unbreak building for FreeBSD #103

Unbreak building for FreeBSD #103

mateuszkwiatkowski commented Dec 17, 2019 •

edited

Loading

nbrownus commented Dec 17, 2019 •

edited

Loading

mateuszkwiatkowski commented Dec 17, 2019

dch commented Dec 17, 2019

mateuszkwiatkowski commented Jan 24, 2020

nbrownus commented Jan 25, 2020

nbrownus commented Jan 25, 2020

mateuszkwiatkowski commented Jan 31, 2020

wadey Feb 4, 2020

CLAassistant commented Mar 5, 2020

cellinix3m commented Mar 27, 2020

wadey commented Apr 8, 2020

wadey commented Apr 8, 2020 •

edited

Loading

mateuszkwiatkowski commented Apr 8, 2020

wadey commented Apr 8, 2020

cellinix3m commented Apr 8, 2020

cellinix3m commented Apr 9, 2020

wadey commented Apr 11, 2020

cellinix3m left a comment

wadey left a comment

Unbreak building for FreeBSD #103

Unbreak building for FreeBSD #103

Conversation

mateuszkwiatkowski commented Dec 17, 2019 • edited Loading

nbrownus commented Dec 17, 2019 • edited Loading

mateuszkwiatkowski commented Dec 17, 2019

dch commented Dec 17, 2019

mateuszkwiatkowski commented Jan 24, 2020

nbrownus commented Jan 25, 2020

nbrownus commented Jan 25, 2020

mateuszkwiatkowski commented Jan 31, 2020

wadey Feb 4, 2020

Choose a reason for hiding this comment

CLAassistant commented Mar 5, 2020

cellinix3m commented Mar 27, 2020

wadey commented Apr 8, 2020

wadey commented Apr 8, 2020 • edited Loading

mateuszkwiatkowski commented Apr 8, 2020

wadey commented Apr 8, 2020

cellinix3m commented Apr 8, 2020

cellinix3m commented Apr 9, 2020

`anders@X1_bsd:~/devel/nebula/build/freebsd-amd64 % iperf -c 192.168.100.9 && iperf -c 10.100.100.12 && iperf -c 134.209.204.190

Client connecting to 192.168.100.9, TCP port 5001 TCP window size: 33.2 KByte (default)

[ 3] local 192.168.100.2 port 36584 connected with 192.168.100.9 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.1 sec 3.38 MBytes 2.81 Mbits/sec

Client connecting to 10.100.100.12, TCP port 5001 TCP window size: 32.3 KByte (default)

[ 3] local 10.100.100.3 port 56129 connected with 10.100.100.12 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 3.25 MBytes 2.71 Mbits/sec

Client connecting to 134.209.204.190, TCP port 5001 TCP window size: 32.8 KByte (default)

Ran the same tests from a test machine at work to the same digital ocean server. anders@sundet_bsd:~/nebula/build/freebsd-amd64 % echo nebula: && iperf -c 192.168.100.9 && echo wireguard: && iperf -c 10.100.100.12 && echo direct: && iperf -c 134.209.204.190 nebula:

Client connecting to 192.168.100.9, TCP port 5001 TCP window size: 33.2 KByte (default)

[ 3] local 192.168.100.3 port 22874 connected with 192.168.100.9 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.1 sec 15.1 MBytes 12.5 Mbits/sec wireguard:

Client connecting to 10.100.100.12, TCP port 5001 TCP window size: 32.3 KByte (default)

[ 3] local 10.100.100.10 port 62461 connected with 10.100.100.12 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.1 sec 10.4 MBytes 8.64 Mbits/sec direct:

Client connecting to 134.209.204.190, TCP port 5001 TCP window size: 32.8 KByte (default)

wadey commented Apr 11, 2020

cellinix3m left a comment

Choose a reason for hiding this comment

wadey left a comment

Choose a reason for hiding this comment

mateuszkwiatkowski commented Dec 17, 2019 •

edited

Loading

nbrownus commented Dec 17, 2019 •

edited

Loading

wadey commented Apr 8, 2020 •

edited

Loading

Client connecting to 192.168.100.9, TCP port 5001
TCP window size: 33.2 KByte (default)

[ 3] local 192.168.100.2 port 36584 connected with 192.168.100.9 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 3.38 MBytes 2.81 Mbits/sec

Client connecting to 10.100.100.12, TCP port 5001
TCP window size: 32.3 KByte (default)

[ 3] local 10.100.100.3 port 56129 connected with 10.100.100.12 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 3.25 MBytes 2.71 Mbits/sec

Client connecting to 134.209.204.190, TCP port 5001
TCP window size: 32.8 KByte (default)

Ran the same tests from a test machine at work to the same digital ocean server.
anders@sundet_bsd:~/nebula/build/freebsd-amd64 % echo nebula: && iperf -c 192.168.100.9 && echo wireguard: && iperf -c 10.100.100.12 && echo direct: && iperf -c 134.209.204.190
nebula:

Client connecting to 192.168.100.9, TCP port 5001
TCP window size: 33.2 KByte (default)

[ 3] local 192.168.100.3 port 22874 connected with 192.168.100.9 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 15.1 MBytes 12.5 Mbits/sec
wireguard:

Client connecting to 10.100.100.12, TCP port 5001
TCP window size: 32.3 KByte (default)

[ 3] local 10.100.100.10 port 62461 connected with 10.100.100.12 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.1 sec 10.4 MBytes 8.64 Mbits/sec
direct:

Client connecting to 134.209.204.190, TCP port 5001
TCP window size: 32.8 KByte (default)