Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

headscale doesn't stop gracefully #2

Open
dustymabe opened this issue Jul 6, 2023 · 5 comments
Open

headscale doesn't stop gracefully #2

dustymabe opened this issue Jul 6, 2023 · 5 comments
Assignees
@codyro

Comments

@dustymabe
Copy link
Contributor

If I systemctl stop headscale it doesn't appear to stop gracefully.

This may be because I'm running it in a container. Is anyone else able to reproduce?

This is with headscale-0.22.3-1.aarch64 from the copr repo.

[root@c5b9776eab18 /]# systemctl stop headscale
Jul 06 17:23:43 c5b9776eab18 headscale[26]: 2023-07-06T17:23:43-04:00 INF Client is ready to access the tailnet handler=PollNetMap machine=hattopvm noise=true
Jul 06 17:23:43 c5b9776eab18 headscale[26]: 2023-07-06T17:23:43-04:00 INF Sending initial map handler=PollNetMap machine=hattopvm noise=true
Jul 06 17:23:43 c5b9776eab18 headscale[26]: 2023-07-06T17:23:43-04:00 INF Notifying peers handler=PollNetMap machine=hattopvm noise=true
Jul 06 17:23:46 c5b9776eab18 headscale[26]: 2023-07-06T17:23:46-04:00 INF The client has closed the connection handler=PollNetMapStream machine=hattopvm
Jul 06 17:23:46 c5b9776eab18 headscale[26]: 2023-07-06T17:23:46-04:00 INF Client is ready to access the tailnet handler=PollNetMap machine=hattopvm noise=true
Jul 06 17:23:46 c5b9776eab18 headscale[26]: 2023-07-06T17:23:46-04:00 INF Sending initial map handler=PollNetMap machine=hattopvm noise=true
Jul 06 17:23:46 c5b9776eab18 headscale[26]: 2023-07-06T17:23:46-04:00 INF Notifying peers handler=PollNetMap machine=hattopvm noise=true
Jul 06 17:27:38 c5b9776eab18 headscale[26]: 2023-07-06T17:27:38-04:00 INF Received signal to stop, shutting down gracefully signal=terminated
Jul 06 17:27:38 c5b9776eab18 headscale[26]: 2023-07-06T17:27:38-04:00 INF The long-poll handler is shutting down handler=PollNetMapStream machine=hattopvm noise=true
Jul 06 17:27:38 c5b9776eab18 headscale[26]: 2023-07-06T17:27:38-04:00 INF The long-poll handler is shutting down handler=PollNetMapStream machine=vanilla-f37 noise=true
Jul 06 17:27:38 c5b9776eab18 systemd[1]: Stopping headscale.service - headscale coordination server for Tailscale...
Jul 06 17:27:38 c5b9776eab18 headscale[26]: 2023-07-06T17:27:38-04:00 INF Headscale stopped
Jul 06 17:27:42 c5b9776eab18 headscale[26]: 2023-07-06T17:27:42-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:27:42 c5b9776eab18 headscale[26]: 2023-07-06T17:27:42-04:00 ERR error getting routes error="sql: database is closed"
Jul 06 17:27:42 c5b9776eab18 headscale[26]: 2023-07-06T17:27:42-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:27:43 c5b9776eab18 headscale[26]: 2023-07-06T17:27:43-04:00 ERR Failed to fetch machine from the database with node key: 
Jul 06 17:27:45 c5b9776eab18 headscale[26]: 2023-07-06T17:27:45-04:00 ERR Failed to fetch machine from the database with node key: 
Jul 06 17:27:47 c5b9776eab18 headscale[26]: 2023-07-06T17:27:47-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:27:47 c5b9776eab18 headscale[26]: 2023-07-06T17:27:47-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:27:47 c5b9776eab18 headscale[26]: 2023-07-06T17:27:47-04:00 ERR error getting routes error="sql: database is closed"
Jul 06 17:27:51 c5b9776eab18 headscale[26]: 2023-07-06T17:27:51-04:00 ERR Failed to fetch machine from the database with node key: 
Jul 06 17:27:52 c5b9776eab18 headscale[26]: 2023-07-06T17:27:52-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:27:52 c5b9776eab18 headscale[26]: 2023-07-06T17:27:52-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:27:52 c5b9776eab18 headscale[26]: 2023-07-06T17:27:52-04:00 ERR error getting routes error="sql: database is closed"
Jul 06 17:27:56 c5b9776eab18 headscale[26]: 2023-07-06T17:27:56-04:00 ERR Failed to fetch machine from the database with node key: 
Jul 06 17:27:57 c5b9776eab18 headscale[26]: 2023-07-06T17:27:57-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:27:57 c5b9776eab18 headscale[26]: 2023-07-06T17:27:57-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:27:57 c5b9776eab18 headscale[26]: 2023-07-06T17:27:57-04:00 ERR error getting routes error="sql: database is closed"
Jul 06 17:28:02 c5b9776eab18 headscale[26]: 2023-07-06T17:28:02-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:28:02 c5b9776eab18 headscale[26]: 2023-07-06T17:28:02-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:28:02 c5b9776eab18 headscale[26]: 2023-07-06T17:28:02-04:00 ERR error getting routes error="sql: database is closed"
Jul 06 17:28:03 c5b9776eab18 headscale[26]: 2023-07-06T17:28:03-04:00 ERR Failed to fetch machine from the database with node key: 
Jul 06 17:28:07 c5b9776eab18 headscale[26]: 2023-07-06T17:28:07-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:28:07 c5b9776eab18 headscale[26]: 2023-07-06T17:28:07-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:28:07 c5b9776eab18 headscale[26]: 2023-07-06T17:28:07-04:00 ERR error getting routes error="sql: database is closed"
Jul 06 17:28:07 c5b9776eab18 headscale[26]: 2023-07-06T17:28:07-04:00 ERR Failed to fetch machine from the database with node key: 
Jul 06 17:28:12 c5b9776eab18 headscale[26]: 2023-07-06T17:28:12-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:28:12 c5b9776eab18 headscale[26]: 2023-07-06T17:28:12-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:28:12 c5b9776eab18 headscale[26]: 2023-07-06T17:28:12-04:00 ERR error getting routes error="sql: database is closed"
Jul 06 17:28:16 c5b9776eab18 headscale[26]: 2023-07-06T17:28:16-04:00 ERR Failed to fetch machine from the database with node key: 
Jul 06 17:28:17 c5b9776eab18 headscale[26]: 2023-07-06T17:28:17-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:28:17 c5b9776eab18 headscale[26]: 2023-07-06T17:28:17-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:28:17 c5b9776eab18 headscale[26]: 2023-07-06T17:28:17-04:00 ERR error getting routes error="sql: database is closed"
Jul 06 17:28:19 c5b9776eab18 headscale[26]: 2023-07-06T17:28:19-04:00 ERR Failed to fetch machine from the database with node key: 
Jul 06 17:28:22 c5b9776eab18 headscale[26]: 2023-07-06T17:28:22-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:28:22 c5b9776eab18 headscale[26]: 2023-07-06T17:28:22-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:28:22 c5b9776eab18 headscale[26]: 2023-07-06T17:28:22-04:00 ERR error getting routes error="sql: database is closed"
Jul 06 17:28:23 c5b9776eab18 headscale[26]: 2023-07-06T17:28:23-04:00 ERR Failed to fetch machine from the database with node key: 
Jul 06 17:28:24 c5b9776eab18 systemd[1]: headscale.service: State 'stop-sigterm' timed out. Aborting.
Jul 06 17:28:24 c5b9776eab18 systemd[1]: headscale.service: Killing process 26 (headscale) with signal SIGABRT.
Jul 06 17:28:24 c5b9776eab18 headscale[26]: SIGABRT: abort
Jul 06 17:28:24 c5b9776eab18 headscale[26]: PC=0xaaaae1d80110 m=2 sigcode=0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 0 [idle]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.futex()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/sys_linux_arm64.s:663 +0x20 fp=0xffff81659710 sp=0xffff81659710 pc=0xaaaae1d80110
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.futexsleep(0x2ae5452697b88?, 0xb83c3?, 0xffff81659798?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/os_linux.go:75 +0x90 fp=0xffff81659760 sp=0xffff81659710 pc=0xaaaae1d44c30
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.notetsleep_internal(0xaaaae368e3c0, 0x9635ee87)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/lock_futex.go:202 +0xcc fp=0xffff816597a0 sp=0xffff81659760 pc=0xaaaae1d1b89c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.notetsleep(0x2ae5452697797?, 0xffff816597f0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/lock_futex.go:225 +0x34 fp=0xffff816597c0 sp=0xffff816597a0 pc=0xaaaae1d1b9b4
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.sysmon()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:5333 +0x220 fp=0xffff81659830 sp=0xffff816597c0 pc=0xaaaae1d56cb0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.mstart1()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:1498 +0x88 fp=0xffff81659860 sp=0xffff81659830 pc=0xaaaae1d4da28
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.mstart0()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:1455 +0x70 fp=0xffff81659890 sp=0xffff81659860 pc=0xaaaae1d4d980
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.mstart()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:128 +0x10 fp=0xffff816598a0 sp=0xffff81659890 pc=0xaaaae1d7c5f0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 1 [semacquire, 45 minutes]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0xaaaae3698ca0?, 0xaaaae1d1d16c?, 0x60?, 0x0?, 0x18?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000859640 sp=0x4000859620 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goparkunlock(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:387
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.semacquire1(0x4000706190, 0x80?, 0x1, 0x0, 0xd0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/sema.go:160 +0x21c fp=0x40008596a0 sp=0x4000859640 pc=0xaaaae1d5cf6c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: sync.runtime_Semacquire(0x4000ad1718?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/sema.go:62 +0x2c fp=0x40008596e0 sp=0x40008596a0 pc=0xaaaae1d7a72c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: sync.(*WaitGroup).Wait(0x4000706188)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/sync/waitgroup.go:116 +0x78 fp=0x4000859700 sp=0x40008596e0 pc=0xaaaae1d8ad68
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/sync/errgroup.(*Group).Wait(0x4000706180)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/sync/errgroup/errgroup.go:53 +0x2c fp=0x4000859720 sp=0x4000859700 pc=0xaaaae2332a9c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).Serve(0x40000cacc0)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:833 +0x1410 fp=0x4000859ac0 sp=0x4000859720 pc=0xaaaae27fff20
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/cmd/headscale/cli.glob..func26(0xaaaae35989a0?, {0xaaaae28e1118?, 0x0?, 0x0?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/cmd/headscale/cli/server.go:24 +0x74 fp=0x4000859b00 sp=0x4000859ac0 pc=0xaaaae28a6424
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/spf13/cobra.(*Command).execute(0xaaaae35989a0, {0xaaaae36c3f00, 0x0, 0x0})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/github.com/spf13/cobra/command.go:944 +0x5b0 fp=0x4000859c40 sp=0x4000859b00 pc=0xaaaae2856910
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/spf13/cobra.(*Command).ExecuteC(0xaaaae3597580)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/github.com/spf13/cobra/command.go:1068 +0x35c fp=0x4000859d00 sp=0x4000859c40 pc=0xaaaae285707c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/spf13/cobra.(*Command).Execute(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/github.com/spf13/cobra/command.go:992
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/cmd/headscale/cli.Execute()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/cmd/headscale/cli/root.go:101 +0x28 fp=0x4000859d50 sp=0x4000859d00 pc=0xaaaae28a4a28
Jul 06 17:28:24 c5b9776eab18 headscale[26]: main.main()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/cmd/headscale/headscale.go:56 +0x3b0 fp=0x4000859f70 sp=0x4000859d50 pc=0xaaaae28d0960
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.main()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:250 +0x248 fp=0x4000859fd0 sp=0x4000859f70 pc=0xaaaae1d4ad18
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000859fd0 sp=0x4000859fd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 2 [force gc (idle), 3 minutes]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x2ae44060a3878?, 0x0?, 0x0?, 0x0?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x400005cfa0 sp=0x400005cf80 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goparkunlock(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:387
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.forcegchelper()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:305 +0xb8 fp=0x400005cfd0 sp=0x400005cfa0 pc=0xaaaae1d4af88
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x400005cfd0 sp=0x400005cfd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by runtime.init.7
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:293 +0x24
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 3 [GC sweep wait]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x400005d760 sp=0x400005d740 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goparkunlock(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:387
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.bgsweep(0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgcsweep.go:319 +0x110 fp=0x400005d7b0 sp=0x400005d760 pc=0xaaaae1d35740
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gcenable.func1()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgc.go:178 +0x28 fp=0x400005d7d0 sp=0x400005d7b0 pc=0xaaaae1d2a268
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x400005d7d0 sp=0x400005d7d0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by runtime.gcenable
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgc.go:178 +0x74
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 4 [GC scavenge wait]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x2ae449c5114f5?, 0x1087cc?, 0x0?, 0x0?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x400005df50 sp=0x400005df30 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goparkunlock(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:387
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.(*scavengerState).park(0xaaaae368cec0)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgcscavenge.go:400 +0x5c fp=0x400005df80 sp=0x400005df50 pc=0xaaaae1d335cc
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.bgscavenge(0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgcscavenge.go:633 +0xac fp=0x400005dfb0 sp=0x400005df80 pc=0xaaaae1d33b8c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gcenable.func2()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgc.go:179 +0x28 fp=0x400005dfd0 sp=0x400005dfb0 pc=0xaaaae1d2a208
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x400005dfd0 sp=0x400005dfd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by runtime.gcenable
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgc.go:179 +0xb8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 5 [finalizer wait, 43 minutes]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x0?, 0x4000710108?, 0x50?, 0x82?, 0x1000000010?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000636d80 sp=0x4000636d60 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.runfinq()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mfinal.go:193 +0x10c fp=0x4000636fd0 sp=0x4000636d80 pc=0xaaaae1d292fc
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000636fd0 sp=0x4000636fd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by runtime.createfing
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mfinal.go:163 +0x84
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 19 [GC worker (idle)]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x2ae44063d3359?, 0x4000091128?, 0x98?, 0xe7?, 0xaaaae2302828?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x400005e740 sp=0x400005e720 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gcBgMarkWorker()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgc.go:1275 +0xec fp=0x400005e7d0 sp=0x400005e740 pc=0xaaaae1d2c1ac
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x400005e7d0 sp=0x400005e7d0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by runtime.gcBgMarkStartWorkers
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgc.go:1199 +0x28
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 20 [GC worker (idle)]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x2ae44063d36e8?, 0x0?, 0x0?, 0x0?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000058740 sp=0x4000058720 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gcBgMarkWorker()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgc.go:1275 +0xec fp=0x40000587d0 sp=0x4000058740 pc=0xaaaae1d2c1ac
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x40000587d0 sp=0x40000587d0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by runtime.gcBgMarkStartWorkers
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgc.go:1199 +0x28
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 26 [select, 7 minutes]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x400006df88?, 0x2?, 0x80?, 0x1?, 0x400006df74?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x400006de10 sp=0x400006ddf0 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.selectgo(0x400006df88, 0x400006df70, 0x4000343f78?, 0x0, 0x4000343f98?, 0x1)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/select.go:327 +0x690 fp=0x400006df30 sp=0x400006de10 pc=0xaaaae1d5c040
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/patrickmn/go-cache.(*janitor).Run(0x40002117b0, 0x4000220840?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/github.com/patrickmn/go-cache/cache.go:1079 +0x78 fp=0x400006dfb0 sp=0x400006df30 pc=0xaaaae22e42f8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/patrickmn/go-cache.runJanitor.func1()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/github.com/patrickmn/go-cache/cache.go:1099 +0x2c fp=0x400006dfd0 sp=0x400006dfb0 pc=0xaaaae22e44fc
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x400006dfd0 sp=0x400006dfd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by github.com/patrickmn/go-cache.runJanitor
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/github.com/patrickmn/go-cache/cache.go:1099 +0x100
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 10 [chan receive]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x400007eba0?, 0x4?, 0xc0?, 0xbb?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000addea0 sp=0x4000adde80 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.chanrecv(0x40006aed20, 0x4000addf90, 0x1)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/chan.go:583 +0x45c fp=0x4000addf30 sp=0x4000addea0 pc=0xaaaae1d1667c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.chanrecv2(0x40000cacc0?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/chan.go:447 +0x14 fp=0x4000addf60 sp=0x4000addf30 pc=0xaaaae1d16204
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).expireEphemeralNodes(0x0?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:217 +0x60 fp=0x4000addfb0 sp=0x4000addf60 pc=0xaaaae27fc8e0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).Serve.func14()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:555 +0x2c fp=0x4000addfd0 sp=0x4000addfb0 pc=0xaaaae28008fc
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000addfd0 sp=0x4000addfd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by github.com/juanfont/headscale/hscontrol.(*Headscale).Serve
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:555 +0x29c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 9 [select, 45 minutes]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x40005b7f08?, 0x2?, 0x40?, 0x60?, 0x40005b7ef4?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000635d40 sp=0x4000635d20 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.selectgo(0x4000635f08, 0x40005b7ef0, 0x22?, 0x0, 0x0?, 0x1)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/select.go:327 +0x690 fp=0x4000635e60 sp=0x4000635d40 pc=0xaaaae1d5c040
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).scheduledDERPMapUpdateWorker(0x40000cacc0, 0x4000648120)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/derp.go:143 +0xd0 fp=0x4000635fb0 sp=0x4000635e60 pc=0xaaaae2808980
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).Serve.func13()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:552 +0x2c fp=0x4000635fd0 sp=0x4000635fb0 pc=0xaaaae280095c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000635fd0 sp=0x4000635fd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by github.com/juanfont/headscale/hscontrol.(*Headscale).Serve
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:552 +0x258
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 53 [chan receive]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0xaaaae2d81420?, 0x4000040170?, 0x68?, 0x1d?, 0xaaaae1f0e0e4?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000841d30 sp=0x4000841d10 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.chanrecv(0x40006af200, 0x4000841f28, 0x1)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/chan.go:583 +0x45c fp=0x4000841dc0 sp=0x4000841d30 pc=0xaaaae1d1667c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.chanrecv1(0x4000903500?, 0xaaaae28f244d?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/chan.go:442 +0x14 fp=0x4000841df0 sp=0x4000841dc0 pc=0xaaaae1d161e4
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).Serve.func6(0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:753 +0xa8 fp=0x4000841f40 sp=0x4000841df0 pc=0xaaaae2800128
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).Serve.func7()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:828 +0x28 fp=0x4000841f60 sp=0x4000841f40 pc=0xaaaae2800058
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/sync/errgroup.(*Group).Go.func1()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x5c fp=0x4000841fd0 sp=0x4000841f60 pc=0xaaaae2332c0c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000841fd0 sp=0x4000841fd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by golang.org/x/sync/errgroup.(*Group).Go
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 11 [chan receive]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x4000af02a0?, 0x4?, 0x40?, 0xbd?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000c09ea0 sp=0x4000c09e80 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.chanrecv(0x40002201e0, 0x4000c09f90, 0x1)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/chan.go:583 +0x45c fp=0x4000c09f30 sp=0x4000c09ea0 pc=0xaaaae1d1667c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.chanrecv2(0x40000cacc0?, 0x40004a1459?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/chan.go:447 +0x14 fp=0x4000c09f60 sp=0x4000c09f30 pc=0xaaaae1d16204
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).expireExpiredMachines(0x3b?, 0x40004a159e?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:226 +0x60 fp=0x4000c09fb0 sp=0x4000c09f60 pc=0xaaaae27fc970
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).Serve.func15()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:556 +0x2c fp=0x4000c09fd0 sp=0x4000c09fb0 pc=0xaaaae280089c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000c09fd0 sp=0x4000c09fd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by github.com/juanfont/headscale/hscontrol.(*Headscale).Serve
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:556 +0x2e0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 12 [chan receive]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x4000220180?, 0xaaaae1d16204?, 0xe0?, 0xed?, 0xaaaae1d7b95c?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x40011bbe90 sp=0x40011bbe70 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.chanrecv(0x4000220120, 0x40011bbf90, 0x1)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/chan.go:583 +0x45c fp=0x40011bbf20 sp=0x40011bbe90 pc=0xaaaae1d1667c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.chanrecv2(0x40000cacc0?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/chan.go:447 +0x14 fp=0x40011bbf50 sp=0x40011bbf20 pc=0xaaaae1d16204
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).failoverSubnetRoutes(0x0?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:233 +0x50 fp=0x40011bbfb0 sp=0x40011bbf50 pc=0xaaaae27fc9f0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).Serve.func16()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:558 +0x2c fp=0x40011bbfd0 sp=0x40011bbfb0 pc=0xaaaae280083c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x40011bbfd0 sp=0x40011bbfd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by github.com/juanfont/headscale/hscontrol.(*Headscale).Serve
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:558 +0x324
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 42 [IO wait, 21 minutes]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000071b90 sp=0x4000071b70 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.netpollblock(0x0?, 0x0?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/netpoll.go:527 +0x158 fp=0x4000071bd0 sp=0x4000071b90 pc=0xaaaae1d43ce8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.runtime_pollWait(0xffff7edd1088, 0x72)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/netpoll.go:306 +0xa0 fp=0x4000071c00 sp=0x4000071bd0 pc=0xaaaae1d78c40
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.(*pollDesc).wait(0x40002e8a00?, 0x0?, 0x0)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x4000071c30 sp=0x4000071c00 pc=0xaaaae1db64f8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.(*pollDesc).waitRead(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.(*FD).Accept(0x40002e8a00)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/internal/poll/fd_unix.go:614 +0x250 fp=0x4000071ce0 sp=0x4000071c30 pc=0xaaaae1dbafa0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net.(*netFD).accept(0x40002e8a00)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/fd_unix.go:172 +0x28 fp=0x4000071da0 sp=0x4000071ce0 pc=0xaaaae1f9bd58
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net.(*TCPListener).accept(0x4000202e28)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/tcpsock_posix.go:148 +0x28 fp=0x4000071dd0 sp=0x4000071da0 pc=0xaaaae1fb3c18
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net.(*TCPListener).Accept(0x4000202e28)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/tcpsock.go:297 +0x2c fp=0x4000071e10 sp=0x4000071dd0 pc=0xaaaae1fb2d6c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.(*onceCloseListener).Accept(0x40008838c0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         <autogenerated>:1 +0x30 fp=0x4000071e30 sp=0x4000071e10 pc=0xaaaae20b2be0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.(*Server).Serve(0x400012e0f0, {0xaaaae2d802a0, 0x4000202e28})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:3059 +0x304 fp=0x4000071f60 sp=0x4000071e30 pc=0xaaaae208e4a4
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.(*Server).ListenAndServe(0x400012e0f0)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:2988 +0x84 fp=0x4000071f90 sp=0x4000071f60 pc=0xaaaae208e164
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).getTLSSettings.func1()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:873 +0x20 fp=0x4000071fd0 sp=0x4000071f90 pc=0xaaaae28012d0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000071fd0 sp=0x4000071fd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by github.com/juanfont/headscale/hscontrol.(*Headscale).getTLSSettings
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:872 +0x340
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 54 [select]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x4000ae13d8?, 0x5?, 0x0?, 0x0?, 0x4000ae0f9e?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000ae0df0 sp=0x4000ae0dd0 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.selectgo(0x4000ae13d8, 0x4000ae0f94, 0xaaaae36c3f00?, 0x0, 0x0?, 0x1)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/select.go:327 +0x690 fp=0x4000ae0f10 sp=0x4000ae0df0 pc=0xaaaae1d5c040
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.(*serverConn).serve(0x4000261200)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/server.go:938 +0x718 fp=0x4000ae1430 sp=0x4000ae0f10 pc=0xaaaae2122f38
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.(*Server).ServeConn(0x40006ff130, {0xaaaae2d86af0?, 0x400012e4b0}, 0x4000ae1840)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/server.go:531 +0x9fc fp=0x4000ae1700 sp=0x4000ae1430 pc=0xaaaae212165c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).NoiseUpgradeHandler(0x40000cacc0, {0xaaaae2d804b0, 0x400025ea80}, 0x40000cd500)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/noise.go:114 +0x4f8 fp=0x4000ae1920 sp=0x4000ae1700 pc=0xaaaae2816408
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).NoiseUpgradeHandler-fm({0xaaaae2d804b0?, 0x400025ea80?}, 0x4000714780?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         <autogenerated>:1 +0x40 fp=0x4000ae1950 sp=0x4000ae1920 pc=0xaaaae283b1b0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.HandlerFunc.ServeHTTP(0x40000cd400?, {0xaaaae2d804b0?, 0x400025ea80?}, 0xaaaae1db69d8?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:2122 +0x38 fp=0x4000ae1980 sp=0x4000ae1950 pc=0xaaaae208b338
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/gorilla/mux.(*Router).ServeHTTP(0x4000ac4180, {0xaaaae2d804b0, 0x400025ea80}, 0x40000cd100)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/github.com/gorilla/mux/mux.go:210 +0x19c fp=0x4000ae1ab0 sp=0x4000ae1980 pc=0xaaaae228d33c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.serverHandler.ServeHTTP({0x400056bda0?}, {0xaaaae2d804b0, 0x400025ea80}, 0x40000cd100)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:2936 +0x2d8 fp=0x4000ae1b60 sp=0x4000ae1ab0 pc=0xaaaae208dfd8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.(*conn).serve(0x40004125a0, {0xaaaae2d81490, 0x400064a2a0})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:1995 +0x560 fp=0x4000ae1fa0 sp=0x4000ae1b60 pc=0xaaaae208a120
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.(*Server).Serve.func3()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:3089 +0x30 fp=0x4000ae1fd0 sp=0x4000ae1fa0 pc=0xaaaae208e800
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000ae1fd0 sp=0x4000ae1fd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by net/http.(*Server).Serve
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:3089 +0x520
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 49 [select, 45 minutes, locked to thread]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x40004f57a0?, 0x2?, 0x78?, 0x56?, 0x40004f579c?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x40004f5620 sp=0x40004f5600 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.selectgo(0x40004f57a0, 0x40004f5798, 0x0?, 0x0, 0x1?, 0x1)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/select.go:327 +0x690 fp=0x40004f5740 sp=0x40004f5620 pc=0xaaaae1d5c040
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.ensureSigM.func1()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/signal_unix.go:1000 +0x1bc fp=0x40004f57d0 sp=0x40004f5740 pc=0xaaaae1d7536c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x40004f57d0 sp=0x40004f57d0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by runtime.ensureSigM
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/signal_unix.go:983 +0xf4
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 66 [syscall]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.notetsleepg(0xffffffffffffffff?, 0x40004f5f20?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/lock_futex.go:236 +0x34 fp=0x40004f5f90 sp=0x40004f5f50 pc=0xaaaae1d1ba24
Jul 06 17:28:24 c5b9776eab18 headscale[26]: os/signal.signal_recv()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/sigqueue.go:152 +0x30 fp=0x40004f5fb0 sp=0x40004f5f90 pc=0xaaaae1d7af90
Jul 06 17:28:24 c5b9776eab18 headscale[26]: os/signal.loop()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/os/signal/signal_unix.go:23 +0x1c fp=0x40004f5fd0 sp=0x40004f5fb0 pc=0xaaaae229858c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x40004f5fd0 sp=0x40004f5fd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by os/signal.Notify.func1.1
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/os/signal/signal.go:151 +0x2c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 71 [IO wait]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x0?, 0x3b9?, 0xc8?, 0x25?, 0xaaaae1d25b74?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000632540 sp=0x4000632520 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.netpollblock(0x0?, 0xffffffff?, 0xff?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/netpoll.go:527 +0x158 fp=0x4000632580 sp=0x4000632540 pc=0xaaaae1d43ce8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.runtime_pollWait(0xffff7edd0f98, 0x72)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/netpoll.go:306 +0xa0 fp=0x40006325b0 sp=0x4000632580 pc=0xaaaae1d78c40
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.(*pollDesc).wait(0x40002e9600?, 0x40005c8000?, 0x0)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x40006325e0 sp=0x40006325b0 pc=0xaaaae1db64f8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.(*pollDesc).waitRead(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.(*FD).Read(0x40002e9600, {0x40005c8000, 0x900, 0x900})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/internal/poll/fd_unix.go:167 +0x200 fp=0x4000632680 sp=0x40006325e0 pc=0xaaaae1db7860
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net.(*netFD).Read(0x40002e9600, {0x40005c8000?, 0x4000632838?, 0xaaaae1ff1f1c?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/fd_posix.go:55 +0x28 fp=0x40006326d0 sp=0x4000632680 pc=0xaaaae1f9a0b8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net.(*conn).Read(0x4000010990, {0x40005c8000?, 0x4000632758?, 0xaaaae1d1cf20?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/net.go:183 +0x34 fp=0x4000632720 sp=0x40006326d0 pc=0xaaaae1faaf54
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net.(*TCPConn).Read(0x4000632798?, {0x40005c8000?, 0x4000bfddd0?, 0x18?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         <autogenerated>:1 +0x2c fp=0x4000632750 sp=0x4000632720 pc=0xaaaae1fbdbec
Jul 06 17:28:24 c5b9776eab18 headscale[26]: crypto/tls.(*atLeastReader).Read(0x4000bfddd0, {0x40005c8000?, 0x4000bfddd0?, 0x0?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/crypto/tls/conn.go:788 +0x40 fp=0x40006327a0 sp=0x4000632750 pc=0xaaaae1ff3700
Jul 06 17:28:24 c5b9776eab18 headscale[26]: bytes.(*Buffer).ReadFrom(0x4000416990, {0xaaaae2d73ec0, 0x4000bfddd0})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/bytes/buffer.go:202 +0x90 fp=0x4000632800 sp=0x40006327a0 pc=0xaaaae1df2980
Jul 06 17:28:24 c5b9776eab18 headscale[26]: crypto/tls.(*Conn).readFromUntil(0x4000416700, {0xaaaae2d76ea0?, 0x4000010990}, 0x8c7?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/crypto/tls/conn.go:810 +0xd4 fp=0x4000632840 sp=0x4000632800 pc=0xaaaae1ff38f4
Jul 06 17:28:24 c5b9776eab18 headscale[26]: crypto/tls.(*Conn).readRecordOrCCS(0x4000416700, 0x0)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/crypto/tls/conn.go:617 +0xd8 fp=0x4000632bb0 sp=0x4000632840 pc=0xaaaae1ff1748
Jul 06 17:28:24 c5b9776eab18 headscale[26]: crypto/tls.(*Conn).readRecord(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/crypto/tls/conn.go:583
Jul 06 17:28:24 c5b9776eab18 headscale[26]: crypto/tls.(*Conn).Read(0x4000416700, {0x400012e560, 0x3, 0xaaaae27a2788?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/crypto/tls/conn.go:1316 +0x178 fp=0x4000632c20 sp=0x4000632bb0 pc=0xaaaae1ff6698
Jul 06 17:28:24 c5b9776eab18 headscale[26]: tailscale.com/net/netutil.(*drainBufConn).Read(0x4000632c78?, {0x400012e560?, 0x400012e520?, 0x4000418000?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/tailscale.com/net/netutil/netutil.go:87 +0x94 fp=0x4000632c50 sp=0x4000632c20 pc=0xaaaae278d854
Jul 06 17:28:24 c5b9776eab18 headscale[26]: tailscale.com/control/controlhttp.(*corkConn).Read(0x4000632ca8?, {0x400012e560?, 0x0?, 0x5?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/tailscale.com/control/controlhttp/server.go:193 +0x5c fp=0x4000632c80 sp=0x4000632c50 pc=0xaaaae27c923c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: tailscale.com/control/controlbase.(*Conn).readNLocked(0x400012e4b0, 0x3)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/tailscale.com/control/controlbase/conn.go:115 +0xe0 fp=0x4000632ce0 sp=0x4000632c80 pc=0xaaaae27a2070
Jul 06 17:28:24 c5b9776eab18 headscale[26]: tailscale.com/control/controlbase.(*Conn).decryptOneLocked(0x400012e4b0)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/tailscale.com/control/controlbase/conn.go:223 +0x1b0 fp=0x4000632d20 sp=0x4000632ce0 pc=0xaaaae27a2730
Jul 06 17:28:24 c5b9776eab18 headscale[26]: tailscale.com/control/controlbase.(*Conn).Read(0x400012e4b0, {0x400025ec80, 0x9, 0x0?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/tailscale.com/control/controlbase/conn.go:253 +0x13c fp=0x4000632d90 sp=0x4000632d20 pc=0xaaaae27a295c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: io.ReadAtLeast({0xffff7e940318, 0x400012e4b0}, {0x400025ec80, 0x9, 0x9}, 0x9)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/io/io.go:332 +0xa0 fp=0x4000632de0 sp=0x4000632d90 pc=0xaaaae1db0f50
Jul 06 17:28:24 c5b9776eab18 headscale[26]: io.ReadFull(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/io/io.go:351
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.readFrameHeader({0x400025ec80?, 0x9?, 0x400056b050?}, {0xffff7e940318?, 0x400012e4b0?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/frame.go:237 +0x58 fp=0x4000632e30 sp=0x4000632de0 pc=0xaaaae21179b8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.(*Framer).ReadFrame(0x400025ec40)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/frame.go:498 +0x84 fp=0x4000632ee0 sp=0x4000632e30 pc=0xaaaae21181b4
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.(*serverConn).readFrames(0x4000261200)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/server.go:818 +0x88 fp=0x4000632fb0 sp=0x4000632ee0 pc=0xaaaae21222a8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.(*serverConn).serve.func7()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/server.go:930 +0x28 fp=0x4000632fd0 sp=0x4000632fb0 pc=0xaaaae21235e8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000632fd0 sp=0x4000632fd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by golang.org/x/net/http2.(*serverConn).serve
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/server.go:930 +0x4a4
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 324 [select]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x400085d3d8?, 0x5?, 0x0?, 0x0?, 0x400085cf9e?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x400085cdf0 sp=0x400085cdd0 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.selectgo(0x400085d3d8, 0x400085cf94, 0xaaaae36c3f00?, 0x0, 0x0?, 0x1)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/select.go:327 +0x690 fp=0x400085cf10 sp=0x400085cdf0 pc=0xaaaae1d5c040
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.(*serverConn).serve(0x40008d1b00)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/server.go:938 +0x718 fp=0x400085d430 sp=0x400085cf10 pc=0xaaaae2122f38
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.(*Server).ServeConn(0x4000ced810, {0xaaaae2d86af0?, 0x400012fc20}, 0x400085d840)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/server.go:531 +0x9fc fp=0x400085d700 sp=0x400085d430 pc=0xaaaae212165c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).NoiseUpgradeHandler(0x40000cacc0, {0xaaaae2d804b0, 0x400025e540}, 0x40008bf600)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/noise.go:114 +0x4f8 fp=0x400085d920 sp=0x400085d700 pc=0xaaaae2816408
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).NoiseUpgradeHandler-fm({0xaaaae2d804b0?, 0x400025e540?}, 0x4000714780?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         <autogenerated>:1 +0x40 fp=0x400085d950 sp=0x400085d920 pc=0xaaaae283b1b0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.HandlerFunc.ServeHTTP(0x40008bf500?, {0xaaaae2d804b0?, 0x400025e540?}, 0xaaaae1db69d8?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:2122 +0x38 fp=0x400085d980 sp=0x400085d950 pc=0xaaaae208b338
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/gorilla/mux.(*Router).ServeHTTP(0x4000ac4180, {0xaaaae2d804b0, 0x400025e540}, 0x40008bf200)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/github.com/gorilla/mux/mux.go:210 +0x19c fp=0x400085dab0 sp=0x400085d980 pc=0xaaaae228d33c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.serverHandler.ServeHTTP({0x4000ce80f0?}, {0xaaaae2d804b0, 0x400025e540}, 0x40008bf200)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:2936 +0x2d8 fp=0x400085db60 sp=0x400085dab0 pc=0xaaaae208dfd8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.(*conn).serve(0x4000cccbd0, {0xaaaae2d81490, 0x400064a2a0})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:1995 +0x560 fp=0x400085dfa0 sp=0x400085db60 pc=0xaaaae208a120
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.(*Server).Serve.func3()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:3089 +0x30 fp=0x400085dfd0 sp=0x400085dfa0 pc=0xaaaae208e800
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x400085dfd0 sp=0x400085dfd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by net/http.(*Server).Serve
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:3089 +0x520
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 328 [IO wait, 5 minutes]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x0?, 0x1ed?, 0xc8?, 0x85?, 0xaaaae1d25b74?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000978540 sp=0x4000978520 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.netpollblock(0x0?, 0xffffffff?, 0xff?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/netpoll.go:527 +0x158 fp=0x4000978580 sp=0x4000978540 pc=0xaaaae1d43ce8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.runtime_pollWait(0xffff7edd1718, 0x72)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/netpoll.go:306 +0xa0 fp=0x40009785b0 sp=0x4000978580 pc=0xaaaae1d78c40
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.(*pollDesc).wait(0x4000898180?, 0x4000c7d000?, 0x0)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x40009785e0 sp=0x40009785b0 pc=0xaaaae1db64f8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.(*pollDesc).waitRead(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.(*FD).Read(0x4000898180, {0x4000c7d000, 0x1000, 0x1000})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/internal/poll/fd_unix.go:167 +0x200 fp=0x4000978680 sp=0x40009785e0 pc=0xaaaae1db7860
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net.(*netFD).Read(0x4000898180, {0x4000c7d000?, 0x4000978838?, 0xaaaae1ff1f1c?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/fd_posix.go:55 +0x28 fp=0x40009786d0 sp=0x4000978680 pc=0xaaaae1f9a0b8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net.(*conn).Read(0x4000010058, {0x4000c7d000?, 0x40009787b8?, 0xaaaae1df250c?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/net.go:183 +0x34 fp=0x4000978720 sp=0x40009786d0 pc=0xaaaae1faaf54
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net.(*TCPConn).Read(0x4000978798?, {0x4000c7d000?, 0x40006ed2d8?, 0x18?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         <autogenerated>:1 +0x2c fp=0x4000978750 sp=0x4000978720 pc=0xaaaae1fbdbec
Jul 06 17:28:24 c5b9776eab18 headscale[26]: crypto/tls.(*atLeastReader).Read(0x40006ed2d8, {0x4000c7d000?, 0x40006ed2d8?, 0x0?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/crypto/tls/conn.go:788 +0x40 fp=0x40009787a0 sp=0x4000978750 pc=0xaaaae1ff3700
Jul 06 17:28:24 c5b9776eab18 headscale[26]: bytes.(*Buffer).ReadFrom(0x4000c25410, {0xaaaae2d73ec0, 0x40006ed2d8})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/bytes/buffer.go:202 +0x90 fp=0x4000978800 sp=0x40009787a0 pc=0xaaaae1df2980
Jul 06 17:28:24 c5b9776eab18 headscale[26]: crypto/tls.(*Conn).readFromUntil(0x4000c25180, {0xaaaae2d76ea0?, 0x4000010058}, 0x1000?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/crypto/tls/conn.go:810 +0xd4 fp=0x4000978840 sp=0x4000978800 pc=0xaaaae1ff38f4
Jul 06 17:28:24 c5b9776eab18 headscale[26]: crypto/tls.(*Conn).readRecordOrCCS(0x4000c25180, 0x0)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/crypto/tls/conn.go:617 +0xd8 fp=0x4000978bb0 sp=0x4000978840 pc=0xaaaae1ff1748
Jul 06 17:28:24 c5b9776eab18 headscale[26]: crypto/tls.(*Conn).readRecord(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/crypto/tls/conn.go:583
Jul 06 17:28:24 c5b9776eab18 headscale[26]: crypto/tls.(*Conn).Read(0x4000c25180, {0x400012fcd0, 0x3, 0xaaaae27a2788?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/crypto/tls/conn.go:1316 +0x178 fp=0x4000978c20 sp=0x4000978bb0 pc=0xaaaae1ff6698
Jul 06 17:28:24 c5b9776eab18 headscale[26]: tailscale.com/net/netutil.(*drainBufConn).Read(0x4000978c78?, {0x400012fcd0?, 0x400012fc90?, 0x40000ff000?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/tailscale.com/net/netutil/netutil.go:87 +0x94 fp=0x4000978c50 sp=0x4000978c20 pc=0xaaaae278d854
Jul 06 17:28:24 c5b9776eab18 headscale[26]: tailscale.com/control/controlhttp.(*corkConn).Read(0x4000978ca8?, {0x400012fcd0?, 0x0?, 0x5?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/tailscale.com/control/controlhttp/server.go:193 +0x5c fp=0x4000978c80 sp=0x4000978c50 pc=0xaaaae27c923c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: tailscale.com/control/controlbase.(*Conn).readNLocked(0x400012fc20, 0x3)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/tailscale.com/control/controlbase/conn.go:115 +0xe0 fp=0x4000978ce0 sp=0x4000978c80 pc=0xaaaae27a2070
Jul 06 17:28:24 c5b9776eab18 headscale[26]: tailscale.com/control/controlbase.(*Conn).decryptOneLocked(0x400012fc20)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/tailscale.com/control/controlbase/conn.go:223 +0x1b0 fp=0x4000978d20 sp=0x4000978ce0 pc=0xaaaae27a2730
Jul 06 17:28:24 c5b9776eab18 headscale[26]: tailscale.com/control/controlbase.(*Conn).Read(0x400012fc20, {0x400025e740, 0x9, 0x0?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/tailscale.com/control/controlbase/conn.go:253 +0x13c fp=0x4000978d90 sp=0x4000978d20 pc=0xaaaae27a295c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: io.ReadAtLeast({0xffff7e940318, 0x400012fc20}, {0x400025e740, 0x9, 0x9}, 0x9)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/io/io.go:332 +0xa0 fp=0x4000978de0 sp=0x4000978d90 pc=0xaaaae1db0f50
Jul 06 17:28:24 c5b9776eab18 headscale[26]: io.ReadFull(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/io/io.go:351
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.readFrameHeader({0x400025e740?, 0x9?, 0x400056bdd0?}, {0xffff7e940318?, 0x400012fc20?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/frame.go:237 +0x58 fp=0x4000978e30 sp=0x4000978de0 pc=0xaaaae21179b8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.(*Framer).ReadFrame(0x400025e700)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/frame.go:498 +0x84 fp=0x4000978ee0 sp=0x4000978e30 pc=0xaaaae21181b4
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.(*serverConn).readFrames(0x40008d1b00)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/server.go:818 +0x88 fp=0x4000978fb0 sp=0x4000978ee0 pc=0xaaaae21222a8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.(*serverConn).serve.func7()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/server.go:930 +0x28 fp=0x4000978fd0 sp=0x4000978fb0 pc=0xaaaae21235e8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000978fd0 sp=0x4000978fd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by golang.org/x/net/http2.(*serverConn).serve
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/server.go:930 +0x4a4
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r0      0xfffffffffffffffc
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r1      0x80
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r2      0x0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r3      0xffff81659748
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r4      0x0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r5      0x0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r6      0xb83c3
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r7      0x2cddc5c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r8      0x62
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r9      0x3d633814959ed
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r10     0xffffffffffffff
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r11     0x53a596422cfe
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r12     0xaaaae36c395e
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r13     0xffffe1d1b760
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r14     0x7
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r15     0x88
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r16     0xffff80e5a018
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r17     0xffff81658fc0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r18     0x6474e552
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r19     0xaaaae1d7c5e0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r20     0xffff81659730
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r21     0x4000060400
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r22     0x1
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r23     0x80ea40
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r24     0xffffe1d1b4c7
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r25     0xffff80e4b000
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r26     0xaaaae2d703c8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r27     0xaaaae35a4000
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r28     0x40000029c0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r29     0xffff81659708
Jul 06 17:28:24 c5b9776eab18 headscale[26]: lr      0xaaaae1d44c30
Jul 06 17:28:24 c5b9776eab18 headscale[26]: sp      0xffff81659710
Jul 06 17:28:24 c5b9776eab18 headscale[26]: pc      0xaaaae1d80110
Jul 06 17:28:24 c5b9776eab18 headscale[26]: fault   0x0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: -----
[root@c5b9776eab18 /]#
[root@c5b9776eab18 /]# Jul 06 17:28:29 c5b9776eab18 systemd[1]: headscale.service: Main process exited, code=dumped, status=6/ABRT
Jul 06 17:28:29 c5b9776eab18 systemd[1]: headscale.service: Failed with result 'timeout'.
Jul 06 17:28:29 c5b9776eab18 systemd[1]: Stopped headscale.service - headscale coordination server for Tailscale.
Jul 06 17:28:29 c5b9776eab18 systemd[1]: headscale.service: Consumed 1.862s CPU time.
@jonathanspw
Copy link
Owner

I've not run into this running it in full VMs. Does it happen only with the RPM or also when installed manually in the container?

@dustymabe
Copy link
Contributor Author

Haven't had a chance to run it in a VM. Will post here if I get a chance to try it this weekend.

@ajayramaswamy
Copy link

This patch fixes it for me
juanfont/headscale#1480

@codyro codyro self-assigned this Aug 7, 2023
codyro added a commit to codyro/headscale-rpm that referenced this issue Aug 8, 2023
@codyro
Add patch to fix jonathanspw#2
2930918 
- PR #1480 upstream
- Adjust day in changelog to by the proper day to fix warning during build
- Bump release to 3
@codyro
Copy link
Collaborator

codyro commented Aug 8, 2023

This patch is against a different version of app.go than the one in the vendored source/tagged v0.22.3 (latest), and the differences around this change are somewhat significant.

I've gone ahead and made a modified patch (adding a return after the close() function; (here) to see if this resolves the issue for you (I'm doubtful, but it's worth a shot).

Could you try this build, and if it resolves the issue, I'll merge it into this repo. Otherwise, we'll wait until the next tagged release (unless @jonathanspw has a better idea).

https://copr.fedorainfracloud.org/coprs/codyrobertson/headscale/build/6252827/

@codyro codyro mentioned this issue Aug 8, 2023
Open
@jonathanspw
Copy link
Owner

@dustymabe any report on if @codyro 's patch above fixed your issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@codyro codyro

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@codyro @ajayramaswamy @dustymabe @jonathanspw