Tailscale on NetBSD - Proof of Concept

2022-02-16

I’m currently working on porting Tailscale to NetBSD. Actually, I already have the core functionality working (see screenshot below). I don’t have a full idea of what the rest of the port will look like, but there’s plenty of additional features and loose ends that I need to chase down until this moves from proof of concept to something upstreamable. This also relies on adding a NetBSD backend into wireguard-go, which I actually have no idea how to upstream, but I’ll burn that bridge when I get to it. Anyway, I’m gonna talk about what I’ve done so far and what needs to come next.

Things are still rough around the edges so I’m not posting the forks I’m working on just yet, but check back in later because I will absolutely be sharing the code for this once I have something with a more solid foundation.

wireguard-go

So the first part of this puzzle is wireguard-go, which is the official golang implementation of wireguard. Tailscale uses this on operating systems that don’t have a native version of wireguard in the kernel. wireguard-go is written with a modular structure such that most of it is independent of the operating system, and then there’s a single file for each OS that implements the necessary plumbing to get it up and running as a network device. Now, there’s no official NetBSD backend for wireguard-go, but I found this weird fork on the deep web that implements the interface with NetBSD’s tun devices. It hasn’t been updated in a couple years, so I made a couple minor modifications and rebased it on a more recent stable release, and wadya know we’re in business!

I haven’t run this code through an extensive test to make sure it handles any potential edge cases appropriately, but we’ve got a good starting point to work off of. It’d be really cool to get this upstreamed into the main wireguard-go project after a bit more work on it; until that happens, I’m using an override to build against a local fork of the code.

cd tailscale
go mod edit -replace golang.zx2c4.com/wireguard=/path/to/local/fork

At this point I’ve got a local clone of my fork of snow’s fork of wireguard-go and I’ve told go to use it when building tailscaled. So then, let’s do that!

tailscaled

tailscaled is the daemon that holds all the magick to make Tailscale work. It’s got some OS-specific codepaths for network diagnostics and configuring the network stack’s routing tables that we need to address before we can give it a try.

Routing

When an application asks the operating system to send a packet to an IP address, the OS’s network stack checks a list of what IP address ranges are accessible through which network devices to figure out which device the packet should go through. Every OS has a different way to configure this, so tailscale has OS-specific implementations in wgengine/router that make this work.

I must confess to you, dear reader, that I have committed computer crimes, because I didn’t actually write a NetBSD backend for this. Instead I did a bit of the ol’

cp router_openbsd.go router_netbsd.go

And uh, it just worked? It’s not ready to ship, but it was enough to get a connection to another device on the tailnet, so that’s a win! I did also try the router_userspace_bsd backend used by FreeBSD and macOS but that one failed immediately. For now we’re using the OpenBSD one, and I’ll work on changes to it as necessary to iron things out.

And speaking of ironing things out, we’ve already got one candidate problem off the bat from the health check in tailscale status:

# Health check:
#     - router: exit status 1

What’s up with this? Well, if we take a look at tailscaled’s logs, we can find

router: route del failed: [route -q -n del -inet 100.110.144.67/32 -iface 100.113.133.86]: exit status 1
route: botched keyword: del
Usage: route [-dfLnqSsTtv] cmd [[-<qualifers>] args]

So adding routes is working, but cleaning them up afterwards isn’t- why? It’s pretty simple actually. Searching through NetBSD’s man pages, there’s no mention of del as a shorthand for delete:

root@localhost ~/tailscale (main)# man route
   The route utility provides several commands:

   add         Add a route.
   flush       Remove all routes.
   flushall    Remove all routes including the default gateway.
   delete      Delete a specific route.
   change      Change aspects of a route (such as its gateway).
   get         Lookup and display the route for a destination.
   show        Print out the route table similar to "netstat -r" (see
               netstat(1)).
   monitor     Continuously report any changes to the routing information
               base, routing lookup misses, or suspected network
               partitionings.

I double checked the source code and sure enough, delete is a keyword but del isn’t.

The BSDs share a lot of history but there’s often little quirks like this that you’ve got to look out for with the userspace utilities. I’ll need to fix that up to use delete, and check for any other problems while I’m at it.

Port list

The port list is a preview feature of tailscale that you can turn on which shows a view in the admin of all the ports open on your tailscale interface. tailscaled uses the command line tool netstat to collect this information on the other BSDs, and getting this working was just a case of turning it on in the build flags.

diff --git a/portlist/netstat_exec.go b/portlist/netstat_exec.go
index 77972d98..3959d291 100644
--- a/portlist/netstat_exec.go
+++ b/portlist/netstat_exec.go
@@ -2,7 +2,7 @@
 // Use of this source code is governed by a BSD-style
 // license that can be found in the LICENSE file.

-//go:build (windows || freebsd || openbsd || darwin) && !ios
+//go:build (windows || freebsd || openbsd || darwin || netbsd) && !ios
 // +build windows freebsd openbsd darwin
 // +build !ios

What’s next?

With that rough draft of functionality up and running, what else is there to do? I think the first thing is to go through and refine the route control code to fully function on NetBSD. At that point I’ll post my forks of tailscale and snow’s NetBSD wireguard-go backend without worrying that it’s going to do terrible things to peoples’ routing tables.

Longer term, tailscale has a whole host of other features that need to be tested and verified for this to function, but the big wildcard is the tunnel interface implementation. I doubt tailscale wants to merge in a pull request that relies on a third party fork of wireguard-go, but I also have no idea what goes in to acceptance testing and merging in a new backend for wireguard-go, so there’s a lot of unknowns there. If you have any advice, please let me know, I’d love to hear from you.

Either way, even if I don’t get anything upstreamed I’ll still upload my forks with instructions for anyone willing to do a bit of DIY, so stay tuned!