Chris Goffinet
ThreeComma

ThreeComma

CircleCI meet Tailscale

Easily connect CircleCI to your private Tailscale network

CircleCI meet Tailscale
Chris Goffinet's photo
Chris Goffinet

Published on Sep 15, 2021

4 min read

Subscribe to my newsletter and never miss my upcoming articles

Over the past year, I've been watching closely a company called Tailscale in excitement. They make software with the tagline:

A secure network that just works. It's a zero-config VPN.

As someone who understands the difficulty of making software simple, let me tell you it's no easy feat. Their tagline alone piqued my interest! At ThreeComma we are in love with writing Go, automation, and helping companies solve complex engineering problems. We believe software should be simple and make our lives more enjoyable.

These last few months we've had many clients trying out Tailscale to replace their OpenVPN installations. One customer of ours Bolt Financial, uses a popular CI/CD solution called CircleCI for building out their container images. They still have a few legacy systems in place that use Jenkins inside their network. Recently it was brought to our attention that in their pipeline of CircleCI they reach out to a bastion host over SSH, that fires off some requests inside their infrastructure.

This kind of problem sounds perfect for Tailscale. We can remove the need for connecting to this external bastion, not worry about firewalls, rotation of ssh keys, and more. Starting with Tailscale 1.8, they introduced userspace networking that makes it possible in places like containers where you are unable to create a VPN tunnel device.

I've been using Tailscale inside Cloud Run successfully within the last month. We open-sourced a tailscale-cloudrun-reverse proxy recently that makes it dead simple to create incoming webhooks with a public endpoint, that could route to our internal network. So this all sounded easy, let's just create a CircleCI Orb for others to easily add to their workflows! Turns out, there's was a problem...

CircleCI containers security != the same as Cloud Run's container security. When attempting to run the following:

$ sudo tailscaled --tun=userspace-networking
Program starting: v1.14.0-t5cff36945-g809e87bba, Go 1.17-tsdd6b269c77: []string{"tailscaled", "-verbose", "10", "--tun=userspace-networking"}
LogID: 7226a55e02478af0f1ea71db909049bc7042675461dba456521d594577bd3870
logpolicy: using system state directory "/var/lib/tailscale"
.....
link state: interfaces.State{defaultRoute=eth0 ifs={eth0:[192.168.96.3/20]} v4=true v6=false}
magicsock: unable to bind udp4 port 0: listen udp4 :0: setting SO_BINDTODEVICE: operation not permitted
wgengine.NewUserspaceEngine(tun "userspace-networking") error: wgengine: magicsock: initialBind IPv4 failed: failed to bind any ports (tried [0])
wgengine.New: wgengine: magicsock: initialBind IPv4 failed: failed to bind any ports (tried [0])
flushing log.

Thankfully the great people at Tailscale helped resolve this fairly quickly after opening issue #2827. I decided to spend my Friday night testing that new build, finding another bug, fix, re-compile, and finally nailing it all down! It looks like the issue was their library called netns. It tries using FW marks (which requires CAP_NET_ADMIN) permission, and falls back to SO_BINDTODEVICE which requires (CAP_NET_RAW). It looks like in both of those cases CircleCI's container security does not have these permissions when running in their container environment.

Luckily we don't need to enable netns since we are using the userspace-networking option when running tailscaled. Bypassing that early in the process allows everything to work correctly.

After we had this running, we decided to open-source our CircleCI Tailscale Orb!

I really hope this help's others who decide to jump on the Tailscale bandwagon and just want a simple drop-in for their CircleCI workflows to make Tailscale just work

Quick Start Guide

  1. Goto the Tailscale Admin Console and create a new Auth Key. Select Ephemeral Key

Ephemeral Keys do not associate an IPv4 address, only IPv6. This means if you use this type, the machine you are trying to hit must have IPv6 enabled. It's recommended that you use Magic DNS, as it will resolve the AAAA record (IPv6) automatically if you try to address the machine through tools like curl

  1. Create an environment variable in your project: TAILSCALE_AUTH_KEY and paste the new key you created.

  2. The orb automatically exposes environment variables: [http_proxy,https_proxy,ALL_PROXY,HTTP_PROXY,HTTPS_PROXY] that populates to socks5h://localhost:1055/.

This makes it compatible with various applications like curl that respect these environment variables to proxy through a socks5 proxy.

The reason we use socks5h is to force DNS resolution through the socks5 proxy that is setup with Tailscale.

Sample workflow in CircleCI

Here is a sample .circleci/config.yml If you would like to change the tailscale version you can set the parameter tailscale-version.

version: 2.1

orbs:
  circleci-tailscale: threecomma/circleci-tailscale@1.0.0

jobs:
  build:
    docker:
      - image: circleci/node:fermium-stretch
    parameters:
      tailscale-auth-key:
        type: env_var_name
        default: TAILSCALE_AUTH_KEY
    steps:
      - checkout
      - circleci-tailscale/connect
      - run:
          name: curl a tailscale machine over port 8080
          command: |
             until curl "http://[machine].[namespace].beta.tailscale.net:8080/"
             do
              sleep 1
             done