Original post: headscale多租户改造，ACL、路由、DNS等全隔离

Transformation Goals​

Headscale is originally single-instance and effectively single-tailnet by default. Multi-tenant support requires strict per-tenant isolation for:

• tailnet address spaces
• ACL policies
• route definitions
• MagicDNS namespaces
• CLI management flow
• relay server strategy

Key Points​

Tailnet​

• Each tenant has an independent tailnet.
• Default pool 10.64.0.0/10 can remain, or be customized (for example 192.168.6.0/24).
• A built-in default tenant keeps backward-compatible behavior.

ACL and Routes​

• ACL policies are fully tenant-scoped.
• Route configs are tenant-scoped as well.

MagicDNS​

• FQDN pattern: hostname.<tenant_key>.<dns.base_domain>.
• The default tenant uses default as subdomain.

CLI​

• CLI commands add tenant targeting (for example -t).

Relay Servers​

• Tenants can share relay infra.
• Or each tenant can use dedicated relay nodes for stronger isolation.

Example Screenshots​

This article is mirrored on the Larktun blog. For source updates and original context, refer to: headscale多租户改造，ACL、路由、DNS等全隔离

Original post: headscale-系列：Headscale-SaaS-环境下的用户无缝切换实战

Architecture Recap​

• User subdomains (for example hsa.demo.com) point to Traefik.
• Traefik routes by host to a target headscale node.
• Backend data is isolated and migrated by cluster_id.

Migration Challenge​

Existing TCP long-lived connections usually stay attached to the old backend even after proxy route updates.

Practical Migration Steps​

1) Move user data​

# Export user-specific data from old node
pg_dump -h node3-db -U headscale -t nodes -t ip_addresses \
  --where="user_id = (SELECT id FROM users WHERE name = 'hsa')" \
  > hsa_data.sql

# Update cluster_id and import into target node
sed -i 's/cluster_id: 3/cluster_id: 4/g' hsa_data.sql
psql -h node4-db -U headscale -d headscale < hsa_data.sql

2) Hot-update Traefik routing​

http:
  routers:
    hsa-router:
      rule: "Host(`hsa.demo.com`)"
      service: "headscale-cluster4"
      entryPoints: ["https"]

curl -X POST http://traefik/api/providers/file?dynamic=true

3) Precisely cut old connection on source node​

ss -K "dport = 7890 and src 203.0.113.5"

Clients usually reconnect to the new node within seconds using their existing node keys, without re-authentication.

Observed Results (from original post)​

• Migration completion: typically under 15 seconds
• Client interruption: typically 1-5 seconds
• Re-authentication rate: near 0%

This article is mirrored on the Larktun blog. For source updates and original context, refer to: headscale-系列：Headscale-SaaS-环境下的用户无缝切换实战

Original post: headscale 系列：组建 headscale 集群，把设备接入能力做成“无限接近无限”

Core Takeaway​

These building blocks move headscale toward a SaaS-style control plane:

• cluster_id logical sharding
• Shared PG/PG cluster
• ACS for tenant assignment and sticky routing
• Mandatory cluster_id filtering in DAO/ORM
• Incremental netmap calculation instead of full rebuilds

Architecture Highlights​

1. Multiple headscale nodes run in parallel, each bound to a unique cluster_id.
2. All nodes share one database layer, but every query must include cluster_id.
3. ACS handles first-time assignment, policy constraints, and balancing.
4. LISTEN/NOTIFY + debounce limits netmap recomputation scope and CPU impact.

Startup Example​

CLUSTER_ID=A \
PG_DSN="postgres://..." \
DERP_MAP_URL="https://derp.example.com/map.json" \
./headscale --config ./config.yaml

Isolation Snippets​

ALTER TABLE nodes ADD COLUMN cluster_id TEXT NOT NULL DEFAULT 'default';
CREATE INDEX idx_nodes_cluster_id ON nodes(cluster_id);

func (c *ClusterDB) Scoped() *gorm.DB {
  return c.db.Where("cluster_id = ?", c.clusterID)
}

Incremental Netmap Strategy​

• On data changes, emit NOTIFY netmap_dirty(cluster_id, scope_key).
• Each node consumes only matching cluster_id events.
• Rebuild only affected scope (user/tag/route domain) rather than full sets.
• Add caching + versioning to minimize redundant computation.

Demo Screenshots​

Related Link​

• Project: OwnDing/headscale-saas

This article is mirrored on the Larktun blog. For source updates and original context, refer to: headscale 系列：组建 headscale 集群，把设备接入能力做成“无限接近无限”

Original post: headscale系列：headsale压力测试

Test Scripts​

• Repository: OwnDing/headscale-test-scripts

Quick Runbook​

# 1) Prepare env
cp .env.example .env

# 2) Validate host environment
bash scripts/00_check_env.sh

# 3) Bootstrap clients (default 200)
bash scripts/01_bootstrap_up.sh

# 4) Export node list
bash scripts/02_list_nodes.sh

# 5) Churn test (terminal A)
bash scripts/03_churn.sh

# 6) Traffic rounds (terminal B)
watch -n 20 'bash scripts/04_traffic_round.sh'

# 7) Cleanup
bash scripts/99_cleanup.sh

Key Environment Variables​

HEADSCALE_URL=https://headscale.example.com
TS_AUTHKEY=tskey-auth-xxxxxxxxxxxxxxxxxxxxxxxxxxxx

REPLICAS=200
CHURN_INTERVAL=30
CHURN_BATCH_PERCENT=10
TRAFFIC_PAIR_COUNT=30
TRAFFIC_TCP_DURATION=10
TRAFFIC_TCP_PARALLEL=4
TRAFFIC_UDP_DURATION=10
TRAFFIC_UDP_BW=50M

Recommended Sizing (200 devices)​

• Load generator: 16 vCPU / 32 GB / 100 GB NVMe
• Headscale control plane target: 4 vCPU / 8 GB / 100 GB
• Optional DERP relay: 2-4 vCPU / 2-4 GB

Script Roles​

• 00_check_env.sh: validate docker/compose/tun and pre-pull images.
• 01_bootstrap_up.sh: scale client containers to REPLICAS.
• 02_list_nodes.sh: export nodes.tsv (container -> Tailnet IPv4).
• 03_churn.sh: periodic random up/down/restart.
• 04_traffic_round.sh: random TCP/UDP pair traffic rounds.
• 99_cleanup.sh: stop and remove test containers and sidecars.

Result Screenshots​

This article is mirrored on the Larktun blog. For source updates and original context, refer to: headscale系列：headsale压力测试

Original post: headscale系列：如何在headscale中使用MagicDNS

Scenario​

• After login, clients can communicate with peers using hostnames in addition to IP.
• On Windows, tailscale writes host mappings into the system hosts file.
• This is especially useful when using features like 4via6.

Enable MagicDNS​

Update your config.yaml (core options shown below):

dns:
  magic_dns: true
  base_domain: example.com
  override_local_dns: false

  nameservers:
    global:
      - 1.1.1.1
      - 1.0.0.1

  search_domains: []
  extra_records_path: /var/lib/headscale/extra-records.json

Using extra_records_path makes DNS record updates easier to maintain.

extra-records.json Example​

[
  {
    "name": "192-168-6-1-via-7",
    "type": "AAAA",
    "value": "fd7a:115c:a1e0:b1a:0:7:c0a8:601"
  }
]

• type supports A (IPv4) and AAAA (IPv6).
• On first migration from inline extra_records to file-based records, one restart is recommended.
• After that, changing extra-records.json is usually enough.

Usage​

After clients come online, you can access target services by the custom DNS name.

This article is mirrored on the Larktun blog. For source updates and original context, refer to: headscale系列：如何在headscale中使用MagicDNS

This post documents a closer-to-official approach: use ko to package Headscale into a container image, instead of maintaining a Dockerfile.

Original post: Headscale series: package source-built headscale as Docker image

Why ko​

ko is a lightweight tool for building and packaging Go applications for container platforms (Docker / Kubernetes). It can build container images directly from Go source, without writing a Dockerfile.

Install ko​

Install ko as a regular user:

go install github.com/google/ko@latest

After installation, the binary is typically available at ~/go/bin/ko.

Build Headscale Image with ko​

Run in the Headscale source directory:

# --local means build locally without pushing to a registry
/home/djc/go/bin/ko build --local ./cmd/headscale

After build, local images named ko.local/... will be created. Example docker images output:

REPOSITORY                                            TAG                                                                IMAGE ID       CREATED      SIZE
headscale                                             v0.26.1-r                                                          bf66da388ca1   6 days ago   87.5MB
ko.local/headscale-f40b3d8640713cd381403459ebd67e78   38aefca56cab7d9b11692c61968915fb59fdf1dce134e52fed02ae2fa3a0e871   bf66da388ca1   6 days ago   87.5MB
ko.local/headscale-f40b3d8640713cd381403459ebd67e78   latest                                                             bf66da388ca1   6 days ago   87.5MB
ghcr.io/juanfont/headscale                            v0.26.1                                                            b9e7b75fd3b0   N/A          80.8MB

You can rename the generated image with docker tag.

Runtime Notes​

At runtime, prepare:

• config.yaml
• Read/write permissions for /var/run/headscale/ and /var/lib/headscale/

In config.yaml, you can change:

unix_socket: /var/run/headscale/headscale.sock

to:

unix_socket: /var/lib/headscale/headscale.sock

Then run the container with only /var/lib/headscale/ mounted:

docker run -d \
  -v ./headscale/config.yaml:/etc/headscale/config.yaml \
  -v ./doc:/var/lib/headscale \
  --name headscale \
  -p 8080:8080 \
  -p 9090:9090 \
  headscale:v0.26.1-r serve

Reference​

• Source build article: Build Headscale from Source

This article is mirrored on the Larktun blog. For source updates and original context, refer to: Headscale series: package source-built headscale as Docker image

Original post: Using Headscale Source Code to Build and Package

Context​

• OS: Windows 11 + WSL2 Ubuntu 24.04 LTS
• Example version: Headscale v0.26.1
• Repository: github.com/juanfont/headscale
• Recommendation: clone with Git instead of downloading a source archive

Environment Setup​

1) Clone the source​

git clone https://github.com/juanfont/headscale.git

# Checkout v0.26.1
git checkout -b release-v0.26.1 v0.26.1

Then copy config-example.yaml to config.yaml in the project root.

2) Configure WSL​

Create .wslconfig in your Windows user directory (for example, C:\Users\XXX):

[wsl2]
nestedVirtualization=true
ipv6=true

[experimental]
autoMemoryReclaim=gradual
networkingMode=mirrored
dnsTunneling=true
firewall=true
autoProxy=true

Restart WSL before continuing.

3) Install Nix (Multi-user)​

sh <(curl --proto '=https' --tlsv1.2 -L https://nixos.org/nix/install) --daemon

Reference: nixos.org/download

Build Flow​

Run build steps as a regular user (do not run make build as root):

# Enter source directory
cd /mnt/c/Users/XXX/Documents/develop/0me/headscale

# Enter dev shell
nix develop

make generate
make test
make build

# Check build output
ls -la result/
cd result/bin
./headscale version

After a successful build, a result directory is generated.

Running the Binary​

After copying the built binary to a target server, check dependencies first:

ldd headscale
file headscale

If your binary points to a Nix interpreter path, prepare the required path and linker file based on ldd output. Also verify that config.yaml, /root/.headscale/, and /var/lib/headscale/ exist with correct permissions.

Packaging as Docker Image​

If you want to package the source-built binary into a Docker image, see: Headscale series: package source-built headscale as Docker image

Common Build Errors​

Issue 1: dirtyShortRev missing​

When make build fails with attribute 'dirtyShortRev' missing, add a fallback in flake.nix:

headscaleVersion = if self ? shortRev
                  then self.shortRev
                  else if self ? dirtyShortRev
                  then self.dirtyShortRev
                  else "v0.26.1";

Issue 2: missing config.yaml during test phase​

If build fails in checks, prepare config and build binary directly:

cp config-example.yaml config.yaml
go mod tidy
go build -o headscale ./cmd/headscale

Issue 3: nix-command disabled​

echo 'experimental-features = nix-command flakes' >> /etc/nix/nix.conf

Run the config command with root.

Issue 4: flake.nix not found​

If nix develop reports no flake.nix, switch to the source directory first, then retry.

Issue 5: initdb cannot run as root​

PostgreSQL init in tests cannot run as root. Use a regular user for the build and test flow.

Manual Dependencies​

# Install Buf
go install github.com/bufbuild/buf/cmd/buf@v1.55.1

# Install Protobuf
sudo apt update
sudo apt install protobuf-compiler

This article is mirrored on the Larktun blog. For source updates and original context, refer to: Using Headscale Source Code to Build and Package