# Dokploy on Docker Swarm — Hetzner Cloud, Fedora 44

**Target architecture**

```
Internet
   │
   ▼
Hetzner Load Balancer (public IP, TCP passthrough 80/443 + PROXY protocol)
   │  (Hetzner private network 10.100.1.0/24 — NOT filtered by Cloud Firewall)
   ▼
dokploy-mgr-1 ─ dokploy-mgr-2 ─ dokploy-mgr-3     ← Swarm managers, Traefik (global,
   │                                                 host-mode 80/443, /ping on 8080)
dokploy-wrk-1 ─ dokploy-wrk-2 ─ dokploy-wrk-3     ← Swarm workers (app containers only)

Admin plane: Tailscale on every node (Tailscale SSH enabled, public sshd also open on 22)
Dokploy UI:  port 3000, host-mode on mgr-1 only, reachable via Tailscale only
TLS:         Traefik + Let's Encrypt (DNS-01), LB is pure TCP passthrough
```

* 6 × Fedora 44 cloud servers in one Hetzner project, all x86.
* Swarm control plane and data plane run over the Hetzner **private network**. Tailscale is admin-only.
* The Hetzner **Cloud Firewall** guards only the public interfaces: inbound 22/tcp (SSH), 41641/udp (Tailscale direct connections), ICMP. Nothing else — 80/443 traffic reaches the nodes via the LB over the private network, which the Cloud Firewall does not filter.
* Health check: every LB target answers `GET /ping` on **:8080 with HTTP 200** (Traefik's ping endpoint, published in host mode so the check reflects that node's actual Traefik, not the routing mesh).

---

## 0. Verified vs. unverified — read before starting

**Verified (July 2026):**

1. Hetzner offers `fedora-44` as a Rapid Deploy image (x86 + ARM); its cloud-init 25.3 auto-configures Hetzner private networks. *(Hetzner Cloud API changelog)*
2. Docker CE 29.x installs and runs on Fedora 44 from Docker's official repo, using DNF5 syntax. *(docs.docker.com + multiple verified installs on fc44 kernels)*
3. Dokploy supports Fedora; its documented **manual / existing-swarm install path** is what we use, because the one-liner installer runs `docker swarm leave --force` and would destroy the swarm we prepare. *(docs.dokploy.com/docs/core/manual-installation)*
4. Dokploy officially supports Traefik as a **Docker Swarm service with host-mode published ports** ("multi-node/production" mode) and auto-detects service vs. container when it reloads Traefik. App/domain routing is generated as **swarm labels** read via Traefik's swarm provider — consistent across all Traefik instances on managers.
5. Hetzner Cloud Firewalls **do not filter private-network traffic** (Hetzner FAQ: private networks are considered "secure").
6. Hetzner private networks have an **MTU of 1450**. Docker overlay networks default to 1500 → VXLAN packets get dropped → the exact failure is documented in Dokploy issue #3446 (Traefik 504s to workers). Fix: create every overlay network with MTU **1400** (1450 underlay − 50 bytes VXLAN overhead). Some guides say 1450; that ignores VXLAN overhead on the private underlay — use 1400.
7. Dokploy cluster mode requires a **container registry**; current docs steer to external registries (GHCR, Docker Hub, …). The registry is needed for images **Dokploy builds** — prebuilt public images are pulled by each node directly.
8. Hetzner Load Balancers support **TCP/HTTP/HTTPS only — no UDP**. Therefore HTTP/3 (QUIC, 443/udp) cannot pass through this setup; we remove the `http3` advertisement so browsers don't attempt it.
9. Hetzner Cloud Firewalls attach to **servers**, not Load Balancers, and LB services have no source-IP filtering. The canonical Hetzner pattern is: firewall the servers, leave the LB publicly open. Consequence: "allow only Cloudflare" cannot be enforced at the Hetzner firewall in front of the LB — it is enforced at Traefik (§10-B).
10. Fedora 41+ replaced classic `dnf-automatic` with **`dnf5-plugin-automatic`**: timer unit `dnf5-automatic.timer`, config `/etc/dnf/automatic.conf` (same format), `reboot = never|when-changed|when-needed` supported, and a `[base]` section that overrides dnf settings **for automatic runs only** — which is what lets §2-B exclude the container engine and kernel from unattended updates without blocking manual upgrades.

**Unverified — check yourself where flagged:**

* **A.** *(resolved — user-verified)* The Hetzner Fedora 44 image ships **without** firewalld. §4 explains why leaving it absent is the recommended choice here, and what installing it would and would not buy you.
* **B.** Exact `hcloud` CLI flag names for LB health checks drift between CLI versions. The commands below match the current documented pattern; confirm with `hcloud load-balancer add-service --help` before running.
* **C.** The default `traefik.yml` Dokploy generates can change between Dokploy releases. All Traefik edits below are given as **targeted diffs against the documented v0.26-era default** — inspect your generated file first, don't blind-paste.
* **D.** Running Traefik **global on all three managers** is my adaptation: Dokploy supports service mode officially, but ships it single-node. Global-on-managers is the standard Swarm HA pattern; it is not a Dokploy-documented topology. A fully stock single-node variant is given in §9-B.

---

## 1. Prerequisites (workstation)

* `hcloud` CLI installed and authenticated: `hcloud context create dokploy` (paste a project API token with read/write).
* A Tailscale account (tailnet).
* A domain you control, with DNS at a provider Traefik's ACME DNS-01 supports — this tutorial shows **Hetzner DNS** and **Cloudflare**. You need an API token for it (Step 9).
* An SSH keypair.
* Optional: `jq`.

```bash
# Upload your SSH key to the Hetzner project
hcloud ssh-key create --name lvq --public-key-from-file ~/.ssh/id_ed25519.pub
```

## 2. Private network, firewall, servers

**Why 10.100.0.0/16 and a custom swarm pool:** Docker's default swarm address pool is 10.0.0.0/8 (allocated as /24s from 10.0.0.0/24 upward — the ingress network takes 10.0.0.0/24 first). Picking 10.100.1.0/24 for Hetzner and explicitly pinning the swarm pool to 10.201.0.0/16 in Step 6 makes the two provably disjoint.

```bash
hcloud network create --name dokploy-net --ip-range 10.100.0.0/16
hcloud network add-subnet dokploy-net --type cloud --network-zone eu-central --ip-range 10.100.1.0/24
```

```bash
hcloud firewall create --name dokploy-fw

# Public SSH (your choice: SSH stays publicly reachable)
hcloud firewall add-rule dokploy-fw --direction in --protocol tcp --port 22 \
  --source-ips 0.0.0.0/0 --source-ips ::/0 --description "public ssh"

# Tailscale direct connections (optional but improves p2p; Tailscale also works
# via outbound-only hole punching / DERP relays since the firewall is stateful)
hcloud firewall add-rule dokploy-fw --direction in --protocol udp --port 41641 \
  --source-ips 0.0.0.0/0 --source-ips ::/0 --description "tailscale wireguard"

hcloud firewall add-rule dokploy-fw --direction in --protocol icmp \
  --source-ips 0.0.0.0/0 --source-ips ::/0 --description "icmp"
```

Deliberately **not** opened publicly: 80, 443, 3000, 8080, 2377, 7946, 4789. The LB reaches the nodes over the private network (unfiltered by the Cloud Firewall); the UI travels over Tailscale.

```bash
# Prepare the cloud-init bootstrap first — see §2-B for the file and the
# Tailscale prerequisites, then inject the auth key:
export TS_AUTHKEY='tskey-auth-…'           # reusable, tagged tag:dokploy, short expiry
sed "s|__TS_AUTHKEY__|$TS_AUTHKEY|" cloud-init-dokploy-node.yaml > /tmp/ci.yaml

# Managers get the label the LB will target. cx32 = 4 vCPU / 8 GB (managers run
# Dokploy + builds); cx22 is fine for workers. Adjust types/location to taste.
for n in 1 2 3; do
  hcloud server create --name dokploy-mgr-$n --type cx32 --image fedora-44 \
    --location fsn1 --ssh-key lvq --network dokploy-net --firewall dokploy-fw \
    --user-data-from-file /tmp/ci.yaml \
    --label cluster=dokploy --label role=manager
done
for n in 1 2 3; do
  hcloud server create --name dokploy-wrk-$n --type cx22 --image fedora-44 \
    --location fsn1 --ssh-key lvq --network dokploy-net --firewall dokploy-fw \
    --user-data-from-file /tmp/ci.yaml \
    --label cluster=dokploy --label role=worker
done
rm /tmp/ci.yaml
```

Omit `--user-data-from-file` to provision bare servers and follow §3/§5/§6 by hand instead. Hetzner injects the `--ssh-key` alongside custom user-data, so public-key root SSH works either way.

Capture the DHCP-assigned private IPs — **do not assume ordering**:

```bash
for s in dokploy-mgr-1 dokploy-mgr-2 dokploy-mgr-3 dokploy-wrk-1 dokploy-wrk-2 dokploy-wrk-3; do
  printf '%-14s public=%-15s private=%s\n' "$s" \
    "$(hcloud server describe $s -o json | jq -r '.public_net.ipv4.ip')" \
    "$(hcloud server describe $s -o json | jq -r '.private_net[0].ip')"
done
```

The rest of the tutorial uses `10.100.1.2` as mgr-1's private IP purely as an example — substitute your captured values. Hetzner images log you in as `root`.

## 2-B. Cloud-init bootstrap (automates §3, §5, §6 + unattended updates)

One role-agnostic user-data file, identical for all six nodes. It deliberately stops at the point where nodes stop being interchangeable — swarm init/join, Dokploy, and Traefik are sequential, node-specific operations and stay manual.

**Tailscale prerequisites — do these once in the admin console before creating servers.** Unattended joining needs an auth key, and the correct kind is a **tagged** one, for two documented reasons: node keys of tagged devices never expire (untagged, user-owned devices need browser re-auth every 180 days — six times over), and the default SSH policy (`dst: autogroup:self`) does not cover tagged devices, so Tailscale SSH would deny without an explicit rule.

1. Access Controls — add the tag and an SSH rule (keep your existing rules alongside):

```json
"tagOwners": {
  "tag:dokploy": ["autogroup:admin"]
},
"ssh": [
  { "action": "accept", "src": ["autogroup:member"], "dst": ["tag:dokploy"], "users": ["root"] }
]
```

`action: accept` grants sessions without per-session browser re-auth; use `check` instead if you want that friction on production servers.

2. Settings → Keys → *Generate auth key*: **Reusable** (one key, six nodes), **Tags: `tag:dokploy`**, expiry as short as the console allows (1 day is plenty), **not** Ephemeral (ephemeral devices are removed when they go offline — wrong for servers). Pre-approved if your tailnet uses device approval.

3. Secret hygiene, stated plainly: the key ends up in Hetzner user-data (readable from each node's metadata endpoint and by anyone with project access) and under `/var/lib/cloud/` on the nodes. A short expiry bounds that, and you should additionally **revoke the key in the admin console the moment all six nodes appear** — revocation doesn't affect already-joined devices.

**The file — save the block below verbatim as `cloud-init-dokploy-node.yaml`** (the §2 provisioning loop injects the auth key into it and passes it to `hcloud server create`):

```yaml
#cloud-config
# Dokploy node bootstrap — Hetzner Cloud, Fedora 44 (x86)
# Automates tutorial steps 3 (base OS), 5 (Tailscale+SSH), 6 (Docker CE),
# and configures unattended SECURITY updates via dnf5-automatic.
# Role-specific steps stay manual by design: swarm init/join (§7, §9),
# Dokploy (§8), Traefik (§9-A) — they are sequential and node-specific.
#
# Before use: inject the Tailscale auth key (see tutorial §2-B):
#   sed "s|__TS_AUTHKEY__|$TS_AUTHKEY|" cloud-init-dokploy-node.yaml > /tmp/ci.yaml

package_update: true
package_upgrade: true          # full `dnf upgrade` on first boot (was manual step 3)

write_files:
  # Unattended-update policy. Deliberately conservative for a Swarm node:
  #  - security advisories only
  #  - never reboot unattended (3-manager quorum; timers fire within the same
  #    randomized hour on every node)
  #  - never touch the container engine unattended: a dockerd restart kills
  #    every task on the node, and live-restore does not apply in swarm mode
  #  - kernel changes only during deliberate §13 drain/reboot maintenance
  # Engine and kernel updates therefore follow the manual §13 procedure.
  - path: /etc/dnf/automatic.conf
    permissions: '0644'
    content: |
      [commands]
      upgrade_type = security
      apply_updates = yes
      reboot = never

      [emitters]
      emit_via = stdio

      [base]
      # Overrides dnf config for AUTOMATIC runs only —
      # a manual `dnf upgrade` still sees these packages.
      excludepkgs = docker-ce* docker-ce-cli containerd.io* docker-buildx-plugin docker-compose-plugin kernel*

runcmd:
  - set -e
  # Vendor repositories — identical sources to manual steps 5/6
  - curl -fsSL -o /etc/yum.repos.d/docker-ce.repo https://download.docker.com/linux/fedora/docker-ce.repo
  - curl -fsSL -o /etc/yum.repos.d/tailscale.repo https://pkgs.tailscale.com/stable/fedora/tailscale.repo
  # Latest Docker CE at provision time + Tailscale + the DNF5 automatic plugin
  - dnf -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin tailscale dnf5-plugin-automatic
  - systemctl enable --now docker
  - systemctl enable --now tailscaled
  # Unattended tailnet join, Tailscale SSH enabled. Requires a REUSABLE,
  # TAGGED (tag:dokploy) auth key — see §2-B for the ACL prerequisites.
  # Device name defaults to the hostname Hetzner set from the server name.
  - tailscale up --ssh --authkey=__TS_AUTHKEY__
  - systemctl enable --now dnf5-automatic.timer
  # Sentinel: the reboot below only fires if everything above succeeded
  - touch /run/bootstrap-ok

# One clean reboot at the end — first-boot package_upgrade almost certainly
# staged a kernel. On failure the node stays up for debugging (public SSH
# works; see /var/log/cloud-init-output.log).
power_state:
  mode: reboot
  condition: test -f /run/bootstrap-ok
  message: "bootstrap complete, rebooting onto updated kernel"
```

**Design decisions worth understanding, not just copying:**

*Repos via `runcmd`, not the `packages:` module* — cloud-init installs `packages:` before `runcmd` executes, i.e. before the Docker/Tailscale repos exist. Fetching the vendors' own `.repo` files also beats embedding their contents, which I could not fully verify (GPG-check flags change).

*"Latest Docker", precisely:* latest at provision time from Docker's repo — and then deliberately **pinned out of unattended updates**. An engine update restarts `dockerd`, which kills every task on the node (live-restore does not apply in swarm mode), and `dnf5-automatic.timer` fires within the same randomized hour on all six nodes — an unattended engine update night is a cluster-wide restart. Engine and kernel updates go through the §13 procedure: drain, update, reboot, reactivate, one node at a time. Everything else with a security advisory patches itself.

*`reboot = never`* — same quorum logic: three managers rebooting inside one randomized-delay window is a self-inflicted outage. The trade-off is explicit: a kernel CVE requires you to act (§13), it will not self-apply.

*Conditional final reboot* — the `power_state` block only fires if the sentinel exists, i.e. every `runcmd` step succeeded (`set -e`). A failed bootstrap leaves the node up and reachable over public SSH for debugging: `cat /var/log/cloud-init-output.log`.

**Verification** (first boot takes a few minutes — full upgrade, then reboot; watch nodes appear in the Tailscale admin console as they join):

```bash
for s in dokploy-mgr-1 dokploy-mgr-2 dokploy-mgr-3 dokploy-wrk-1 dokploy-wrk-2 dokploy-wrk-3; do
  ssh root@$s 'printf "%s: " "$(hostname -s)"; cloud-init status; docker --version; \
    tailscale status --self --peers=false; systemctl is-enabled dnf5-automatic.timer'
done
```

Then run the private-NIC/MTU check from §3 — it applies regardless of provisioning path.

## 3. Base OS — manual path (automated by §2-B; the NIC check below applies either way)

```bash
ssh root@<public-ip>
dnf -y update && systemctl reboot   # new kernel likely on a fresh image
```

Reconnect, then confirm the private NIC (typically `enp7s0`) exists with MTU 1450:

```bash
ip -4 addr show
# expect: enp7s0 ... mtu 1450 ... inet 10.100.1.x/32
```

If the private interface has no IP, cloud-init's network auto-configuration didn't run — stop and fix that first (Verified item 1 says fedora-44 supports it; a `cloud-init clean && reboot` normally recovers a one-off failure).

## 4. Host firewall — firewalld: deliberately absent (ALL nodes)

Verified by you: the Hetzner Fedora 44 image ships **without** firewalld (consistent with Fedora Cloud Base being minimal). Recommendation: **leave it that way** in this architecture. The reasoning, stated fully:

**What already protects the host.** The Hetzner Cloud Firewall is default-deny, stateful, and enforced *outside* the host — sshd, tailscaled, dockerd and the Swarm ports are unreachable from the public internet regardless of anything on the node. The private network carries only your six nodes and the LB. The tailnet interface is governed by Tailscale ACLs, which is the correct control for that path.

**What firewalld would genuinely add.** Two things: insurance against Cloud-Firewall misconfiguration (a detached firewall, a fat-fingered rule), and port-scoping of lateral movement between nodes on the private network (a compromised node could then only reach 2377/7946/4789/80/443/8080 on its peers, not every host-bound service).

**What it would NOT add — the commonly overestimated part.** (1) Docker installs its own nftables chains; ports published by containers largely bypass firewalld's INPUT policy, so it does not meaningfully gate container-published services. (2) Container↔container traffic on overlay networks is VXLAN-encapsulated — a host firewall sees only udp/4789 and cannot segment application traffic at all. (3) It cannot protect the Docker socket or anything else a container reaches via bind mounts.

**What it costs.** The firewalld↔Docker interplay is a known source of subtle breakage: a `firewall-cmd --reload` can clobber Docker's chains until the Docker daemon reinstalls them, and zone/interface drift produces hard-to-diagnose connectivity loss. On a running swarm that is real operational risk purchased for the thin margin above.

Net: skip it. If you have a hard requirement for a host firewall anyway (compliance posture, insurance against control-plane mistakes), install it **before Step 7** — retrofitting onto a live swarm invites exactly the breakage described:

```bash
dnf -y install firewalld && systemctl enable --now firewalld
firewall-cmd --permanent --new-zone=hcloud-priv
firewall-cmd --permanent --zone=hcloud-priv --add-source=10.100.1.0/24
firewall-cmd --permanent --zone=hcloud-priv \
  --add-port=2377/tcp --add-port=7946/tcp --add-port=7946/udp --add-port=4789/udp \
  --add-port=80/tcp --add-port=443/tcp --add-port=8080/tcp
firewall-cmd --reload
systemctl restart docker   # let Docker reinstall its chains after any reload
```

## 5. Tailscale with SSH — manual path (automated by §2-B)

```bash
dnf -y install dnf-plugins-core
dnf config-manager addrepo --from-repofile=https://pkgs.tailscale.com/stable/fedora/tailscale.repo
dnf -y install tailscale
systemctl enable --now tailscaled
tailscale up --ssh --hostname="$(hostname -s)"
```

Each `tailscale up` prints an auth URL — open it, approve the node. Verify with `tailscale status`.

Facts worth being precise about:

* `--ssh` enables **Tailscale SSH**: tailscaled answers SSH on the node's tailnet address. It does **not** touch the public `sshd` on port 22 — both paths coexist, matching your choice (UI Tailscale-only, SSH also public).
* Access is governed by your tailnet policy's `ssh` section. The **default** policy allows members to reach **their own** devices with `action: check` (periodic browser re-auth). Since you authenticate each node interactively as yourself, `ssh root@dokploy-mgr-1` from your workstation works out of the box. If you later switch to auth keys + tags, you must add an explicit `ssh` rule for the tag or Tailscale SSH will deny.
* Test now: `ssh root@dokploy-mgr-1` from your workstation (MagicDNS name).

## 6. Docker CE — manual path (automated by §2-B)

```bash
dnf -y install dnf-plugins-core
dnf config-manager addrepo --from-repofile=https://download.docker.com/linux/fedora/docker-ce.repo
dnf -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
systemctl enable --now docker
docker run --rm hello-world
```

SELinux stays **Enforcing** on the host. Docker CE does not enable SELinux confinement for containers by default (`selinux-enabled` is off in the daemon), which is precisely why Dokploy's `docker.sock` bind mounts work unmodified on Fedora. Trade-off stated plainly: container processes are not SELinux-confined; do **not** set `"selinux-enabled": true` in `daemon.json` — it will break Dokploy.

## 7. Swarm init with correct MTU (mgr-1 ONLY)

This is the step the one-liner installer cannot do for you, and the reason we install manually (Verified items 3 and 6).

```bash
PRIVATE_IP=10.100.1.2   # <-- mgr-1's actual private IP from Step 2

docker swarm init \
  --advertise-addr "$PRIVATE_IP" \
  --data-path-addr "$PRIVATE_IP" \
  --default-addr-pool 10.201.0.0/16 \
  --default-addr-pool-mask-length 24
```

`--data-path-addr` pins VXLAN traffic to the private NIC; the custom pool guarantees overlay subnets never collide with 10.100.1.0/24. The /16 pool with /24 masks allows 256 overlay networks — plenty here.

Recreate the ingress network with MTU 1400 **before any service exists**:

```bash
docker network rm ingress        # answer y
sleep 3
docker network create --driver overlay --ingress \
  --opt com.docker.network.driver.mtu=1400 ingress
```

(Our core services publish in host mode and don't use ingress, but any app that later publishes a port via the routing mesh will — fixing it now costs nothing; fixing it later requires removing every published service first.)

## 8. Install Dokploy (mgr-1 ONLY)

These are Dokploy's official "Existing Docker Swarm" manual steps, with two deliberate deviations, each flagged:

* **MTU:** `dokploy-network` is created with MTU 1400 (Verified item 6 — this exact omission is Dokploy issue #3446, and retrofitting the network after install breaks Dokploy's core services).
* **Placement:** the official snippet constrains `dokploy` and `dokploy-postgres` to `node.role==manager`. With **three** managers that is a footgun: a reschedule to another manager would silently start Postgres on an **empty local volume**. We pin both to mgr-1 by hostname. Consequence stated honestly in §13: the Dokploy control plane is not HA; your apps and ingress are.

```bash
# 1. Overlay network (deviation: MTU)
docker network create --driver overlay --attachable \
  --opt com.docker.network.driver.mtu=1400 dokploy-network

# 2. Config directory
mkdir -p /etc/dokploy
chmod 777 /etc/dokploy

# 3. Secrets
openssl rand -base64 32 | tr -d "=+/" | cut -c1-32 | docker secret create dokploy_postgres_password -
openssl rand -hex 32 | docker secret create dokploy_auth_secret -

# 4. Postgres (deviation: hostname pin)
docker service create \
  --name dokploy-postgres \
  --constraint 'node.hostname == dokploy-mgr-1' \
  --network dokploy-network \
  --env POSTGRES_USER=dokploy \
  --env POSTGRES_DB=dokploy \
  --secret source=dokploy_postgres_password,target=/run/secrets/postgres_password \
  --env POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password \
  --mount type=volume,source=dokploy-postgres,target=/var/lib/postgresql/data \
  postgres:16

# 5. Dokploy (deviation: hostname pin)
docker service create \
  --name dokploy \
  --replicas 1 \
  --network dokploy-network \
  --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \
  --mount type=bind,source=/etc/dokploy,target=/etc/dokploy \
  --mount type=volume,source=dokploy,target=/root/.docker \
  --secret source=dokploy_postgres_password,target=/run/secrets/postgres_password \
  --secret source=dokploy_auth_secret,target=/run/secrets/dokploy_auth_secret \
  --publish published=3000,target=3000,mode=host \
  --update-parallelism 1 \
  --update-order stop-first \
  --constraint 'node.hostname == dokploy-mgr-1' \
  -e POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password \
  -e BETTER_AUTH_SECRET_FILE=/run/secrets/dokploy_auth_secret \
  dokploy/dokploy:latest
```

Do **not** run the official step 6 (`docker run … dokploy-traefik`) — §9 replaces it with the swarm-service variant. Wait for Dokploy to generate the Traefik config, and make sure no stray container appeared:

```bash
sleep 20
ls -l /etc/dokploy/traefik/traefik.yml /etc/dokploy/traefik/dynamic/
docker ps --filter name=dokploy-traefik   # must be empty; `docker rm -f dokploy-traefik` if not
```

**First login (Tailscale-only):** port 3000 is published in **host mode on mgr-1 only** and the Cloud Firewall blocks it publicly. From your workstation on the tailnet, open:

```
http://dokploy-mgr-1.<your-tailnet>.ts.net:3000
```

Create the admin account. Under Web Server settings, do **not** assign a public domain to the panel — that would route it through Traefik and undo the Tailscale-only exposure.

## 9. Join the remaining nodes

On mgr-1:

```bash
docker swarm join-token manager   # for mgr-2, mgr-3
docker swarm join-token worker    # for wrk-1..3
```

On **each** joining node, using **that node's own** private IP:

```bash
MY_PRIVATE_IP=10.100.1.x
docker swarm join \
  --advertise-addr "$MY_PRIVATE_IP" \
  --data-path-addr "$MY_PRIVATE_IP" \
  --token <TOKEN-FROM-ABOVE> 10.100.1.2:2377
```

Verify on any manager — all six `Ready`, three `Leader/Reachable`:

```bash
docker node ls
```

Three managers = Raft quorum 2 → the cluster tolerates the loss of exactly one manager. Dokploy's UI Cluster page will show these nodes automatically (its "Add Node" button wraps the same join-token flow; it stays locked until §11's registry is configured, but plain `docker swarm join` is equivalent and fine).

## 9-A. Traefik as a global service on the managers (PROXY protocol, /ping, DNS-01)

Why this shape, precisely:

* **Global + `node.role==manager`:** Traefik's swarm provider reads services from the local `docker.sock`, and only a **manager** socket can list swarm services. Workers therefore cannot run this Traefik; they receive traffic from manager-Traefik over `dokploy-network`.
* **Host-mode ports:** Dokploy's own service mode uses host publishing to preserve client IPs; it also makes the :8080 health check reflect *that node's* Traefik instead of the routing mesh answering for a dead one.
* **PROXY protocol:** the LB is TCP passthrough, so without it every request would appear to come from the LB's private IP.
* **DNS-01 instead of the default HTTP-01:** each Traefik instance keeps its own node-local `acme.json`. Behind a multi-target LB, an HTTP-01 challenge lands on a random instance — only the one that requested it can answer, so issuance fails ~2/3 of the time. DNS-01 proves control via a temporary `_acme-challenge` TXT record instead: Traefik's embedded ACME client creates and deletes that record itself through the Cloudflare API, so each instance completes challenges independently — Dokploy plays no part in issuance, and your A records are never touched. Cost, stated plainly: three instances each issue certificates for the same names — Let's Encrypt allows 5 identical certificates per week, fine for a stable domain set, tight if you churn domains rapidly.

**First: distribute the config.** The bind mounts must exist on mgr-2/mgr-3. From your **workstation** (Tailscale SSH makes this trivial):

```bash
rsync -a root@dokploy-mgr-1:/etc/dokploy/traefik/ ./traefik-conf/
for m in dokploy-mgr-2 dokploy-mgr-3; do
  ssh root@$m 'mkdir -p /etc/dokploy/traefik'
  rsync -a ./traefik-conf/ root@$m:/etc/dokploy/traefik/
done
```

**Second: edit `/etc/dokploy/traefik/traefik.yml`** — on mgr-1 first, then re-run the rsync above. ⚠ Unverified item C: these are diffs against the documented default; read your generated file before applying.

```yaml
entryPoints:
  web:
    address: ':80'
    proxyProtocol:                 # ADD
      trustedIPs:
        - 10.100.1.0/24
  websecure:
    address: ':443'
    proxyProtocol:                 # ADD
      trustedIPs:
        - 10.100.1.0/24
    # http3: ...                   # REMOVE the http3 block if present —
    #                                Hetzner LBs cannot forward UDP (Verified item 8)
    http:
      tls:
        certResolver: letsencrypt  # keep as-is

ping: {}                           # ADD at top level — serves GET /ping on :8080
                                   # (the default config already has `api.insecure: true`,
                                   # so :8080 exists; ping attaches to it)

certificatesResolvers:
  letsencrypt:
    acme:
      email: you@example.com       # set your real email
      storage: /etc/dokploy/traefik/dynamic/acme.json
      dnsChallenge:                # REPLACES the default `httpChallenge:` block
        provider: cloudflare
        resolvers:
          - '1.1.1.1:53'
          - '8.8.8.8:53'
```

Exposure note, since `api.insecure: true` ships in Dokploy's default: publishing :8080 in host mode exposes the Traefik API/dashboard *and* /ping to the private network only (nodes + LB) and to the tailnet. The Cloud Firewall keeps it off the public internet because we never opened 8080.

**Third: create the service** (on mgr-1). The credential Traefik's ACME client uses for the Cloudflare API goes in as an env var. Create the token at Cloudflare → My Profile → API Tokens → *Create Custom Token* with permissions **Zone → Zone → Read** and **Zone → DNS → Edit**, scoped to your zone, and pass it as `CF_DNS_API_TOKEN`. (Any other lego-supported DNS provider works the same way — change `provider:` and the provider-specific env var per Traefik's ACME provider table.)

```bash
docker service create \
  --name dokploy-traefik \
  --mode global \
  --constraint 'node.role==manager' \
  --network dokploy-network \
  --env CF_DNS_API_TOKEN='<your-cloudflare-api-token>' \
  --mount type=bind,source=/etc/dokploy/traefik/traefik.yml,target=/etc/traefik/traefik.yml \
  --mount type=bind,source=/etc/dokploy/traefik/dynamic,target=/etc/dokploy/traefik/dynamic \
  --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock,readonly \
  --publish mode=host,published=80,target=80 \
  --publish mode=host,published=443,target=443 \
  --publish mode=host,published=8080,target=8080 \
  traefik:v3.6.7
```

(§14's script hardens the token handling: it stores the Cloudflare token as a Docker secret and points lego at it via `CF_DNS_API_TOKEN_FILE` — lego's `_FILE` convention — instead of a plaintext env var readable in `docker service inspect`.)

`traefik:v3.6.7` is the version Dokploy's docs currently pair with — check Dokploy release notes before bumping it; incompatible Traefik majors are a documented source of 404s. Dokploy detects at runtime that `dokploy-traefik` is a *service* and will correctly reload it with `docker service update --force` when you change settings in the UI.

Verify one task per manager and a 200 on every node:

```bash
docker service ps dokploy-traefik
for ip in 10.100.1.2 10.100.1.3 10.100.1.4; do   # your three manager private IPs
  curl -s -o /dev/null -w "$ip -> %{http_code}\n" http://$ip:8080/ping
done
```

## 9-B. Stock single-node variant (if you'd rather not adapt Traefik)

Fully Dokploy-stock alternative: keep the default HTTP-01 resolver, skip the config rsync, and create the service with `--constraint 'node.hostname == dokploy-mgr-1'` and `--replicas 1` instead of `--mode global`. Then in §10 target **only mgr-1** (`hcloud load-balancer add-target dokploy-lb --server dokploy-mgr-1 --use-private-ip`). Everything else is identical, including PROXY protocol and /ping. Trade-off stated plainly: mgr-1 becomes the single ingress point — the LB then buys you a stable public IP and health-gated traffic, not ingress HA.

## 10. Hetzner Load Balancer

⚠ Unverified item B: confirm flag names with `hcloud load-balancer add-service --help` on your CLI version before running.

```bash
hcloud load-balancer create --name dokploy-lb --type lb11 --location fsn1
hcloud load-balancer attach-to-network dokploy-lb --network dokploy-net

# Targets: the three managers (where Traefik runs), reached via private IPs
hcloud load-balancer add-target dokploy-lb --label-selector role=manager --use-private-ip

# TCP passthrough with PROXY protocol; health = your required
# "HTTP on :8080 -> 200", implemented as Traefik's /ping
hcloud load-balancer add-service dokploy-lb \
  --protocol tcp --listen-port 80 --destination-port 80 --proxy-protocol \
  --health-check-protocol http --health-check-port 8080 --health-check-http-path /ping \
  --health-check-interval 5s --health-check-timeout 3s --health-check-retries 3

hcloud load-balancer add-service dokploy-lb \
  --protocol tcp --listen-port 443 --destination-port 443 --proxy-protocol \
  --health-check-protocol http --health-check-port 8080 --health-check-http-path /ping \
  --health-check-interval 5s --health-check-timeout 3s --health-check-retries 3
```

Two precision notes:

* `--proxy-protocol` on the LB and `proxyProtocol.trustedIPs` in Traefik (§9-A) are a **pair**. Enable both or neither; one without the other breaks all traffic on that port.
* Health checks are plain HTTP against :8080, which is exactly why the ping entrypoint must **not** have proxyProtocol on it — and it doesn't.

Wait ~30 s, then confirm all three targets are `healthy`:

```bash
hcloud load-balancer describe dokploy-lb
LB_IP=$(hcloud load-balancer describe dokploy-lb -o json | jq -r '.public_net.ipv4.ip')
echo "$LB_IP"
```

**DNS:** point every app domain (A record, plus AAAA for the LB's IPv6 if you use it) at `$LB_IP`. The records will run **proxied (orange-cloud)** so Cloudflare's edge shields the origin — but follow the exact ordering in §10-B (SSL mode first, allowlist last), otherwise you either create a redirect loop or lock real users out mid-migration. DNS-01 issuance is unaffected by proxy status; the records must live in the zone your API token controls.

## 10-B. Cloudflare proxy in front — origin shielding

Final traffic path: client → **Cloudflare edge** (client-facing TLS, IP hidden from DNS) → **Hetzner LB** (TCP passthrough + PROXY protocol, carries the CF edge IP) → **Traefik** (terminates TLS with the LE certs from §9-A, rejects anything that didn't come through Cloudflare).

Where enforcement lives, and why (Verified item 9): the component holding the public 80/443 is the LB, and Hetzner firewalls attach to servers only — there is no source-IP filter you can put in front of an LB. So "deny everything but Cloudflare" is enforced one hop inward, at Traefik, using the PROXY-protocol source address (which *is* the Cloudflare edge IP). Do the steps **in this order**:

**1. Cloudflare zone → SSL/TLS → encryption mode `Full (strict)`.** Do this before anything else. The default/`Flexible` mode makes Cloudflare fetch the origin over plain HTTP:80, which meets Dokploy's `redirect-to-https` middleware and produces an infinite redirect loop — the classic Cloudflare+Traefik failure. `Full (strict)` requires a publicly trusted origin certificate covering the hostname: the DNS-01 Let's Encrypt certs from §9-A already satisfy it — no Cloudflare Origin CA cert needed, nothing changes on the Traefik side. Optionally enable *Always Use HTTPS* at the edge.

**2. Flip the app records to Proxied (orange cloud).** The site now serves through Cloudflare; the origin is still directly reachable — fixed in step 3. DNS-01 renewals are unaffected (`_acme-challenge` TXT records are DNS-level, never proxied).

**3. Generate the Cloudflare allowlist** (on mgr-1). Cloudflare publishes its edge ranges; don't hardcode them — generate from the live list:

```bash
cat > /usr/local/bin/update-cf-allowlist.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail
RANGES="$(curl -fsS https://www.cloudflare.com/ips-v4) $(curl -fsS https://www.cloudflare.com/ips-v6)"
{
  echo "http:"
  echo "  middlewares:"
  echo "    cf-only:"
  echo "      ipAllowList:"
  echo "        sourceRange:"
  for r in $RANGES; do echo "          - $r"; done
} > /etc/dokploy/traefik/dynamic/cf-allowlist.yml
echo "cf-allowlist.yml written ($(wc -w <<< "$RANGES") ranges)."
echo "--- paste the same list under forwardedHeaders.trustedIPs in traefik.yml: ---"
for r in $RANGES; do echo "        - $r"; done
EOF
chmod +x /usr/local/bin/update-cf-allowlist.sh
/usr/local/bin/update-cf-allowlist.sh
```

**4. Add two things to *both* the `web` and `websecure` entrypoints in `traefik.yml`:**

```yaml
entryPoints:
  websecure:                       # and identically under `web:`
    address: ':443'
    proxyProtocol:                 # already there from §9-A
      trustedIPs:
        - 10.100.1.0/24
    forwardedHeaders:              # ADD — trust Cloudflare's X-Forwarded-For,
      trustedIPs:                  #        so logs/upstreams see the real visitor IP
        - 173.245.48.0/20          # <- paste the full list the script printed
        # ... rest of the printed ranges ...
    http:
      middlewares:                 # ADD — reject anything not arriving via Cloudflare
        - cf-only@file
      tls:
        certResolver: letsencrypt
```

Why both keys: PROXY protocol makes Traefik's remote address the **Cloudflare edge IP** — that is what `cf-only` matches against. `forwardedHeaders.trustedIPs` then tells Traefik to believe the `X-Forwarded-For` Cloudflare appends, so the real visitor IP flows into access logs and upstream headers. Apps should read `CF-Connecting-IP` (or the first XFF entry).

**5. Distribute and reload:** rsync `/etc/dokploy/traefik/` to mgr-2/3 with `--exclude acme.json` (§9-A pattern), then `docker service update --force dokploy-traefik`. The allowlist activates on all three nodes — this is the moment direct access dies, which is why it comes after step 2.

**6. Verify both directions:**

```bash
curl -s -o /dev/null -w '%{http_code}\n' https://test.<yourdomain>                # 200, via Cloudflare
curl -sk -o /dev/null -w '%{http_code}\n' \
  --resolve test.<yourdomain>:443:$LB_IP https://test.<yourdomain>                 # 403, direct hit rejected
```

**Honest scope of the shield — three statements:**

*What it does:* the LB IP no longer appears in DNS, and the origin serves nothing to non-Cloudflare clients — direct hits get 403 (known host) or 404 (unknown host). One caveat if these hostnames ever pointed at this LB **unproxied**: DNS-history archives (SecurityTrails and friends) have the IP forever. Cheap fix: create a fresh LB (new public IP), retarget, delete the old one — the §10 commands rerun in minutes.

*What it does not do:* an active prober who already **suspects** the IP can still complete a TLS handshake with a guessed SNI and receive the real certificate, confirming the mapping — an HTTP-layer allowlist can't prevent that, and mTLS (Cloudflare *Authenticated Origin Pulls*) only moves the rejection inside the handshake; the certificate is still served before client auth completes. If origin-confirmation resistance ever becomes a hard requirement, the endgame is **Cloudflare Tunnel** (outbound-only connectors, zero inbound listeners — Dokploy has an official guide) — a different ingress design that replaces the LB entirely.

*Residual nuance of IP-based allowlisting:* "came from Cloudflare" ≠ "came from **your** zone" — another Cloudflare customer could point their own proxied hostname at your LB IP and pass the IP check. In practice Traefik's Host-based routing 404s them (their hostname matches none of your routers, and Host-header override is a Cloudflare Enterprise feature). Per-hostname Authenticated Origin Pulls is the cryptographic upgrade if you ever want it.

**Maintenance:** Cloudflare's ranges change rarely. When they do: re-run the script (the middleware file hot-reloads via Traefik's file provider), re-paste `forwardedHeaders.trustedIPs` (static config), rsync, `docker service update --force dokploy-traefik`.

## 11. Registry (unlocks Dokploy's Cluster features)

Dokploy needs a registry to push images **it builds** so the other nodes can pull them (prebuilt public images bypass this). Any OCI registry works identically — Dokploy just needs credentials that can push and the nodes need to pull. With the GitLab registry:

1. GitLab → Access Tokens (a personal access token, or a group/project **deploy token**) with scopes `read_registry` + `write_registry`.
2. Dokploy UI → **Registry** → Add: registry URL `registry.gitlab.com`, username = your GitLab username (for deploy tokens, the generated token username), password = the token.
3. The **Cluster** section unlocks once the registry test passes.

## 12. End-to-end test

Deploy a placement-aware echo service and verify every design property at once:

1. Dokploy UI → new Project → new **Application** → provider *Docker image* → `traefik/whoami` (public image — no registry involvement, isolates the ingress test).
2. Advanced → Swarm settings → replicas **3** (or a placement preference across workers).
3. Domains → add `test.<yourdomain>` → container port **80** → HTTPS on, certificate *Let's Encrypt* → Deploy.

```bash
# From your workstation (public internet path: DNS -> LB -> Traefik -> app):
curl -s https://test.<yourdomain> | grep -E 'Hostname|X-Forwarded-For'
curl -s https://test.<yourdomain> | grep Hostname   # repeat: hostname varies across replicas
```

What each line proves: valid HTTPS = DNS-01 issuance works; `X-Forwarded-For` showing **your real IP** = the Cloudflare-header + PROXY-protocol chain works (§10-B); varying `Hostname` = cross-node overlay routing works, i.e. the MTU fix holds. For an explicit MTU probe between two containers on `dokploy-network`: `ping -c1 -M do -s 1372 <other-container-ip>` must succeed (1372 = 1400 − 28 ICMP/IP overhead) and `-s 1373` must fail.

Also verify the failover behavior: `ssh root@dokploy-mgr-2 'docker ps -q --filter name=dokploy-traefik | xargs docker stop'` → within ~15 s `hcloud load-balancer describe dokploy-lb` shows that target `unhealthy` while traffic keeps flowing; Swarm restarts the task and it returns to `healthy`.

## 13. Operational truths, stated plainly

**What is HA:** deployed applications (multi-replica across nodes), ingress (three Traefik instances behind the LB, health-gated), and the Swarm control plane (tolerates one manager loss).

**What is not HA:** the Dokploy panel and its Postgres, pinned to mgr-1 (§8). If mgr-1 dies, deployments/UI stop; already-running apps and ingress on mgr-2/3 keep serving. Recovery = restore mgr-1 or restore `/etc/dokploy` + the `dokploy-postgres` volume onto a new node with the same hostname. Back both up (Dokploy's UI has S3 backup destinations).

**Dynamic-file caveat (only if you use path middlewares):** plain host-based domains are pure swarm labels — consistent on all three Traefik instances with zero sync. Dokploy's *Internal Path / Strip Path* middlewares, however, are written as files under `/etc/dokploy/traefik/dynamic/` on **mgr-1 only**. If you use those features, re-run the §9-A rsync afterwards with `--exclude acme.json` (each node must keep its own certificate store). If you never use them, no sync is ever needed.

**Updating:** Dokploy — `curl -sSL https://dokploy.com/install.sh | sh -s update` is safe on this setup; the `update` argument only pulls the image and runs `docker service update`, it never touches the swarm (verified against the published script source). Traefik — `docker service update --image traefik:vX.Y.Z dokploy-traefik`, after checking Dokploy release notes for compatibility. Fedora — normal `dnf update`; drain first (`docker node update --availability drain <node>`), reboot, reactivate.

**Unattended patching:** §2-B configures dnf5-automatic for security-only updates with the container engine and kernel excluded — those two categories exist *only* via the drain procedure above, on purpose.

**Worker disk hygiene:** Dokploy does not garbage-collect images on non-panel nodes (documented). Add a cron on the workers: `docker system prune -af --filter "until=168h"`, or manage them as Dokploy "remote servers" with scheduled cleanups.

**HTTP/3:** not available through this stack — Hetzner LBs do not forward UDP. That is why §9-A removes the `http3` advertisement; clients silently use HTTP/2.

**Scaling ingress later:** more capacity = bigger LB type or more managers... with the caveat that manager counts should stay odd (3 → 5). Adding a manager automatically gains a Traefik task (global mode) — remember to rsync the config dir to it first and to check the LE duplicate-certificate math (5 identical certs/week vs. instance count).


## 14. Single-file automation — `dokploy-cluster.sh`

Everything above that is deterministic is scriptable — roughly 90% of the build. Save the block below verbatim as `dokploy-cluster.sh` (next to `cloud-init-dokploy-node.yaml` from §2-B), `chmod +x` it, and run the subcommands from your **workstation** in the pipeline order printed by running it without arguments. It uses `hcloud` for cloud resources and Tailscale-SSH MagicDNS names for everything on the nodes.

**The four manual interludes, and why they stay manual:** the Tailscale admin-console prep (ACL edit + tagged auth key — one-time, security-sensitive, §2-B), the Dokploy admin account + registry credentials (two minutes of UI; Dokploy has an API but I have not verified registry setup through it), and the Cloudflare zone steps (SSL mode + record flips — scriptable via the CF API if you want to hand this script a second credential; deliberately out of scope). Everything else is a subcommand.

**Properties worth knowing before trusting it:**

* *Guarded re-entrancy, not convergence.* Every subcommand checks for its resources and skips what exists, so re-running after a partial failure is safe — but it will not reconcile drift or changed config. If you outgrow that, the honest next step is Terraform (hcloud provider) + Ansible, not a bigger bash script.
* *IPs come from `hcloud`, never from remote guessing* — join/init commands receive each node's private IP from the API, eliminating a whole class of interface-detection bugs.
* *The traefik.yml edits are structural, not textual.* A python/PyYAML step targets keys (`entryPoints.web.proxyProtocol`, the `letsencrypt` resolver, …), which converts Unverified item C from a silent risk into a loud stop: if a future Dokploy release reshapes the file, the edit KeyErrors and the untouched original plus a `.orig` backup remain. Comments in the generated file don't survive the rewrite — accepted trade-off.
* *Secrets never touch argv.* `TS_AUTHKEY`/`CF_DNS_API_TOKEN`/`ACME_EMAIL` enter via environment; the CF token travels over ssh stdin into a Docker secret consumed through lego's `CF_DNS_API_TOKEN_FILE` convention (an upgrade over §9-A's inline env — noted there).
* *The §10-B ordering trap is encoded*: `cf-allowlist` prints the Full-strict/proxied precondition and pauses before acting, and is the same command you re-run when Cloudflare's ranges change.
* *Inherited caveats:* Unverified item B (hcloud LB flag names) applies to `cmd_lb` unchanged; `StrictHostKeyChecking=accept-new` is trust-on-first-use — tighten `SSH_OPTS` if your threat model says so.

```bash
#!/usr/bin/env bash
# dokploy-cluster.sh — single-file bootstrapper for the architecture in
# dokploy-hetzner-swarm-fedora44.md. Subcommands mirror tutorial sections.
#
# Pipeline (manual interludes marked ✋):
#   ✋ Tailscale admin console: tagOwners + ssh rule, tagged auth key   (§2-B)
#   TS_AUTHKEY=tskey-auth-… ./dokploy-cluster.sh provision              (§2)
#   ./dokploy-cluster.sh wait                                           (§2-B)
#   ./dokploy-cluster.sh swarm-init                                     (§7)
#   ./dokploy-cluster.sh dokploy-install                                (§8)
#   ✋ Dokploy UI via Tailscale :3000 — create admin account            (§8)
#   ./dokploy-cluster.sh join                                           (§9)
#   ACME_EMAIL=… CF_DNS_API_TOKEN=… ./dokploy-cluster.sh traefik        (§9-A)
#   ./dokploy-cluster.sh lb                                             (§10)
#   ✋ Cloudflare: SSL mode Full (strict), DNS records → LB IP, proxied (§10-B 1-2)
#   ./dokploy-cluster.sh cf-allowlist                                   (§10-B 3-5)
#   ✋ Dokploy UI: GitLab registry credentials                          (§11)
#   ./dokploy-cluster.sh verify                                         (§12)
#
# Guarantees, stated plainly: guarded re-entrancy (safe to re-run, existing
# resources are skipped), NOT convergence — this script will not reconcile
# drift or config changes. If you want managed end-state, that's Terraform
# (hcloud provider) + Ansible; this is the readable middle ground.
#
# Secrets policy: TS_AUTHKEY / CF_DNS_API_TOKEN enter via environment only.
# The CF token travels to mgr-1 over ssh stdin and lives as a Docker secret,
# consumed via lego's CF_DNS_API_TOKEN_FILE convention — never in argv,
# never in the Traefik service spec as plaintext.

set -euo pipefail

### ── Config: keep in sync with tutorial §2 ────────────────────────────────
LOCATION=fsn1
MGR_TYPE=cx32
WRK_TYPE=cx22
IMAGE=fedora-44
SSH_KEY_NAME=lvq
NET_NAME=dokploy-net   NET_RANGE=10.100.0.0/16
SUBNET=10.100.1.0/24   NET_ZONE=eu-central
FW_NAME=dokploy-fw
LB_NAME=dokploy-lb     LB_TYPE=lb11
CLOUD_INIT=cloud-init-dokploy-node.yaml
TRAEFIK_IMAGE=traefik:v3.6.7          # check Dokploy release notes before bumping
SWARM_POOL=10.201.0.0/16              # disjoint from SUBNET by construction
MANAGERS=(dokploy-mgr-1 dokploy-mgr-2 dokploy-mgr-3)
WORKERS=(dokploy-wrk-1 dokploy-wrk-2 dokploy-wrk-3)
ALL=("${MANAGERS[@]}" "${WORKERS[@]}")
PRIMARY=${MANAGERS[0]}
SSH_OPTS=(-o StrictHostKeyChecking=accept-new -o ConnectTimeout=8)  # TOFU; tighten if you prefer

### ── Helpers ───────────────────────────────────────────────────────────────
die()  { echo "ERROR: $*" >&2; exit 1; }
info() { echo "==> $*"; }
need() { command -v "$1" >/dev/null 2>&1 || die "missing dependency: $1"; }
run()  { local h=$1; shift; ssh "${SSH_OPTS[@]}" "root@$h" "$@"; }   # Tailscale MagicDNS names
priv_ip() { hcloud server describe "$1" -o json | jq -r '.private_net[0].ip'; }

### ── provision (§2): network, firewall, 6 servers with cloud-init ─────────
cmd_provision() {
  need hcloud; need jq
  [[ -n ${TS_AUTHKEY:-} ]] || die "export TS_AUTHKEY=tskey-auth-… (tagged, reusable — §2-B)"
  [[ -f $CLOUD_INIT ]] || die "$CLOUD_INIT not found next to this script (§2-B)"

  hcloud network describe "$NET_NAME" >/dev/null 2>&1 || {
    info "network $NET_NAME"
    hcloud network create --name "$NET_NAME" --ip-range "$NET_RANGE"
    hcloud network add-subnet "$NET_NAME" --type cloud --network-zone "$NET_ZONE" --ip-range "$SUBNET"
  }
  hcloud firewall describe "$FW_NAME" >/dev/null 2>&1 || {
    info "firewall $FW_NAME (public: 22/tcp, 41641/udp, icmp — nothing else, per §2)"
    hcloud firewall create --name "$FW_NAME"
    hcloud firewall add-rule "$FW_NAME" --direction in --protocol tcp  --port 22    --source-ips 0.0.0.0/0 --source-ips ::/0 --description "public ssh"
    hcloud firewall add-rule "$FW_NAME" --direction in --protocol udp  --port 41641 --source-ips 0.0.0.0/0 --source-ips ::/0 --description "tailscale"
    hcloud firewall add-rule "$FW_NAME" --direction in --protocol icmp              --source-ips 0.0.0.0/0 --source-ips ::/0 --description "icmp"
  }

  local tmp; tmp=$(mktemp); trap 'rm -f "$tmp"' RETURN
  sed "s|__TS_AUTHKEY__|$TS_AUTHKEY|" "$CLOUD_INIT" > "$tmp"

  local s
  for s in "${MANAGERS[@]}"; do
    hcloud server describe "$s" >/dev/null 2>&1 && { info "$s exists, skipping"; continue; }
    info "creating $s"
    hcloud server create --name "$s" --type "$MGR_TYPE" --image "$IMAGE" --location "$LOCATION" \
      --ssh-key "$SSH_KEY_NAME" --network "$NET_NAME" --firewall "$FW_NAME" \
      --user-data-from-file "$tmp" --label cluster=dokploy --label role=manager
  done
  for s in "${WORKERS[@]}"; do
    hcloud server describe "$s" >/dev/null 2>&1 && { info "$s exists, skipping"; continue; }
    info "creating $s"
    hcloud server create --name "$s" --type "$WRK_TYPE" --image "$IMAGE" --location "$LOCATION" \
      --ssh-key "$SSH_KEY_NAME" --network "$NET_NAME" --firewall "$FW_NAME" \
      --user-data-from-file "$tmp" --label cluster=dokploy --label role=worker
  done

  info "IP map:"
  for s in "${ALL[@]}"; do
    printf '  %-14s public=%-15s private=%s\n' "$s" \
      "$(hcloud server describe "$s" -o json | jq -r '.public_net.ipv4.ip')" "$(priv_ip "$s")"
  done
  info "Now revoke the auth key in the Tailscale console once all 6 nodes appear, then: $0 wait"
}

### ── wait (§2-B): block until cloud-init finished on every node ───────────
cmd_wait() {
  local s i
  for s in "${ALL[@]}"; do
    info "waiting for $s (bootstrap + reboot can take several minutes)"
    for i in $(seq 1 60); do
      if run "$s" 'cloud-init status 2>/dev/null' | grep -q 'status: done'; then
        echo "    $s ready"; continue 2
      fi
      sleep 10
    done
    die "$s not ready after 10 min — check /var/log/cloud-init-output.log on the node"
  done
}

### ── swarm-init (§7): init on PRIMARY + MTU-1400 ingress ──────────────────
cmd_swarm_init() {
  need hcloud; need jq
  local ip; ip=$(priv_ip "$PRIMARY")
  [[ $ip == 10.100.1.* ]] || die "unexpected private IP for $PRIMARY: $ip"
  if run "$PRIMARY" "docker info --format '{{.Swarm.LocalNodeState}}'" | grep -q '^active'; then
    info "swarm already active on $PRIMARY"; return 0
  fi
  run "$PRIMARY" "docker swarm init --advertise-addr $ip --data-path-addr $ip \
    --default-addr-pool $SWARM_POOL --default-addr-pool-mask-length 24"
  info "recreating ingress at MTU 1400 (§7 — Verified item 6)"
  run "$PRIMARY" "echo y | docker network rm ingress; sleep 3; \
    docker network create --driver overlay --ingress --opt com.docker.network.driver.mtu=1400 ingress"
}

### ── dokploy-install (§8): network, secrets, postgres, dokploy ────────────
cmd_dokploy_install() {
  if run "$PRIMARY" "docker service inspect dokploy >/dev/null 2>&1"; then
    info "dokploy service already exists"; return 0
  fi
  run "$PRIMARY" bash -s <<'REMOTE'
set -euo pipefail
docker network inspect dokploy-network >/dev/null 2>&1 || \
  docker network create --driver overlay --attachable \
    --opt com.docker.network.driver.mtu=1400 dokploy-network
mkdir -p /etc/dokploy && chmod 777 /etc/dokploy
docker secret inspect dokploy_postgres_password >/dev/null 2>&1 || \
  openssl rand -base64 32 | tr -d "=+/" | cut -c1-32 | docker secret create dokploy_postgres_password -
docker secret inspect dokploy_auth_secret >/dev/null 2>&1 || \
  openssl rand -hex 32 | docker secret create dokploy_auth_secret -

docker service create \
  --name dokploy-postgres \
  --constraint "node.hostname == $(hostname)" \
  --network dokploy-network \
  --env POSTGRES_USER=dokploy --env POSTGRES_DB=dokploy \
  --secret source=dokploy_postgres_password,target=/run/secrets/postgres_password \
  --env POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password \
  --mount type=volume,source=dokploy-postgres,target=/var/lib/postgresql/data \
  postgres:16

docker service create \
  --name dokploy --replicas 1 \
  --network dokploy-network \
  --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \
  --mount type=bind,source=/etc/dokploy,target=/etc/dokploy \
  --mount type=volume,source=dokploy,target=/root/.docker \
  --secret source=dokploy_postgres_password,target=/run/secrets/postgres_password \
  --secret source=dokploy_auth_secret,target=/run/secrets/dokploy_auth_secret \
  --publish published=3000,target=3000,mode=host \
  --update-parallelism 1 --update-order stop-first \
  --constraint "node.hostname == $(hostname)" \
  -e POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password \
  -e BETTER_AUTH_SECRET_FILE=/run/secrets/dokploy_auth_secret \
  dokploy/dokploy:latest

echo -n "waiting for Traefik config generation "
for i in $(seq 1 60); do
  [[ -f /etc/dokploy/traefik/traefik.yml ]] && { echo ok; break; }
  echo -n .; sleep 2
done
[[ -f /etc/dokploy/traefik/traefik.yml ]] || { echo "traefik.yml never appeared"; exit 1; }
if docker ps -q --filter name=dokploy-traefik | grep -q .; then
  echo "removing stray standalone traefik container (we deploy the service variant)"
  docker rm -f dokploy-traefik
fi
REMOTE
  info "Dokploy up. ✋ Create the admin account now: http://${PRIMARY}.<tailnet>.ts.net:3000, then: $0 join"
}

### ── join (§9): managers then workers, IPs supplied from hcloud ───────────
cmd_join() {
  need hcloud; need jq
  local p1 mtok wtok s ip
  p1=$(priv_ip "$PRIMARY")
  mtok=$(run "$PRIMARY" docker swarm join-token -q manager)
  wtok=$(run "$PRIMARY" docker swarm join-token -q worker)
  for s in "${MANAGERS[@]:1}" "${WORKERS[@]}"; do
    if run "$s" "docker info --format '{{.Swarm.LocalNodeState}}'" | grep -q '^active'; then
      info "$s already joined"; continue
    fi
    ip=$(priv_ip "$s")
    local tok=$wtok; [[ " ${MANAGERS[*]} " == *" $s "* ]] && tok=$mtok
    info "joining $s ($ip)"
    run "$s" "docker swarm join --advertise-addr $ip --data-path-addr $ip --token $tok ${p1}:2377"
  done
  run "$PRIMARY" docker node ls
}

### ── traefik (§9-A): config edits, distribution, global service ───────────
cmd_traefik() {
  need jq; need rsync; need hcloud
  [[ -n ${ACME_EMAIL:-} ]]        || die "export ACME_EMAIL=you@example.com"
  [[ -n ${CF_DNS_API_TOKEN:-} ]]  || die "export CF_DNS_API_TOKEN=… (Zone:Read + DNS:Edit — §9-A)"

  info "editing traefik.yml structurally (backup kept; key mismatch fails loud, file untouched)"
  run "$PRIMARY" "rpm -q python3-pyyaml >/dev/null 2>&1 || dnf -y install python3-pyyaml"
  run "$PRIMARY" "cp -n /etc/dokploy/traefik/traefik.yml /etc/dokploy/traefik/traefik.yml.orig; \
                  ACME_EMAIL='$ACME_EMAIL' python3 -" <<'PY'
import os, yaml
p = '/etc/dokploy/traefik/traefik.yml'
c = yaml.safe_load(open(p))
for ep in ('web', 'websecure'):
    c['entryPoints'][ep]['proxyProtocol'] = {'trustedIPs': ['10.100.1.0/24']}
c['entryPoints']['websecure'].pop('http3', None)   # no UDP through Hetzner LB (Verified item 8)
c['ping'] = {}                                     # GET /ping on :8080 (api.insecure already opens it)
acme = c['certificatesResolvers']['letsencrypt']['acme']
acme['email'] = os.environ['ACME_EMAIL']
acme.pop('httpChallenge', None)
acme['dnsChallenge'] = {'provider': 'cloudflare', 'resolvers': ['1.1.1.1:53', '8.8.8.8:53']}
yaml.safe_dump(c, open(p, 'w'), sort_keys=False)
print('traefik.yml rewritten')
PY

  info "storing Cloudflare token as a Docker secret (stdin only — never argv)"
  if ! run "$PRIMARY" "docker secret inspect cf_dns_token >/dev/null 2>&1"; then
    printf %s "$CF_DNS_API_TOKEN" | run "$PRIMARY" "docker secret create cf_dns_token -"
  fi

  _sync_traefik_conf

  if run "$PRIMARY" "docker service inspect dokploy-traefik >/dev/null 2>&1"; then
    info "dokploy-traefik service exists — forcing redeploy with current config"
    run "$PRIMARY" "docker service update --force dokploy-traefik" >/dev/null
  else
    run "$PRIMARY" "docker service create \
      --name dokploy-traefik --mode global --constraint 'node.role==manager' \
      --network dokploy-network \
      --secret source=cf_dns_token,target=cf_dns_token \
      --env CF_DNS_API_TOKEN_FILE=/run/secrets/cf_dns_token \
      --mount type=bind,source=/etc/dokploy/traefik/traefik.yml,target=/etc/traefik/traefik.yml \
      --mount type=bind,source=/etc/dokploy/traefik/dynamic,target=/etc/dokploy/traefik/dynamic \
      --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock,readonly \
      --publish mode=host,published=80,target=80 \
      --publish mode=host,published=443,target=443 \
      --publish mode=host,published=8080,target=8080 \
      $TRAEFIK_IMAGE"
  fi
  _ping_managers
}

_sync_traefik_conf() {   # workstation-mediated; acme.json stays per-node (§9-A)
  local t m; t=$(mktemp -d); trap 'rm -rf "$t"' RETURN
  rsync -a --exclude acme.json -e "ssh ${SSH_OPTS[*]}" "root@$PRIMARY:/etc/dokploy/traefik/" "$t/"
  for m in "${MANAGERS[@]:1}"; do
    run "$m" "mkdir -p /etc/dokploy/traefik/dynamic"
    rsync -a --exclude acme.json -e "ssh ${SSH_OPTS[*]}" "$t/" "root@$m:/etc/dokploy/traefik/"
  done
}

_ping_managers() {
  local m ip
  for m in "${MANAGERS[@]}"; do
    ip=$(priv_ip "$m")
    printf '  %s (%s) /ping -> ' "$m" "$ip"
    run "$PRIMARY" "curl -s -o /dev/null -w '%{http_code}' --max-time 5 http://$ip:8080/ping" || true
    echo
  done
}

### ── lb (§10): create, attach, target managers, 80/443 + /ping health ─────
cmd_lb() {
  need hcloud; need jq
  hcloud load-balancer describe "$LB_NAME" >/dev/null 2>&1 || {
    hcloud load-balancer create --name "$LB_NAME" --type "$LB_TYPE" --location "$LOCATION"
    hcloud load-balancer attach-to-network "$LB_NAME" --network "$NET_NAME"
    hcloud load-balancer add-target "$LB_NAME" --label-selector role=manager --use-private-ip
    local p
    for p in 80 443; do   # ⚠ Unverified item B: confirm flags via `hcloud load-balancer add-service --help`
      hcloud load-balancer add-service "$LB_NAME" \
        --protocol tcp --listen-port "$p" --destination-port "$p" --proxy-protocol \
        --health-check-protocol http --health-check-port 8080 --health-check-http-path /ping \
        --health-check-interval 5s --health-check-timeout 3s --health-check-retries 3
    done
  }
  info "LB public IPv4: $(hcloud load-balancer describe "$LB_NAME" -o json | jq -r '.public_net.ipv4.ip')"
  info "✋ Cloudflare next, IN ORDER (§10-B): SSL mode Full (strict) → records to the IP above, Proxied → then: $0 cf-allowlist"
}

### ── cf-allowlist (§10-B steps 3-5) ────────────────────────────────────────
cmd_cf_allowlist() {
  cat >&2 <<'WARN'
!! Ordering check (§10-B): run this ONLY after (1) zone SSL mode = Full (strict)
!! and (2) the app records are Proxied. Running it while records are DNS-only
!! serves 403 to your real users. Ctrl-C now if that isn't done.
WARN
  sleep 5
  run "$PRIMARY" "python3 -" <<'PY'
import urllib.request, yaml
get = lambda u: urllib.request.urlopen(u, timeout=10).read().decode().split()
ranges = get('https://www.cloudflare.com/ips-v4') + get('https://www.cloudflare.com/ips-v6')
with open('/etc/dokploy/traefik/dynamic/cf-allowlist.yml', 'w') as f:
    yaml.safe_dump({'http': {'middlewares': {'cf-only': {'ipAllowList': {'sourceRange': ranges}}}}}, f)
p = '/etc/dokploy/traefik/traefik.yml'
c = yaml.safe_load(open(p))
for ep in ('web', 'websecure'):
    e = c['entryPoints'][ep]
    e['forwardedHeaders'] = {'trustedIPs': ranges}                 # trust CF's X-Forwarded-For
    e.setdefault('http', {})['middlewares'] = ['cf-only@file']     # reject non-Cloudflare
yaml.safe_dump(c, open(p, 'w'), sort_keys=False)
print(f'{len(ranges)} Cloudflare ranges applied (middleware + forwardedHeaders)')
PY
  _sync_traefik_conf
  run "$PRIMARY" "docker service update --force dokploy-traefik" >/dev/null
  info "re-run this same command whenever Cloudflare's published ranges change"
}

### ── verify (§12-ish): cluster, ingress, LB health ─────────────────────────
cmd_verify() {
  need hcloud; need jq
  run "$PRIMARY" docker node ls
  run "$PRIMARY" "docker service ps dokploy-traefik --format '{{.Node}} {{.CurrentState}}'"
  _ping_managers
  hcloud load-balancer describe "$LB_NAME" -o json | \
    jq -r '.targets[].health_status[]? | "LB target \(.listen_port): \(.status)"' 2>/dev/null || \
    hcloud load-balancer describe "$LB_NAME"
}

### ── dispatch ──────────────────────────────────────────────────────────────
usage() { sed -n '3,22p' "$0"; exit 1; }
case "${1:-}" in
  provision)        cmd_provision ;;
  wait)             cmd_wait ;;
  swarm-init)       cmd_swarm_init ;;
  dokploy-install)  cmd_dokploy_install ;;
  join)             cmd_join ;;
  traefik)          cmd_traefik ;;
  lb)               cmd_lb ;;
  cf-allowlist)     cmd_cf_allowlist ;;
  verify)           cmd_verify ;;
  *)                usage ;;
esac
```