engineering writeup
Publishing a homelab with zero open inbound ports
The problem
Running public services from a residential connection normally means forwarding ports and accepting that anything reachable is also scannable. I wanted the services publicly available and the network itself unreachable — nothing listening on the WAN side at all.
The shape
client → CDN edge → outbound-only tunnel → reverse-proxy hub
↓
forward-auth → service
config: git repo → reconciler (2 min) → validate → reload → health check
↓ fail
auto-rollback
The hard parts
TLS terminates at the edge, and forgetting that cost me an afternoon. The tunnel hands the proxy a plain HTTP request that was HTTPS a moment earlier. If the proxy is configured to expect HTTPS upstream, it issues a redirect to the HTTPS URL, which arrives back as plain HTTP, and the loop never terminates. The fix is a one-word scheme change, but the symptom — an infinite redirect on a config that looks correct — points nowhere near the cause.
The reconciler cannot live on the host it reconciles. The proxy host has no outbound internet access, deliberately — it is the most exposed thing in the network and that restriction is a large part of why it is safe. But a GitOps reconciler must reach the git remote. Rather than punching an egress hole through the constraint that provides the security, the reconciler runs on a different host and pushes rendered config in. The restriction stayed intact.
Reload before validation is a self-inflicted outage. A malformed config that reaches a live reload takes every service down at once, and the blast radius is the whole fleet. The reconciler validates the rendered config, reloads, then health-checks — and rolls back automatically if the check fails. The rollback path matters more than the apply path, because the apply path is the one that gets tested constantly.
Drift correction needs a break-glass. Hand edits revert within two minutes, which is exactly what I wanted until I was debugging at 2am and my change kept vanishing with no explanation. Stopping the reconciler timer is the documented escape hatch. An enforcement loop without an off switch is a trap I'd built for myself.
What I'd do differently
Drift is detected by comparing rendered output, so a change that is semantically identical but textually different still reads as drift. Comparing parsed config rather than text would cut the false positives.
The health check after reload is shallow — it confirms the proxy is answering, not that every route behind it still resolves correctly. A bad route that returns a valid error page passes today.
Rollback restores the previous config but does not alert loudly enough that it happened. A silent successful rollback looks identical to a successful apply from the outside, which is precisely when I most want to know.