StartLobster — Technische Architektur

Overview

System Architecture

The platform consists of a highly available reverse proxy with automatic SSL termination, self-registration of backend servers, and a fully automated provisioning pipeline.

┌─────────────────────────────────────────────────────────────────────┐ │ INTERNET / CLIENTS │ │ │ │ *.startlobster.de → DNS Wildcard A-Record │ └──────────────────────────────┬──────────────────────────────────────┘ │ ▼ ┌──────────────────────────────────────────────────────────────────────┐ │ FLOATING IP: floating-ip.example │ │ Keepalived VRRP Failover │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ │ │ │ proxy-1 (MASTER) │◄──VRRP──►│ proxy-2 (BACKUP) │ │ │ │ Caddy v2.11 │ sync │ Caddy v2.11 │ │ │ │ SSL Termination │◄──────►│ SSL Termination │ │ │ │ Registration API│ :2019 │ Registration API│ │ │ │ :9876 │ │ :9876 │ │ │ └────────┬─────────┘ └────────┬─────────┘ │ └───────────┼──────────────────────────────┼───────────────────────────┘ │ │ │ Private Network 10.0.0.0/16 │ │ ┌───────────┼──────────────────────────────┼───────────────────────────┐ │ ▼ ▼ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ s1 (10.0.1.x) │ │ s2 (10.0.1.x) │ │ sN (10.0.1.x) │ ... ×500 │ │ │ │ │ │ │ │ │ │ │ Gateway │ │ Gateway │ │ Gateway │ │ │ │ :18789 │ │ :18789 │ │ :18789 │ │ │ │ Chat UI │ │ Chat UI │ │ Chat UI │ │ │ │ :8888 │ │ :8888 │ │ :8888 │ │ │ │ Auth │ │ Auth │ │ Auth │ │ │ │ :3000 │ │ :3000 │ │ :3000 │ │ │ └──────────────┘ └──────────────┘ └──────────────┘ │ │ BACKEND SERVER │ └──────────────────────────────────────────────────────────────────────┘

Components

System Components

Each component is independently scalable and connected via clearly defined interfaces.

🛡️

Reverse Proxy (Caddy)

Zwei Caddy v2.11 Instanzen mit Wildcard SSL-Zertifikat (Let's Encrypt). Automatische SSL-Terminierung für alle *.startlobster.de Subdomains. JSON-basierte Admin API für dynamisches Routing.

🔄

HA Failover (Keepalived)

VRRP-basiertes Failover mit Floating IP. proxy-1 als Master, proxy-2 als Backup. Bei Ausfall automatische IP-Umschaltung via Hetzner API in <5 Sekunden.

📝

Registration API

REST API (Port 9876) auf dem Private Network. Backends registrieren sich selbst beim Proxy. Unterstützt Simple Routes (Gateway) und Full Routes (Auth + UI + Gateway).

🔐

SSL / Zertifikate

Wildcard-Zertifikat via acme.sh mit Custom DNS Hook für http.net API. Automatisches Renewal. Cert-Sync zwischen beiden Proxy-Instanzen.

🌐

DNS (http.net)

Wildcard A-Record *.startlobster.de → Floating IP. Kein individueller DNS-Eintrag pro Server mehr nötig. Verwaltung über http.net Partner API.

⚡

Provisioning Pipeline

Vollautomatisiert: HubSpot-Formular → Hetzner Server → Private Network → Docker + Services → Proxy-Registration. Pool-basiert für Instant-Assignment (<30s).

Infrastructure

Server & Network

Component	Specification	IP / Netzwerk	Location
proxy-1 (Master)	CX23 · Debian 12 · Caddy + Keepalived	`proxy-1.example` / `10.0.1.x`	Nuremberg (nbg1)
proxy-2 (Backup)	CX23 · Debian 12 · Caddy + Keepalived	`proxy-2.example` / `10.0.1.y`	Nuremberg (nbg1)
Floating IP	IPv4 · VRRP-gesteuert	`floating-ip.example`	Nuremberg
Private Network	startlobster-net · 10.0.0.0/16	Subnet: `10.0.1.0/24`	eu-central
Backend Server	CX23 · Ubuntu 24.04 · Docker	`10.0.1.x` (Private)	Nuremberg (nbg1)

💡 Netzwerk-Architektur

Backend servers are only accessible via the Private Network. All services (Gateway :18789, Chat UI :8888, Auth :3000) bind only to the private IP. No public access to backend ports — all traffic runs through the reverse proxy.

Routing

Request Flow

How a request from a client is routed to the correct backend.

DNS Resolution

Client ruft s42.startlobster.de auf → Wildcard DNS löst auf Floating IP floating-ip.example auf.

SSL Termination

Caddy auf dem aktiven Proxy terminiert TLS mit dem Wildcard-Zertifikat *.startlobster.de.

Route Matching

Caddy matched den Host-Header gegen registrierte Routen. Für s42.startlobster.de wird die Full Route verwendet (Auth + Chat UI + Gateway).

Forward Auth

Caddy prüft via forward_auth beim Auth-Container (10.0.1.x:3000), ob der User authentifiziert ist. Bei Erfolg wird der Gateway-Token als Header injiziert.

Reverse Proxy

Je nach Path wird zum Backend geroutet:
/auth/* → Auth :3000
/api/* + /ws → Gateway :18789
/* → Chat UI :8888

Self-Registration

Backend Self-Registration API

New servers automatically register with the reverse proxy — no manual DNS entry or config change needed.

Endpoint

HTTP
POST http://10.0.1.x:9876/register
Authorization: Bearer <SHARED_SECRET>
Content-Type: application/json

Simple Route (z.B. Gateway-Subdomain)

JSON
{
  "subdomain": "s42-gw",
  "backend_ip": "10.0.1.x",
  "backend_port": 18789,
  "type": "simple"
}

Full Route (Auth + Chat UI + Gateway)

JSON
{
  "subdomain": "s42",
  "backend_ip": "10.0.1.x",
  "type": "full"
}

✅ Auto-Sync

Registrations are automatically synced to both proxy instances. Only one API call needed — the receiving proxy forwards the route to the peer.

Additional Endpoints

Method	Path	Description
`POST`	`/register`	Register route (simple or full)
`POST`	`/deregister`	Remove route
`GET`	`/routes`	List all active routes
`GET`	`/health`	Health Check

Provisioning

Automated Server Provisioning

From HubSpot form to functioning server in under 30 seconds thanks to server pool.

HubSpot Trigger

New form submission → Pipeline detects new customer.

Pool Assignment

Pre-configured server from the pool is assigned. Pool maintains 5 ready servers. Automatically re-provisioned upon assignment.

Hetzner Server + Private Network

CX23 server is created in startlobster-net. Automatically receives a private IP (10.0.1.x).

Proxy Registration

Pipeline registers Full Route + Gateway Route with the reverse proxy. Instantly accessible via sXX.startlobster.de.

Ansible Provisioning

Playbook install-docker-proxy.yml: Docker, OpenClaw Gateway, Chat UI, Auth — alle Services nur auf Private Network Interface gebunden.

Verification & Notification

Gateway reachable? → HubSpot note with credentials → WhatsApp notification to team.

Security

Security Architecture

🔒

Network Isolation

Backend services are only accessible via the Private Network. No public ports on backend servers. All external traffic runs through the reverse proxy.

🛡️

Firewall

iptables auf allen Proxies: Port 22/80/443 öffentlich, Admin API (:2019) und Registration API (:9876) nur aus dem Private Network. VRRP-Traffic erlaubt.

🔑

Authentication

Registration API: Bearer Token (Shared Secret). Forward Auth auf dem Proxy für Enduser-Zugang. Gateway Token pro Backend-Instanz. Kein öffentlicher Zugang zu Admin APIs.

⚠️ Sicherheitshinweis

The Registration API only accepts requests from the Private Network (10.0.0.0/16). Public requests on ports 9876 and 2019 are dropped via iptables before reaching the service.

Backend

Backend Server Stack

Each backend server runs lean without its own reverse proxy — SSL and auth are handled centrally.

Container	Port (Private IP)	Function
openclaw-gateway	`10.0.1.x:18789`	AI Gateway — WebSocket + REST API
startlobster-chat	`10.0.1.x:8888`	Chat UI (nginx + Static Files)
startlobster-auth	`10.0.1.x:3000`	Auth Container (Token-basiert)

🚀 Vergleich: Legacy vs. Proxy-Modus

Legacy: Each server had its own Caddy + Let's Encrypt + DNS records → 50 certs/week rate limit, slow provisioning.
Proxy Mode: One wildcard cert, no Caddy on backends, instant registration. Provisioning ~80% faster.

High Availability

Failover Mechanism

⚡

Normal State

proxy-1 (Master, Priority 101) holds the Floating IP. proxy-2 (Backup, Priority 100) stands ready. Keepalived checks every 5 seconds if Caddy responds.

🔴

Master Failure

Keepalived on proxy-2 detects missing VRRP advertisements. After 3 failed health checks (15s), proxy-2 takes over the master role.

🔄

Floating IP Migration

Keepalived triggers hetzner-failover.sh → Hetzner API reassigns the Floating IP to proxy-2. DNS remains unchanged. Downtime: <15 seconds.

✅

Recovery

When proxy-1 comes back online, it automatically takes back the master role (higher priority). Floating IP migrates back.

Tech Stack

Technologys Used

Category	Technology	Version / Details
Reverse Proxy	Caddy	v2.11.1 (Custom Build via xcaddy)
HA / Failover	Keepalived	VRRP auf Private Network Interface
SSL Certificates	Let's Encrypt + acme.sh	Custom DNS Hook für http.net API
DNS	http.net Partner API	Wildcard A-Record
Cloud	Hetzner Cloud	CX23, Debian 12 / Ubuntu 24.04
Networking	Hetzner Private Network	10.0.0.0/16, Floating IP
Container Runtime	Docker + Compose	Latest stable
Provisioning	Ansible + Python Pipeline	Idempotent, Pool-basiert
Registration API	Python (stdlib)	http.server, Port 9876
CRM Integration	HubSpot API	Form Submissions + Contact Notes
Notifications	WhatsApp (OpenClaw)	Auto-Notify bei Provisioning

StartLobster Architektur

Reverse Proxy (Caddy)

HA Failover (Keepalived)

Registration API

SSL / Zertifikate

DNS (http.net)

Provisioning Pipeline

DNS Resolution

SSL Termination

Route Matching

Forward Auth

Reverse Proxy

Endpoint

Simple Route (z.B. Gateway-Subdomain)

Full Route (Auth + Chat UI + Gateway)

Additional Endpoints

HubSpot Trigger

Pool Assignment

Hetzner Server + Private Network

Proxy Registration

Ansible Provisioning

Verification & Notification

Network Isolation

Firewall

Authentication

Normal State

Master Failure

Floating IP Migration

Recovery