Journal

Decisions

Decisions

Newest first. Date, decision, rejected alternatives, reason, consequences.

2026-11-05 — Machine brain is a files-first Markdown vault

Decision: Journals, system facts, settings and reference notes live as plain Markdown files in this git repo. Obsidian is the browsing layer. LLMs read and write the files directly. BookStack (pdb.seabee.me) becomes a published mirror, not the source of truth.

Rejected:

Reason: The bottleneck was findability and LLM access, not note features. Files solve both with zero glue: Obsidian for humans, plain paths for agents.

Consequences:

2026-11-05 — vps2 replaces the old vps as the public edge

Decision: Everything the old vps (203.57.114.42 / 100.64.0.2) did moves to vps2 (187.52.126.150 / 100.64.0.16), and the old box is retired: public Caddy edge, HAProxy mail ingress, outbound Postfix relay, and headscale. The Ansible repo now treats the vps2 Caddyfile as the authoritative edge.

Why: One VPS instead of two, and the replacement already carries the workloads (marketdata, strata, Orca) that were spread across cbfarm and cbcore.

Alternatives rejected:

Consequences:

Ops Journal — issues, root causes, and fixes

Ops Journal — issues, root causes, and fixes

A running log of things that broke, why they broke, how they were fixed, and what to do so they never happen again. Newest entry first. Each entry follows the same shape so it stays useful as a reference, not just a diary.

Maintained by Claude across sessions. When a real issue is found and fixed, add an entry here — symptom, root cause, fix, and the prevention that makes it non-recurring. Keep it concrete: exact files, commands, and IPs.

Template:

## YYYY-MM-DD — <short title>
**Symptom / how it surfaced:**
**Root cause:**
**Fix (what was actually done):**
**Prevention (so it never happens again):**
**Lessons / notes:**

2026-11-05 — vps → vps2 migration (headscale, marketdata, strata; mail staged)

Symptom / how it surfaced: Not an incident: vps (203.57.114.42 / 100.64.0.2) is being deleted, so everything it did had to move to vps2 (187.52.126.150 / 100.64.0.16) without breaking sites or mail.

Root cause: The old box was doing three unrelated jobs — public TLS edge, mail ingress (HAProxy), and outbound SMTP relay — plus running headscale.

Fix (what was actually done):

Prevention (so it never happens again):

Lessons / notes:

2026-08-16 — mailcow open relay: ~94k spam messages sent from our IP

Symptom / how it surfaced: Abusix network-abuse report, forwarded by BinaryLane, naming 203.57.114.42 (the vps, mail.seabee.me) as a spam source. Investigation found the server had been an open relay from 2026-08-05 15:52 UTC to 2026-08-13 18:06 UTC: 94,223 messages accepted, 93,602 delivered. Spoofed senders test@seabee.me (82k), info@seabee.me (11k), and MetaMask phishing. Not a compromise — no breached accounts, no rogue SASL logins; the spam never authenticated.

Root cause: Three things lined up:

  1. mailcow trusts its own Docker subnet — mynetworks included 172.29.1.0/24.
  2. Port 25 was published on the Tailscale IP and handled by Docker's userland proxy (docker-proxy), which reopens each connection from the bridge gateway 172.29.1.1. So every external sender appeared to come from 172.29.1.1.
  3. 172.29.1.1 is inside the trusted /24 → permit_mynetworks → relay granted to anyone. HAProxy passthrough was NOT the bug; the bug was source-IP masking turning an untrusted sender into a trusted one.

It stopped by accident on 2026-08-13 when a container restart recreated the DNAT rules (so source IPs were briefly preserved again). The hole was still live — verified a forged connection from 172.29.1.1 got 250 Ok for an external recipient until fixed.

Fix (what was actually done):

Prevention (so it never happens again):

Lessons / notes: