Proof

Production case study · Written August 7, 2026

Helm runs a real floor. Twenty-two gaming stations at a youth esports venue on Long Island come up on it every day the doors open, kids sit down at them, and if it breaks the consequence lands the same night in front of paying parents. I built it, I operate it, and I am the customer. This page is the evidence for all three, with the numbers read off the tree rather than remembered.

22stations, live nightly
4,127tests green
31.NET projects
1.8Mevents ingested

01It is real, and it is live

The venue is Valhallan Wantagh: twenty-two stations, after-school sessions, weekend blocks, camps, birthday parties. A small crew. No IT staff in the building. Sessions run to a fixed clock, so "we will look at it tomorrow" is not available: a station that is wrong at 6 PM on a Friday is wrong in front of a full room.

That constraint is the whole reason this system looks the way it does. Nothing here was designed against a requirements document. Every piece of it exists because a specific evening went badly and it needed to not go badly again.

Helm is my own company and I hold the IP. The venue is my employer's, and it runs Helm as the first production deployment rather than as a customer I sell to. That is why this page can exist at all: the system is mine to document, and I am documenting it while it is live, with every number read off the tree or a run log rather than recalled.

02It is engineered, not scripted

Four runtime pieces and one loop. A WPF launcher on every station, a PowerShell agent running as SYSTEM underneath it, a multi-tenant ASP.NET minimal API on Fly.io, and the operator surfaces the crew actually touches. The arrow that matters is the one going back down.

The loop, not the boxes. Stations report up, the cloud answers down, and a reconciler re-asserts desired state on a timer, so a missed edge heals itself instead of waiting for somebody to notice.

Two decisions carry most of the reliability. The first is reconcilers over edge-triggers: nothing critical depends on a single event arriving. A station that missed a command because it was powered off gets the desired state re-asserted within fifteen minutes of coming back. The second is one authority per question: there is exactly one module that answers "is somebody sitting at this station", one that answers "which house account belongs to this seat", one that rations platform sign-ins. Incident 03 below is what happens without that.

03The numbers, and where each came from

Counted on the tree, not estimated, on August 7, 2026, after the station-auth-gate merge landed.

System scale, measured
.NET projects	31	8 test projects, 23 apps, libraries, tools and samples
C# and XAML	222,410	lines tracked, 1,040 C# files and 45 XAML files
PowerShell in the repo	145	scripts, 21,335 lines, under the same CI as the C#
Cloud routes	252	minimal API, no MVC controllers anywhere
Schema migrations	70	forward-only, SHA-pinned, refuse to start if one changed
Commits	728	105 days, first one April 24, 2026
Events ingested	1,808,232	read off the production cloud, August 7, 2026
Stations in production	22	all on the same launcher build

Tests: 4,127 green, 0 failed, 0 skipped

One full-solution dotnet test -c Release across seven assemblies, rebuilt and run against the tree on August 7, 2026. Per suite:

Test suites and counts
Cloud	1,459	licensing state machine, billing, refunds, tenant isolation, catalog, match ingest, ranks, quests, seasons, power scheduling, briefing evals, station auth gate
Launcher	1,001	launch service and typed outcomes, session tracker, match telemetry relay, per-game adapters, ranks, checkout, idle and abandoned-launch watchers
Front desk	876	pillar tabs, command palette, settlement and licensing flows, readiness and rollout rows
Shared	487	event wire shapes, match summaries, challenge models, request signing and replay protection
Station agent	302	heartbeat, status reporting, job surface auth, replay protection
Smoke	2	front desk cold start, front desk UI
Total	4,127	measured August 7, 2026

CI: three jobs, nine gates

Three jobs on Windows and Linux runners, every push and every pull request, with the SBOM step reserved for release tags. The last gate in the static-analysis job is the one I care about most, and incident 05 is why.

CI jobs and gates
Build (warnings = errors)	build-test	no warning ships
Test	build-test	the full solution, Release
Verify-Demo	build-test	invariants on a cold-start cloud
Verify-Operator-Console	build-test	every drawer and handler wired
Verify-Launcher-Catalog	build-test	every tile resolves to a runnable adapter
PSScriptAnalyzer	static-analysis	the ops layer, linted
Audit grep gate	static-analysis	banned patterns cannot reappear
PowerShell 5.1 compatibility	static-analysis	the shell the stations actually run
Generate SBOM	sbom	CycloneDX, on release tags

04It is operated with discipline

Green tests are necessary and never sufficient. A build is done when the change is verified on a real station, on screen, by capture. What sits between a commit and twenty-two machines:

Artifact
Every launcher build is SHA-256 hashed. The deploy script refuses any binary whose hash it was not given, and the previous known-good build with its own hash stays sealed and ready before the first station updates.
Canary
One station, on the floor, exercising the actual path the change touches. Then a capture, looked at with my own eyes. A build that passed the suite and a laptop smoke test has still died instantly on a real seat over a native packaging difference my laptop cannot reproduce, which is exactly how this rule got written.
Rollout
Staged, never fleet-wide in one shot, with the three watchdog tasks that would relaunch the old build disabled first. The deploy verifies the effect on each seat, not that the command was issued.
Rollback
Named, hashed, and tested before rollout starts. Not a plan, a build. The venue data restore is held to the same standard: rehearsed rather than documented, executed and timed repeatedly at about a minute end to end.
Standing watch
Scheduled reconcilers, per-station watchdogs, telemetry with freshness checks, and a morning readiness pass that runs before anyone arrives and puts what needs eyes on one screen.

The floor readiness board: 20 of 22 stations ready to open, one needs sign-in, one down, over a checklist covering license pools, house accounts, quests, the shutdown schedule and the venue worker heartbeat — The morning pass, before anyone arrives. Twenty-two stations accounted for, overnight drift already repaired or flagged, and what needs human eyes called out by name.

The evening equivalent is the close, and it is verified rather than assumed: on August 6, 2026, all 22 stations powered off at 9:15 PM, stations under a coach hold kept their sign-ins, and every other seat cleaned itself. The same audit came back 22 of 22 a week earlier. Incident 01 below is why that audit exists.

Unreachable is never a pass. A station that did not answer is a station that did not get the change, and reporting it as green is the single easiest way to lose a floor.

05Five incidents, and the rule each one bought

This is the section a skeptic should read. Each of these cost something real, and each one ends in a rule the tooling now enforces rather than a note I promised myself I would remember.

01 The party blackout July 3, 2026

What broke: A one-night early close was implemented by rewriting the standing shutdown task on every station, swapping its weekly triggers for a single daily one. That coupled a temporary intent to permanent infrastructure and erased the real schedule. It then fired at the same time every night. Four quiet evenings in a row hid it. The cancel script was dead too: it searched for task names this fleet has never had, so it reported success while disabling nothing.
What it cost: A 13th birthday party, about 22 kids, half of them first-time visitors. Every station in use powered itself off mid-party, 85 minutes before the party was over.
The fix: A one-night change now ships as a separate, self-deleting task alongside the standing ones, which are only ever disabled by state. A self-deleting startup task that restores the standing schedule is armed before anything depends on it, so canon comes back at the next boot even if every human forgets. StartWhenAvailable is forced false fleet-wide, so a station that was powered off at close never fires that close ten minutes after its next boot.
The rule now: Never mutate standing infrastructure for a temporary intent. Verify-ShutdownSchedule.ps1 audits every station's triggers against config and has to print ALL VERIFIED PCs MATCH THE SCHEDULE before the change is done. Since August 1 it also runs unattended once a day, so the audit no longer depends on somebody remembering to run it.

02 The reconciler that matched the wrong night August 6, 2026

What broke

When a station borrows a pooled game account, the launcher writes a crash-safe intent file so a swap interrupted by a crash can be unwound later. The reconciler that unwinds stranded intents matched on game title alone. A stranded intent from the previous evening for the same title therefore matched a live launch that was five seconds old, and tore it down.

What it cost

A kid tapped a game at 10:04 AM and got nothing, through two automatic retries and a 90-second timeout. The station was put back on the house account, which does not own that game, so every retry was doomed by construction. Three minutes later the seat was signed into no account at all, which is worse than where it started.

10:04:35  launching Meccha Chameleon  ->  POST /licenses/lease
10:04:38  account swap completed: pooled account in place
10:04:45  [WRN] swap-intent reconcile: stranded swap for Meccha Chameleon
          (swapped 2026-08-05 19:55:06Z); restoring house account,
          releasing lease
10:04:49  account swap completed: house account  ->  POST /licenses/release
10:07:23  Steam LogOff()  ->  seat signed into nothing

The fix

Intents match on the specific swap instance and its timestamp, never on title, and the reconciler refuses to unwind an intent younger than the launch currently running.

The rule now

A reconciler repairs the state it was told about, not any state that resembles it: identity-scoped and generation-aware, or it is a random teardown on a timer. Every timer-driven repair now gets walked through four states before it ships, because the same logic means different things at boot, in steady state, at close, and half way through a deploy.

03 The seat that was busy with nobody on it August 4 and 6, 2026

What broke: The house Roblox account auto-starts and sits on screen forever, so the presence check saw a running game and reported a player at an empty station. It could not settle the question another way, because the input-idle probe returns nothing over remote management: those sessions cannot read console input. Underneath that, a sweep found more than twenty hand-typed lists of game process names across the estate, no two identical, with one file disagreeing with itself between two of its own sections.
What it cost: Both directions, and the expensive one is not the obvious one. A station wrongly read as busy silently stops self-healing, because every reconciler is built to refuse a seat in use. A station wrongly read as empty is worse: a coach playing a title that was on nobody's list got kicked off his own account back to the house account, mid-match, four separate times.
The fix: One module, Lib\SeatPresence.ps1, replaced all twenty-plus lists and answers by union of five signals, the strongest of which is where the running process lives on disk rather than what it is called. That signal is catalog-free, so a game nobody has heard of still counts. For the idle question specifically, a task that runs inside the console session reports real seconds since the last keypress. On the disputed station it read two hours and ten minutes. Proof, not inference.
The rule now: One authority per question, and unknown means present. Written into the module as an argument about price: a missing game name costs a customer their session, a missing app name costs a deferred cleanup. Those are not the same mistake. A 59-case test gate holds it, and a real person on their own account gets an explicit hold switch, because no amount of process sniffing will ever infer that.

04 The gate that only guarded lowercase August 4 to 6, 2026

What broke: An authorization gate over the cloud's management and command routes compared request paths with an ordinal, case-sensitive comparison. Web framework routing is case-insensitive. So the same route requested with different capitalization matched the router, did not match the gate, and was never challenged.
What it cost: No exploitation is recorded, but the exposure was real and live rather than theoretical: member records and remote session and power control over any station reachable by anyone who knew the URL shape. The gate read correct in review and its tests were green.
The fix: Case-insensitive comparison, and the path-matching decision made internal so it can be tested without standing up a web host. The gate now resolves once at startup instead of reading an ambient environment variable inside the request path, after an early cut behaved differently on my laptop than it would have in CI. A security control whose verdict depends on the operator's shell is not a control.
The rule now: Verify a security fix by negative control. The fix shipped with 22 new gate tests, 12 of which fail against the old comparison and all 22 of which pass against the new one. A test that cannot fail against the bug proves nothing. Then verify from outside the process, in production, in the variants an attacker would try: lowercase and capitalized both refused, three casings of the command route identical, the operator console with a real token still working, station traffic unchanged.

05 Every check ran on a shell the floor does not have August 5 and 6, 2026

What broke: CI and my laptop run PowerShell 7. All 22 stations run Windows PowerShell 5.1. Newer syntax is a parse error on 5.1, several parameters parse fine and fail at run time, and a file saved as UTF-8 without a byte-order mark is read as a different encoding entirely. A script could clear everything I had and die on the only machines that matter.
What it cost: The nightly cleanup was saved without that byte-order mark. It carries several hundred box-drawing characters in its section headers, so on 5.1 it failed to parse on all 22 stations, dying before its first line ran. That night's close would have failed fleet-wide. Two tools then hid it: the fleet runner discarded the error stream and printed sixteen green OK lines while nothing at all had happened, and the guard whose entire job is answering whether anyone is playing filed all 22 seats unreachable, printed zero reachable, zero in use, and passed.
The fix: tools/ci/Test-Ps51Compat.ps1 is a required CI gate that shells out to the real 5.1 interpreter and fails only on what would actually stop a station. A script that is genuinely laptop-only exempts itself by declaring a version requirement. It found 20 broken files on the laptop and 4 on every station on its first run; a follow-up repaired 121 files byte-identically. Cleanup now parses itself before running and refuses with the line number, then proves the run by diffing the station's own log.
The rule now: Three, and they compound. Where dev and production disagree about a runtime, the disagreement is a CI gate, not a note. A fleet command reports what the station actually did, proved from the station's own artifact, never from the fact that something was launched. And unreachable is never a pass: a fleet check needs three outcomes, not two, and zero reachable stations is a symptom of my laptop, never of an empty floor.

The pattern is deliberate. A fix that only repairs the instance is half a fix. The question after every incident is what class it belongs to, which sibling has the same bug, and what would have caught it without me.

06The AI part, stated plainly

I build this solo, working agentically with Claude Code. The model writes most of the lines. What I own is everything that decides whether those lines can be trusted on a floor: the architecture and its invariants, the release engineering, the verification, and the incident response. The honest version of that sentence is the short one:

The code volume comes from the model. The architecture, verification, and incident response are mine.

That is the method. Separately, there is a shipped LLM feature in the product, which is a different claim and worth stating precisely.

The Floor Briefing

The venue's owner-facing daily report opens with a short written read on the day, generated from that day's real numbers. It shipped into the production cloud on August 7, 2026 (7a79f73, 13e0f8f, 2305779), and the first delivery is log-confirmed. How it is built:

Model and transport. claude-opus-5 through the Anthropic Messages API. Hand-rolled REST over a named HttpClient to match the rest of the cloud's gateway pattern, with zero third-party API SDK dependencies, reading only text content blocks and sending no sampling parameters.
Provider abstraction. One interface, IBriefingProvider, two implementations: the model provider and a fully deterministic template that writes the same facts as plain prose. The service gathers real facts first and only then tries the model, so there is no code path where the briefing fails outright.
Refusal handling. A safety decline arrives as HTTP 200 with stop_reason: "refusal", not as an error, so code that reaches straight for the content silently ships an empty briefing. The provider checks the stop reason first and treats a refusal, an empty content array, and any transport failure identically: throw, log, and let the deterministic path answer. The morning email goes out either way.
Evals, which are the actual point. Both providers are held to one contract: every numeral in the generated prose must exist in the input facts. The check builds a set of every number in the facts packet and asserts that every digit run in the prose is in it, with lookarounds so an identifier like a game slug or a station name is read as text rather than as a claim. A model that invents a session count is a model that lies to an owner about his own venue, so this is a hard gate, not a dashboard. Format evals sit next to it (word ceiling, house copy rules), adversarial fixtures cover the ways a faithfulness check gets fooled (digits inside game names, six-figure minute totals, a contradictory day with zero sessions and high attention counts, single-station days), and the live-model eval runs against the real API when a key is present and skips cleanly when it is not.

The honest limit, since this page is about evidence: no API key is provisioned in the production cloud today, so the briefing the owner receives each morning is currently the deterministic provider's, not the model's. That is the fallback doing exactly its job rather than a thing that broke, and the model path is built, wired into DI, eval-gated and green. But "shipped and proven against the live API in production" would be a bigger claim than I can currently make, so I am not making it.

The design principle underneath the whole feature: an LLM is allowed to write the sentence, and is never allowed to be the source of the number.

07What I have not built

The engineering page keeps the running list of gaps, and it stays honest on purpose. The short version: the database is SQLite with a Postgres path wired but not switched on, the cross-venue franchise rollup is designed and not shipped, payments are live for membership and simulated for the rest, and the Floor Briefing's model path is built and eval-gated but serving from its deterministic provider in production.

08Where to look next

Everything on this page is running on a floor right now. The venue page is what a week on it looks like and the changelog is the week-by-week record. A live look at the system running, or a question about anything here: alexhaeseker@gmail.com.

Alex Haeseker, founder. Every number here is checkable and every incident happened. If something looks off, write back and I will fix the page or the product, whichever one is wrong.