Rolling Out to 1,200 Studios

A franchise is not a data center

Every studio is its own small deployment: a computer on the studio floor, a local network the franchise set up, heart-rate hardware, and tablets on the equipment. Nothing about it is uniform, and none of it is in a rack someone can walk over to.

During the first year on the network, everything that could fail did. Installs failed. Studio networks failed. Computers filled their disks. Hardware in the room died. And it tended to surface at the worst possible moment - the first class of the day.

Two people, one country

From March 2025 to March 2026, on-call was a two-person rotation: me and one other engineer from the team. There was no separate operations group - the people who built the software were the people who got the call.

Studios open at 4am local time, including on the West Coast, and I am on Eastern time. Follow-the-sun was never an option with two people, so the rotation simply absorbed Pacific mornings. Overnight rollouts were watched, not fired and forgotten, and a call could be about the network, the software, or the hardware - there was no such thing as not my layer.

The call that became a script

One 4am call: a studio could not start its class software because the studio computer had run out of disk space. Getting them running meant working through it remotely - compacting the virtual disk, clearing out unused container images, and restarting the Linux environment the software runs in to get its memory back - before the first class walked in.

Once was an incident. It kept happening at other studios, so I wrote a script that does the whole recovery in one step, and it became part of how we handled the fleet. Then I went after why the disks were filling in the first place: runaway logs from one of our API containers. Fixing the cause is what actually made the calls stop.

Turning 4am calls into product work

The most useful thing on-call did was change what we built. When the same failure woke us up twice, it went on the backlog as a product problem rather than staying a runbook step.

Beyond the disk-space fix, storage cleanup became automatic, including a check at the end of each class that catches low storage before it can take down the next one. Connections between the coach console and the studio's screens got liveness checks. Class start and end got guardrails so overlapping classes cannot corrupt the live room. Install and update paths got safer, and whoever was pushing a release got better visibility into where it had landed.

A console for who runs what

Once hundreds of studios were live, the team needed a real answer to three questions: which version is each studio on, which studios get the next release, and which ones are behind. I built the core operator workflows that answer them, replacing an ad-hoc internal tool.

It covers a studios-by-version view, recent activity, deployment groups and assigning studios to them, creating and listing software releases, and choosing which applications a group should run - with overrides for the individual studio that cannot take the default.

The people using it are not all engineers, and a wrong target version can take down a location's next class. So it is built around showing current truth rather than a cached guess, and closed or inactive studios never look like healthy targets.

Staged, never big-bang

You cannot update 1,200 physical locations in one night and find out in the morning. Releases went out to groups of studios in waves, so a problem showed up in a handful of locations instead of across the whole network.

Groups over per-studio snowflakes was the key tradeoff. Every one-off configuration is something the on-call engineer has to remember at 4am, so overrides exist for the studios that genuinely need them, and everything else follows its group.