We own the machines

The software Teralis runs sits on machines we own. Not instances, not a reserved pool. Actual computers, in a room, on a power bill.

This is not a position. We are not against renting servers. It just turned out that for our own work owning was the sensible option, and it stayed sensible long enough to become the default. We do not own the edge in front of the machines either, and we have no ambition to.

The shape of the work

The argument for renting is elasticity. You pay for what you use, you scale up on a Tuesday and back down on a Wednesday, and you never buy a machine you needed for one afternoon. That is a good deal when the load is spiky or short lived.

Ours is neither. One of our machines does audio analysis, so it has a graphics card in it, and that card is busy for a large part of most days. A rented machine of that shape is priced for people who need it occasionally. We need it constantly. Once the utilisation is that flat the arithmetic stops being interesting.

We are not going to publish a comparison. We have not measured it carefully enough to put a number on, and a percentage saved in a blog post is usually a percentage saved on a workload chosen to make the number look good. The shape of the decision is the useful part. Steady work, owned. Spiky work, rented. Most people already know which one they have.

There is a second reason, less tidy. When you own the machine you can look at it. You can put a bigger disk in it. Nobody deprecates the instance type, nobody edits the pricing page. That is worth something, and it is worth more the longer you keep the thing.

What it actually costs

Attention. That is the honest answer, and it is not a small amount of it.

The power here is not reliable. Cuts happen often enough that we plan around them, and the machines sit on different circuits, so a cut does not always take all of them. Two are on battery backup. The third is not, because a unit big enough for it costs more than the risk seemed to be worth, so that one takes the cuts.

Then the embarrassing part. The batteries have data cables. Those cables are not plugged in. So the machines never learn they are running on battery, carry on as normal, and take a hard cut anyway once the battery is flat. We bought the protection and skipped the last step that makes it useful. The fix is a cable and a small daemon. It has been on the list for a while.

Disks keep a counter for unclean shutdowns and ours is not at zero. After every hard cut there is a round of checking: database integrity, the kernel log, the drive's own health report. So far all of it has come back clean. The checking is what turns that from luck into something we know.

What breaks

The most useful failure we have had was not a hardware failure at all.

The process supervisor on one machine had been set up under the wrong account. After a reboot it came back exactly as designed, read its list of things to run, and found the list empty, because the list belonged to a different account. So it started nothing, and reported that it was running perfectly. Which it was.

Every service check was green. Every site returned a 200, because the homepages on that machine are static files and a static file does not need anything to be running. Only the paths that were proxied to an application were down, and nobody was looking at those. It stayed like that for five days.

We found it by looking, not by being told. That is the part worth keeping. A supervisor that is healthy is not the same as a supervisor that is doing anything, and a page that loads is not the same as a system that works. The boot path now gets verified end to end rather than checked for a green light.

What we do about it

Less than you would think, on purpose.

The service that matters most has a warm spare on a second machine, pulled across every few minutes. Failover is manual and stays manual. Doing it automatically would leave both copies live at once, and two databases drifting apart is a worse problem than a few minutes of downtime while somebody types a command.

Separately there is a nightly backup that holds data and no code, so it cannot serve anything. It exists only to restore from. It also never deletes, so a bad day on the application server cannot quietly erase its own history.

Whether you should

Probably not, if the load is spiky, or there is nowhere to put the machines, or the hour a week is worth more than the difference.

We own ours because the work is steady, the machines are close, and we would rather the failures be ours to understand. The failures are real. The power goes out, the batteries do not mention it, and we once spent five days running a system that was quietly doing nothing. None of that is an argument against owning hardware. It is what owning hardware is.