Why We Don't Rewrite Your Data When You Upgrade

The Upgrade That Fixes Things While You Sleep

There is a kind of upgrade everyone likes on the first day. You change the version, the system starts, and somewhere in its startup it quietly brings your stored data up to date. No command, no stop, no reading.

It feels like good engineering. The vendor did the work so you did not have to. My position is that, for a platform that holds other people’s data, it is the wrong design, and that the convenience is borrowed against a day you cannot choose.

A conversion that runs at startup runs when the process starts, not when you are ready. It runs without asking whether a backup exists. It does not tell you beforehand what it is about to change, and often it does not tell you afterwards either. And because startup happens again and again, a conversion that lives there is tempted to run again and again.

So in MaestroHub, the rule is the opposite: running code never converts stored data. Stored data changes in exactly one place, a command you run, after a backup, once. This article is about why, and about what holding to that rule costs.

What We Found in Our Own Code

Before we wrote the rule down, we counted how our own product changed stored data. The answer was uncomfortable.

At the time, stored data was being changed by at least five unrelated mechanisms: database migrations that rewrote rows, data passes that ran at every start and recorded nothing, conversions applied when a record was read or saved, several ways of changing table structure, and upgrade actions behind an admin screen. Most of them were invisible to the operator. They overlapped, some were not safe to run twice, and nothing stored could tell you which version of the product had written it.

None of that was malice or carelessness. Each mechanism had been the easiest way to solve one problem on one day. That is exactly the trap. Boot-time conversion is always the easiest place to put a fix, so a system that allows it grows more of them, and each one is a small surprise waiting for a particular install.

A Default Is Not a Decision

The clearest example of why this matters is a single number: how long a store-and-forward buffer keeps a message.

Store-and-forward is the part of a pipeline that holds data while a destination is away and sends it when the destination returns. In 3.0, every buffer was created with a retention of 24 hours, and that value was stored in the buffer itself. In 3.1, the default is keep until delivered: a buffer is bounded by the disk set aside for buffers, not by the age of a message. An outage that lasts longer than a day should not quietly throw away the readings it held.

So existing buffers need converting. Here is the difficulty. Once the old default is stored, a buffer that holds 24 hours looks exactly like a buffer where an operator chose 24 hours, unless the system can tell which version wrote it. A conversion that cannot tell them apart has two options, and both are bad: leave every old default in place, or overwrite every 24 in sight, including the ones a person chose on purpose.

Our first version of this conversion was written as a startup pass, and review caught the second failure in it: it re-applied the new default on every start. An operator who chose 24 hours by hand after the upgrade would have had it changed back at the next restart, with nothing in any log to explain why.

3.1 does it differently. Every buffer record now carries the data version it was written at. The upgrade converts the old default once, on records written before versions existed, and never touches a retention an operator sets afterwards. A record can no longer look like it is waiting for a conversion it has already had.

For data 3.0 already wrote, the ambiguity cannot be undone after the fact: a 24 hours someone chose deliberately in 3.0 looks the same as the default. A retention other than the old default is kept as it is. Every buffer that holds exactly 24 hours is moved, and upgrade check lists each one before anything changes, so whoever wanted a day can see it and set it again.

Tell Me What You Will Change, Before You Change It

An upgrade that changes data owes the operator a list, before anything is touched.

In our design the list comes from a separate command, upgrade check. It reads copies of your stores, or opens them read-only, and changes nothing, so it is safe to run while the old version is still serving. It reports what the real upgrade would do to your data: which buffers would move to keep until delivered, which records would be converted, and anything that needs a person’s decision. The rule behind it is short: nothing changes, and nothing disappears, without being reported first.

The second half of that promise is what happens if you skip the step. A version that finds older data than it understands does not try to cope. It refuses to start, says what it found, and names the command to run. From 3.1 its health check reports it as not ready, so an orchestrator does not mistake it for a working system. That refusal is deliberately inconvenient. The alternative is a system that starts on data it does not fully understand and makes guesses about it in production.

Back Up First, With Nothing Running

The order of an upgrade is not a detail. Check, stop, back up, convert, verify, start.

Stopping matters because a conversion and a running system should never write to the same store at the same time. Backing up matters because a conversion is code, and code has bugs; the only real undo is a copy taken before it ran. The upgrade command backs up the file stores it changes on its own. Databases it cannot snapshot itself, so where data lives in PostgreSQL, the command asks for the name of the snapshot the operator took and refuses to convert without one. If any step fails, it puts every store back as it was.

That is more ceremony than swapping a tag. It is also the only order in which the worst case is “restore the snapshot and start the old version again” rather than “work out which rows were changed by which start”.

The Backlog Is Not Ours to Drop

The moment an upgrade is most tempting is the worst moment for it to lose anything: when a destination is down and the edge is holding a backlog.

People upgrade in the middle of outages. Sometimes the upgrade is the fix they are waiting for. So the backlog has to come through the upgrade intact, and we tested exactly that. On an installation running 3.0, we cut the link to a destination and let a backlog build. While it was held, we stopped the system, ran the upgrade to 3.1 and started it again. The new version came up still holding the backlog, and when the link returned, it was delivered. The upgrade lost nothing and duplicated nothing.

One limit is worth saying plainly, because it follows from stopping the system. While MaestroHub is stopped it reads nothing from devices and sends nothing. Messages already buffered stay buffered; values a device produces during the stop are not read by anyone. The stop is short, and you choose when it happens. That is the trade.

Kubernetes Gets No Exception

A rule that holds on a single machine and dissolves on a cluster is not a rule.

On Kubernetes, the temptation to convert at startup is even stronger: pods restart all the time, and a migration hidden in startup looks like it comes for free. It does not, for the same reasons, multiplied by every replica that might run it.

For Enterprise installations on Kubernetes, 3.1 runs the upgrade as a Job that follows the same order. It checks first. It refuses to convert until the operator confirms that a database snapshot has been taken. It stops the services before it converts, so nothing writes while the data changes. It uses only MaestroHub’s own image, so a site that mirrors images for an air-gapped cluster has nothing new to mirror. And it is idempotent: run it again on converted data and it changes nothing.

We tested it the way an operator would meet it: a throwaway Kubernetes cluster built with kind, a real 3.0.0 installation, then the upgrade. Without a snapshot it refused and changed nothing. With one, it converted the old buffer default, left a hand-set retention alone, brought the services back, and did nothing on a second run.

The Trade-off, Honestly

This design costs you something, and I would rather say so than pretend it does not.

An upgrade that changes data is one explicit step instead of none. You read a report, you stop the system, you take a backup, you run a command. If you skip it, the new version refuses to start until you do. Some operators will find that annoying, and they are not wrong to.

What you get for it is the absence of surprises at boot. Every change to your stored data happens once, at a time you chose, after a backup you confirmed, with a list you could read beforehand and a record you can read afterwards. A release that changes no stored data needs no step at all; the ceremony appears only when there is something to be careful about. And a value you set by hand after the upgrade stays yours across every start that follows.

I think that is the right side of the trade for anything that sits between machines and the systems that depend on them. A surprise at boot does not happen on a quiet afternoon. It happens on whichever start comes next, which might be the one after a power cut, at night, on a site nobody is watching.

Where This Lives in MaestroHub

The upgrade command, its check, its backup and the startup that refuses data it does not understand arrived with 3.0, for installations coming from 2.6. 3.1 is the first release to use them between two 3.x versions: buffers move from the old 24-hour default to keep until delivered once and only once, every buffer record carries the version that wrote it, and Enterprise installations on Kubernetes upgrade through the Job. The 3.0 to 3.1 upgrade is one upgrade apply, after a backup, and the upgrade guide in the documentation walks through it for Docker Compose, docker run, the single binary and Helm.

The Question I Want to Leave You With

When you upgrade, who decides what happens to your data, and when?

If the answer is “the new version, at whatever start comes first”, then every restart is a small upgrade you did not schedule. I would rather a platform told me what it intends to change, waited for my backup, did it once, and then left my data alone.

If you are planning an upgrade and would like to walk through it with us, get in touch.