A Green Checkmark Is a Promise
Every industrial pipeline meets a bad minute sooner or later: a cable someone unplugs, a proxy that restarts, a broker that loses its leader in the middle of a shift. What the pipeline does in that minute is the whole product.
Connector lists and feature tables are what we compare systems on. The promise underneath them is simpler. When the system tells you a reading was delivered, it was. When it tells you nothing was lost, nothing was.
My position is that this promise is worth more than any feature built on top of it, and that many pipelines break it quietly. Not with a crash. With a green checkmark that was slightly too optimistic. A false “delivered” costs nothing on the day it is said. It costs a great deal weeks later, when someone tries to explain a gap or a double count in a batch record and finds that every dashboard was green.
Store-and-forward is the part of a pipeline that holds data while a destination is away and sends it when the destination returns. Everybody has it. This article is about what it has to guarantee, which is a harder question, and about what we found when we held our own product to the answer.
What a Long Outage Taught Us
This autumn, a long-outage test of our 3.0 release cut the network between MaestroHub and a Kafka cluster for minutes at a time. It is a realistic test. Links fail for minutes, not milliseconds, and nobody restarts the edge box afterwards to tidy up.
The release did not hold up as well as it should have. Once an outage outlasted Kafka’s own delivery timeout, store-and-forward stopped moving. Some readings ended in a generic internal error instead of waiting. Reconnecting through the proxy took minutes after the link was back. The store for failed messages filled up.
None of this was a Kafka problem. The failures came from one mistake, made in code that all of our connectors share: we let time and connectivity decide what happened to a message. A clock decided that a record was too old to send. Each network error counted as an attempt, so an outage burned through a message’s retries. In every case the destination never said no. The machinery around it gave up on the destination’s behalf.
So we rewrote the rule, and everything in the rest of this article follows from it: only the destination decides a message’s fate.
An Outage Should Hold, Not Drop
An outage is time, not attempts. A bad message is attempts, not time. Mixing the two up is where most store-and-forward designs lose data.
While a destination cannot be reached, nothing it could say about any message is known yet. So nothing should be decided. The buffer holds, on disk, for as long as the disk has room, and no message is charged an attempt for the outage. In the new build there is no age limit anywhere, not in our code and not in the client libraries underneath it. Every record keeps the time it was produced, so a reading that waited an hour still arrives saying when it was measured.
Holding everything forever would be easy and wrong, though. Some messages are genuinely bad: a record the broker rejects as invalid, a row that breaks a constraint. If a pipeline that delivers in order held a bad message forever, everything behind it would wait forever too.
The line we drew is behavioural, not temporal. When the destination refuses one write while it accepts others, that refusal counts. In the new build, after ten counted refusals, with waits that grow from one second to a minute, the write is set aside in Failed messages with the destination’s own error, ready to be fixed and replayed, and the pipeline moves on. An outage spends nothing. A refusal from a destination that otherwise works spends an attempt.
Key idea
Never let an outage spend a message's attempts, and never let a clock expire a message nobody refused. The only voice that should decide a write's fate is the destination's.
Why Retries Turn Outages Into Duplicates
Fixing loss exposed the next problem, and it was worse, because it looked like success.
Once nothing was lost through a long cut, we counted the copies. A one-hour cut left 72 extra copies of a single reading at the destination.
To see why, you need one idea from distributed systems: the half-open link. A cable is cut on the far side of a proxy. Your machine can still write bytes into its socket. They leave. They may even reach the broker. But the answer never comes back. From the sender’s side, a delivered write and a lost write look exactly the same: silence.
In our case, the worker checked whether the destination was back by sending the oldest buffered message. On a half-open link the Kafka client kept each of those sends alive and delivered all of them the moment the link healed. MQTT and NATS showed the same pattern for a different reason: their libraries keep nothing, but our engine retried on a timeout, and over a link that delivers late every retry eventually landed. Twenty attempts, twenty-one copies. We also found one cloud library that quietly published the same message up to five times on its own.
The lesson is general. A retry on silence is not a safe operation. Whenever a sender resends something it did not get an answer for, it is betting that the first attempt was lost. Over a real network, that bet is sometimes wrong, and every lost bet is a duplicate.
Three Honest Promises
“Exactly once” across a network you do not control is not something a sender can promise on its own. What a sender can do is choose which mistake it is willing to make, and say so.
There are three classic delivery promises, and it helps to see what each one does with the same silence.
- At most once never retries. Nothing is ever duplicated, and anything lost in silence stays lost. It is the right promise only when there is no acknowledgement to wait for at all.
- At least once retries until the destination confirms. Nothing confirmed is lost, and a write that was in flight when the link failed may arrive twice.
- Effectively once is at least once plus a destination that can tell a repeat from a new write, and drops it or overwrites it. The pipeline still retries; the copy simply has nowhere to go.
Which promise you can keep is not a property of your pipeline alone. It depends on the destination. A deduplicating stream, a SQL table with a key column, an upsert store or an incident tool with a deduplication key can keep effectively once, within its deduplication window. A plain message queue cannot, so the honest promise there is at least once. A UDP datagram has no acknowledgement and no way to say the receiver is down, so buffering it would be theatre: the honest promise is at most once, and the honest product says so instead of pretending.
My position here is simple. Every destination should state its promise, and only the promise it can keep. In the MaestroHub build now in final review, each of the 114 connector operations that can buffer declares one of these three. The promise is shown on the pipeline node, published in the documentation, and held by a check in our build that fails when a connector claims more than its tests show.
The Write Key: Identity That Travels
Effectively once is impossible without one thing: a stable identity for every write.
A destination can only drop a repeat if it can recognise one. Comparing the content does not work, because two readings of the same sensor can legitimately carry the same value. What works is a write key (often called an idempotency key): an identity the pipeline gives a write once, and which stays the same through every retry, every replay after an outage, and every operator action, all the way into the destination.
How the key travels depends on the system. In a stream it is a message header or a message ID. In a database it is a key column. In a document store it is the document ID. In an incident tool it is the deduplication key. The shape changes; the rule does not. The key is minted once and never minted again for the same write.
The key earns its place even where the destination cannot use it. Under at least once, a copy can still arrive, but it arrives carrying the same key as the original, so any consumer that cares can drop it with one comparison. Without a key, a duplicate reading and a genuine second reading are indistinguishable forever.
The key also fixes the sender’s own side. In our rebuild, a ledger at the one place every write passes through allows at most one open attempt per write. A resend of a write whose earlier attempt has not been answered does not go out on the wire. It waits for that attempt’s answer. And when a buffer needs to know whether a destination is back, it asks with a round trip that carries no data, instead of sending the oldest message to find out. After those changes, the same one-hour cut that had produced 72 extra copies produced none, and the copy count no longer grows with the length of the outage.
Key idea
A retry without an identity is a guess. A retry with a write key is a question the destination can answer: "have you seen this one before?"
“Possibly Sent” Is an Honest Answer
The most important word we are adding to the product is not “delivered”. It is “possibly sent”.
Every attempt to deliver a write ends one of three ways, and the system has to know which.
Delivered means the destination confirmed it. Refused means the destination said no, or the write never left the machine, so retrying is safe. Possibly sent means there was no answer. The write may have landed. A system that rounds this third outcome up to “delivered” can lose data without knowing. A system that rounds it down to “failed” and resends creates the duplicates of the previous section. Both are guesses dressed as facts.
There is a fourth, quieter case worth naming. On a plain TCP or syslog connection, the operating system accepts the bytes and that is all anyone will ever know. The write was sent, and nothing confirmed it arrived. The new build calls it exactly that, “sent, not confirmed”, and counts it separately. A report about an outage on such a destination does not say “nothing lost”. It says “none lost before sending”, because that is the truth.
This may sound like a weaker product. I think it is the opposite. “Possibly sent” is not a weaker form of “delivered”. It is the reason you can believe “delivered” when you see it.
Run the Drill Before the Outage Runs You
Claims like these are cheap without a way to break them. The only real test of store-and-forward is an outage, so cause one, on purpose, and count.
Most teams find out how their pipeline behaves in an outage during an outage, which is the worst possible moment to learn. A drill moves that lesson to a quiet afternoon. The idea is old and simple: cut one destination deliberately, let data keep flowing, restore it, and then check the books.
What makes a drill worth running is what you count afterwards. Not the screen, not the logs, but a ledger that has to close:
held = delivered + still waiting + in Failed messages + lost
Every record the buffer took in has to land in exactly one of those boxes. If the two sides of the equation differ, something is unaccounted for, and that is your finding. And the strongest form of the check reads the records back from the destination itself, then counts what is missing, what arrived twice, and what arrived out of order.
In the build now in review, a MaestroHub drill cuts one output of one pipeline for a set time, never a whole connection, and always ends by itself. Other pipelines on the same connection carry on. It runs through the real delivery path, so what you watch is what an outage would do, and it ends with a report like the one above.
You do not need our product to start, though. You need the questions.
Questions for your own pipeline, before the next outage asks them for you.
- If a destination is down for an hour, does anything expire by age, or does a clock decide what is dropped?
- Does an outage spend a message's retries, or only a real refusal from the destination?
- When an attempt gets no answer, what does your system call it: delivered, failed, or something honest in between?
- Does every write carry a key that survives retries and replays, and can your destination use it to drop a repeat?
- Which promise does each of your destinations actually keep: at most once, at least once, or effectively once? Is it written down?
- After your last outage, did held equal delivered plus waiting plus failed plus lost? Did anyone check?
What Holding Ourselves to It Found
We built a harness that is not gentle: a proxy that can refuse, reset, swallow or hang a link, and a ledger that reads every record back from the real destination.
We ran it against real engines in containers: three Kafka brokers behind a proxy, a three-node NATS cluster with full disks, a process killed with kill -9 in the middle of an outage, a duplicate rig for every protocol that can buffer but two, and real-time cuts of fifteen minutes and an hour. Then we drove the real product through its own interface, broke the network under it, and read every record back from Kafka.
The results we wanted are boring, which is the point:
- Nine scenarios through the real product, eight of them with something broken (cuts of two and fifteen minutes, a cut longer than Kafka’s own delivery timeout, a proxy reconnect and restart, a bad message, a crash mid-outage, a drill): 0 missing, 0 duplicates, 0 out of order. The outputs were set to deliver in order.
- The one-hour cut in the test harness: none missing, none duplicated, none out of order, where the same test before the fix had left 72 extra copies of one reading.
- Network checks: 11 connection checks across the product answered “connected” from a local flag that stays true through a black hole. In the new build they make a real round trip.
The harness found more than thirty real bugs on the way, and several of them were in the new code itself, written earlier in the same effort. An in-order buffer that would stall forever after a crash if more than 128 writes were queued behind the one in flight. A client-library iterator leaking on every drain. SQL drivers that hung for forty minutes on a silent link while logging in. A TCP write sent twice because “sent” was read as a failure. Each one was reproduced as a failing test before it was fixed.
I find that last part the most important. A delivery guarantee that has never been run against its own failure modes is a hypothesis, not a guarantee.
Where This Stands, Honestly
What is in the 5 November 2026 release: per-destination buffers on disk, a delivery order you choose per output, a store for failed messages, and an outage simulator (the store-and-forward page lets you pull the cable in a simulation).
What this article describes beyond that (outages that hold without expiry, counted refusals, write keys, the attempt ledger, the three declared promises, “possibly sent”, the drill and its report) is built and tested, and is in final review for the next release. The screenshot above is from that build, on our demo plant.
Three things worth saying plainly. First, delivery order is a choice per output: in order when the output is set to In order or its connector asks for it, otherwise newer messages may overtake a refused one. Second, at least once means what it says: a write in flight at the moment a link half-fails, or a process dies, may arrive one extra time, carrying its write key, so a consumer that cannot tolerate a copy should drop it by key. Third, a buffer holds as long as there is disk; how long that is depends on your data rate and your disk, and the new build warns you when room runs low rather than promising a number of days.
The Question I Want to Leave You With
When your system says “delivered”, how does it know?
If the honest answer is “because nothing went wrong that we noticed”, then the green checkmark is a hope, not a fact. I would rather a pipeline held what it cannot send, counted what it gives up, refused to call a guess a delivery, and told you which promise each destination can keep. That is less comfortable on the dashboard. It is the only kind of “delivered” you can build a batch record on.
If you would like to see a drill run against your own destinations, or compare notes on how you test yours, get in touch.