The Reading That Joined the Wrong Batch: Why Contextualization Is Not a Bigger JSON Payload

A Question That Should Take Five Minutes

Let me tell you about a question that should take five minutes to answer, and the confident wrong answer it got instead.

The plant in this story is a composite and the numbers are illustrative, but the pattern is one our industry keeps repeating.

On a Monday morning, a quality lead gets a complaint about coated parts shipped last week. The question is simple: which batches went through Oven 4 while it was running hot?

The data team is not worried. A year earlier they built a pipeline that does what most of the industry calls contextualization. Every temperature reading from Oven 4 is wrapped with the batch, the shift, the work order, the product and the operator before it is stored. Every message is complete and self-describing:

{
  "asset": "Oven4",
  "temperature": 220,
  "timestamp": "2026-11-02T10:04:00Z",
  "batchId": "Batch43",
  "shiftId": "Morning",
  "productId": "ProductA",
  "workOrder": "WO-1187"
}

They filter for temperatures above 200°C, group by batchId, and have an answer in seconds: Batch 43.

Batch 43 is quarantined. Batch 42 has already shipped.

The answer was wrong.

How a Correct Reading Joined the Wrong Batch

Nobody was careless. The pipeline did exactly what it was designed to do. The design was answering the wrong question.

Oven 4 really did reach 220°C at 10:04. That was during Batch 42.

But a network switch on the line dropped for a few minutes. The edge gateway held the readings and sent them when the link came back, at 10:21. At 10:20, the MES had already moved the line to Batch 43.

The enrichment step looked up the current batch at the moment the reading arrived, and stamped it on the message. A reading taken during Batch 42 was filed under Batch 43, permanently, in a perfectly formatted payload.

Then it got worse. At 10:30 the MES sent a correction: Batch 43 had actually started at 10:18, not 10:20. Thousands of enriched records had already baked in the old answer. Nobody knew which ones needed fixing, or that any did.

Figure 1How a correct reading joined the wrong batch. The pipeline stamped the batch that was current when the reading arrived, not the one that was running when it was measured. The 10:30 correction then changed history, and nothing downstream knew.

This is the misunderstanding I want to name, because it is everywhere in our industry:

Combining IT and OT data into one JSON payload is not contextualization. It is data enrichment.

Enrichment can be useful. Contextualization is a much bigger idea.

The Bill Nobody Sends You

Before the wrong answer, there was the cost of storing it.

Oven 4 produces 10 readings a second. That is 36,000 readings an hour. A batch lasts about 20 minutes, so the same batch ID is written roughly 12,000 times per batch. The shift ID, which changes every eight hours, is written 288,000 times per shift.

Now scale it. Take a hypothetical plant with 100,000 tags, each publishing 10 readings a second:


Readings per day	86.4 billion
Extra context per reading	100 bytes
Extra data per day	about 8.64 TB, before replication and indexes

Even if you can afford that, the question is why you would. The batch did not change 12,000 times. It changed once.

Figure 2Two ways to keep the same facts. On the left, context is copied onto each reading, so a batch that changed once is written thousands of times. On the right, each fact is stored at the rate it really changes and joined when someone asks.

There are good reasons to send fully enriched messages: downstream systems that need self-contained records, audit trails, stream consumers that cannot look context up themselves. Those are real. But they are delivery decisions, made for a specific consumer. They should not be the definition of contextualization.

What Does 220°C Actually Mean?

Without knowledge, 220 is just a number.

Go back to that reading. Now imagine the platform knows that:

Oven 4 belongs to Line 2.
Line 2 is producing Product A.
Product A requires 180°C to 200°C during a specific production phase.
Oven 4 is currently in its heating phase.
The current batch is Batch 42.
An excessive temperature exposure may require a quality inspection.

The important part is not that we could attach six more fields to the payload. It is that 220°C means something different depending on what was happening. During preheating it might be perfectly normal. During active production it is an excursion that puts a batch at risk.

Figure 3The number does not carry its meaning. 220 °C is fine while the oven preheats and a quality risk while Product A is in production. Meaning comes from the observation together with its relationships, its time and the rules that applied.

That is the difference between adding context fields and understanding context. The way I like to express it:

Meaning = f(Observation, Relationships, Time, Operational Rules)

This is not a formal industry definition. It is the model I find most useful when we design.

Three Layers, Three Speeds

Industrial information does not change at one speed, so it should not be stored as if it did.

Figure 4Three kinds of information, three rates of change, drawn over the same hour. Storing all three as if they changed at the rate of the fastest one is where the bill in the previous section comes from.

Layer	How often it changes	Examples	How it is kept
Observations	Many times a second	Temperature, pressure, vibration, current, speed, machine state	Timestamped values, streamed or stored
Operational context	When an event happens	Batch, work order, shift, product, recipe, production phase	Changes recorded with the time they became valid
Semantic knowledge	Rarely	Asset hierarchy, equipment capabilities, material relationships, operating rules, quality dependencies	Versioned definitions and relationships

These layers do not need to update at the same rate, and they do not need to live in the same storage engine. What they need is to be resolvable together whenever someone asks a question.

Connected, Not Copied

Back to Monday. The right question was never “what batch ID is on this message?”

The right question was: which batch was running on Line 2 at the moment Oven 4 measured 220°C?

To answer it, you need three facts, each stored once:

Figure 5The temporal join. Nothing is copied onto the reading. When the question is asked, the reading's event time is matched against the batch that was valid on its line at that moment. The answer is Batch 42, however late the reading arrived.

And one relationship: Oven 4 belongs to Line 2.

The answer is Batch 42. We did not need to append batchId to twelve thousand readings. We needed to know that Batch 42 was valid on Line 2 for a particular interval, and that Oven 4 is part of that line.

This is a temporal relationship, and it is not exotic. Stream processing has established patterns for it. Apache Flink, for example, supports event-time temporal joins, which match each incoming event against the version of changing context that was valid when the event happened.

Four Clocks Every Context Engine Has to Keep

The Monday mistake had a precise cause: it joined against the wrong clock.

Figure 6Four clocks, and why the fourth matters. The top half places each clock on the same morning. The bottom half is a bitemporal view: the same stretch of valid time as it was known before and after the correction. Keeping both rows is what makes the record auditable.

The pipeline in our story joined ingestion time against current context. The correct join is event time against effective time. That alone would have filed the 10:04 reading under Batch 42.

Key idea

Join event time against effective time, and keep knowledge time so you can explain later what you believed and when. Ingestion time is for operations, never for meaning.

The MES correction needs the fourth clock. When the MES says at 10:30 that Batch 43 really started at 10:18, the readings between 10:18 and 10:20 change batch. Now you have real decisions to make. Should earlier outputs be corrected? How are those corrections tracked? What should a consumer that already acted on the old answer see?

For true auditability, you need to keep both when a fact was valid and when it became known. Database people call this bitemporal modeling. And it exposes a quiet trap: a graph that knows today’s relationships cannot, by itself, explain last Tuesday. Relationships need time too.

Where MaestroHub Stands, Honestly

I would rather tell you what is shipped, what is in progress and what is still a direction than blur the three.

Shipped, in the 5 November 2026 release:

Store and forward replays buffered readings with the time they were produced, so a reading delayed by a network outage keeps its event time when it reaches its destination.
Every stored value records where it came from and its quality (good, uncertain or bad, with a reason).
Retention is an operational window by design: each topic keeps 1, 7, 14 or 32 days, one day by default. Long-term history belongs in your system of record.

Beta, in the same release: the semantic layer, which gives signals a meaning and a unit, so you can find signals by what they measure rather than by what someone named them.

In development, no date yet: the Knowledge Graph, a plant model in which every relationship carries a valid-from and a valid-to, so you can ask what the plant looked like at a past instant.

A direction, not built yet: the full contextualization architecture I describe next.

What I Would Build: Context as a Platform Capability

I would not throw away pipeline enrichment. I would make it one output of something more capable.

The shape I have in mind sits between the systems you already own and the applications that consume their data. Connectors bring in PLCs, OPC UA, MES, ERP and quality systems. A context layer holds asset identity, time-aware state, relationships and rules. From there, three kinds of output: the raw namespace for fast signals, a way to ask for context, and enriched streams for the use cases that genuinely need them.

Figure 7Context at rest, context at delivery. The context layer keeps each fact once; each consumer then chooses raw observations, context resolved on request, or a fully enriched stream. This is the direction I describe, not something MaestroHub ships today.

Five capabilities make that work:

A context registry. Asset identity, operational state and relationships from authoritative sources, with provenance.
Temporal context. When batches, recipes, shifts and relationships became valid, including corrections made after the fact.
Context resolution. Given an asset and a timestamp, return the operational and semantic context that applied.
Context-aware processing. Evaluate conditions like “temperature above the limit for the active recipe, during production”.
Optional materialization. Publish fully enriched records only when a consumer actually needs them.

Two principles keep this honest in a real plant.

Performance. High-rate pipelines should resolve context from a local in-memory cache with cheap lookups. A graph database must never become a mandatory network hop for every 100 millisecond reading.

Source of truth is not a single copy of everything. The MES stays authoritative for work orders, ERP for product definitions, the PLC for measurements. The platform’s job is to establish trustworthy identities, relationships, timelines and access across them.

If I had to compress it into one architectural decision, it would be this: separate context at rest from context at delivery. At rest, store facts at their natural rate of change, without duplication. At delivery, let each consumer choose raw observations, resolved context, or a fully enriched event.

That way contextualization becomes something the platform does once, reusably, instead of something every customer rebuilds inside every pipeline.

Three Levels of Knowing

Understanding data is not the finish line. Acting on it is.

Level	What the platform knows	What it enables
Data enrichment	Temperature 220°C, Batch 42	Filtering and dashboards
Semantic contextualization	Oven 4 is processing Product A, whose active recipe limit is 200°C	Interpretation that depends on the situation
Operational intelligence	Oven 4 exceeded the active limit for 40 seconds, and Batch 42 may need inspection	Decisions and workflows

The third level adds reasoning over time, rules and evidence, not just relationships.

This is also where industrial AI becomes real rather than decorative. An agent with governed access could be asked: which batches were potentially affected by temperature excursions in Oven 4 during the last 24 hours? To answer reliably, it needs historical measurements, batch timelines, recipe limits, asset relationships, and an agreed definition of what counts as an excursion. That is far more valuable than a large JSON payload, and no amount of payload enrichment provides it.

It also shows a boundary worth being clear about. An operational platform keeps a bounded window of raw measurements. For older questions, the evidence lives in your historian or data lake. The context layer can preserve the relationships without keeping every raw observation forever.

Test your own pipeline with five questions.

If a gateway holds readings for fifteen minutes during a network drop, which batch do they get when they arrive?
When the MES corrects a batch start after the fact, which stored records change, and can you list them?
How many times is the same batch ID written for one batch?
Can you say which batch was running on a machine at 10:04 last Tuesday, both as it was known then and as it is known now?
Can a consumer ask for context when it needs it, or does every consumer receive it copied onto every message?

Back to Monday

With context resolved instead of copied, the Monday question has a different ending.

The 10:04 reading is matched by event time against the batch timeline: Batch 42. When the MES correction arrives at 10:30, only the readings between 10:18 and 10:20 change batch, and the record shows both what was known at 10:21 and what is known now. The quality lead quarantines the right batch, and can explain why.

Nothing about the sensor changed. Nothing about the MES changed. What changed is that the platform stopped asking what batch ID should I attach? and started asking what was happening when this was measured, and what does it affect?

The Question I Want to Leave You With

What if contextualization is not mainly about combining data, but about establishing relationships that let data be interpreted correctly across time?

That changes what we build. It draws the boundaries between a pipeline engine, a unified namespace, a knowledge graph and a semantic layer.

And it leaves an open product decision I would genuinely like to hear your view on: should a platform contextualize observations automatically, or provide the context infrastructure so applications and agents can contextualize on demand?

My preference is a hybrid: on-demand resolution as the foundation, automatic enrichment for selected real-time use cases. I want to validate that against real customer workloads before we make it a platform-wide principle. If you run a plant where this question matters, I would like to hear how your workloads answer it.