Real-Time vs Batch Reporting: Choosing the Right Data Refresh Strategy for Your UK Business

Ask a UK management team what they want from a new reporting platform and somebody will say live data. It sounds like the obvious specification, and real-time reporting for a UK business is genuinely transformative in a small number of situations — payment fraud screening, live stock availability, a contact-centre wallboard, a production line that can be stopped. In the large majority of cases it is an expensive answer to a question nobody actually asked, bolted onto an organisation that has not yet put a “data as at” timestamp on the reports it already has.

This guide separates the two. It explains what batch, micro-batch and streaming refresh actually mean at the infrastructure level rather than in a vendor diagram, costs all three for a UK SME with real build and run figures, and works through which business functions genuinely need sub-minute data against those that are better served by an hourly or nightly load. It covers the load that live reporting places on your operational database, the reconciliation problem that quietly destroys trust in fast-moving numbers, late-arriving data, the shrinking overnight batch window, and the GDPR consequence of copying personal data into a second store in order to report on it faster. It ends with a decision framework built on decision latency — the only measure that matters — rather than on how modern an architecture sounds.

What batch, micro-batch and streaming actually mean

Batch reporting is a scheduled job that runs at a fixed time, reads a defined set of records from your operational systems, transforms them, and writes the result into a reporting store. In a typical UK SME this is a SQL Server Agent job, a Windows scheduled task running a stored procedure, an Azure Data Factory or Microsoft Fabric pipeline, or dbt invoked from a scheduler. It usually runs once a night between one and four in the morning, it moves either everything or everything changed since a high-water mark, and it either succeeds or fails as a unit. That last property is the underrated one: a batch job can be re-run. If Tuesday night’s load was wrong, you fix the logic and run it again, and Tuesday is correct.

Micro-batch is the same mechanism with the interval shortened to somewhere between five and thirty minutes, which forces one change: you can no longer reload everything, so the job must be genuinely incremental. It tracks what has changed using a modified-date column, a rowversion, SQL Server change tracking, or an incremental refresh policy in the semantic model. Micro-batch is where most businesses that think they need real-time actually end up, and it is materially cheaper and simpler than streaming while delivering latency that no human decision process can distinguish from live.

Streaming, or true real-time, inverts the direction of travel. Instead of a reporting process asking the source what has changed, the source emits each change as it happens — through Change Data Capture reading the database transaction log, or through the application publishing events directly — onto a message bus such as Azure Event Hubs, Service Bus or Kafka. A stream processor consumes that flow continuously and maintains the reporting state. There is no job, no window and no run; there is a pipeline that is either healthy or unhealthy, and when it breaks it breaks silently. Separately, most BI tools offer a live-query mode — DirectQuery in Power BI is the common one — which is not streaming at all but rather pushing every dashboard interaction straight through to the source database as a query.

Those four options — nightly batch, micro-batch, streaming, live query — have different costs, different failure modes and different effects on the systems your business runs on. Choosing between them is an engineering decision with a commercial answer, and the commercial answer almost always turns on one question: how quickly can somebody actually act?

The numbers that frame the choice

Four figures explain why this decision goes wrong so consistently. They are drawn from what we find when assessing reporting estates in UK businesses of roughly twenty to two hundred staff, and from the published limits of the platforms those businesses actually run.

The first card is a hard platform constraint rather than an opinion, and it shapes more UK reporting architectures than any deliberate design decision. A Power BI Pro licence permits eight scheduled refreshes a day on an import-mode semantic model, which works out at roughly two-hourly during a working day. Premium Per User and capacity-backed workspaces raise this to forty-eight, or every thirty minutes, and Fabric capacities add Direct Lake as a further option. Anything faster than that requires a different mechanism entirely — DirectQuery, a push dataset, or a streaming architecture — which means the jump from “every two hours” to “live” is not a slider you drag but an architectural change with a licence cost attached.

The second card is the one that surprises boards. A nightly batch that loads eight tables into a reporting schema and feeds a report pack is a few weeks of work for a competent developer. The streaming equivalent requires change data capture configured on the source, a message bus, a consumer that handles out-of-order and duplicate events, idempotent writes, replay capability for when the consumer was down, freshness monitoring, and an on-call owner. The functional output looks similar on a screen. The engineering underneath is a different discipline, and the ongoing ownership burden never goes away.

The third and fourth cards belong together, and read in sequence they are the argument of this entire guide. Roughly one report in nine that gets specified as real-time has a decision behind it that genuinely needs sub-hourly data. Meanwhile three quarters of all reports do not tell the reader how old the numbers are. The second problem costs nothing to fix and destroys more decisions than the first one ever will, because a figure of unknown age is worse than a figure known to be twelve hours old — a reader who knows the age can compensate, and a reader who does not will either over-trust it or quietly stop using the report.

How fresh does each function actually need its data?

The chart below distributes the reports we assess by their genuine decision latency: the shortest interval within which a named person could and would act differently because of the number in front of them. This is not what stakeholders ask for during requirements gathering. It is what survives the question in the tip above.

Four per cent of reporting genuinely needs sub-minute data, and it is worth being precise about what sits in that band, because it is not really reporting at all. It is automated action: a payment declined by a fraud rule, an order blocked because the last unit sold thirty seconds ago, an alarm raised because a temperature crossed a threshold, a customer told their delivery is eight minutes away. In every one of those cases the consumer of the data is a system, not a person, and the requirement is a transactional or event-driven feature of the application rather than a dashboard. Businesses that recognise this early build it into the application and stop trying to solve it in the BI layer, which is the wrong tool and always will be.

The nine per cent in the one-to-fifteen-minute band is where human real-time lives, and it has a consistent signature: somebody is watching a screen continuously as part of their job. A contact-centre team leader watching queue depth and longest wait. A despatch supervisor watching the pick-and-pack pipeline against the carrier cut-off. A production coordinator watching line output against target. These are wallboards rather than reports, they are looked at for hours rather than opened and closed, and they justify micro-batch or streaming because the cost of a fifteen-minute lag is a bad decision made in the meantime.

Everything from the hourly band downwards — eighty-seven per cent of the total — is served perfectly well by scheduled refresh, and the majority of it by a single nightly load. Finance and management reporting belong here by nature, not by compromise: a revenue figure is not meaningfully more useful at 11am than at 9am, and it is considerably more useful once credit notes, returns and journal corrections have landed. Marketing analytics, HR metrics, pipeline reporting, cohort and retention analysis, budget against actual, and board packs all sit in the same territory. So does most of what businesses describe as their KPI dashboard, which is typically opened on a Monday morning and after month end.

There is one important caveat to reading this chart as a licence to do nothing. A report on a next-morning cadence still needs to actually arrive by next morning, reliably, with a visible timestamp and an alert when it does not. A nightly batch that silently failed on Thursday and served Wednesday’s numbers on Friday has done more damage than a fifteen-minute lag ever would, and that failure mode is far more common in UK SMEs than any latency problem.

What each refresh strategy actually costs

The table below costs the four options for a representative UK SME scenario: roughly 120 staff, a line-of-business system on SQL Server, a Microsoft 365 estate, Power BI as the reporting tool, and a requirement for around fifteen reports across finance, sales, operations and stock. Build figures assume UK development day rates of £650 to £950. Run costs exclude the Power BI user licences themselves, which you pay either way.

The build column is the one that gets negotiated and the run column is the one that gets forgotten, but the ownership column is what determines whether the thing still works in two years. A nightly batch is a job that either ran or did not, and a single email alert on failure is adequate governance. A streaming pipeline is a live system with its own failure modes — consumer lag, poison messages, schema drift when a developer adds a column to the source, replay after an outage — and it needs somebody whose job includes noticing. UK SMEs that build streaming pipelines without allocating that ownership do not usually discover the gap through an outage; they discover it eighteen months later when the person who built it has left and nobody will touch it. The same structural question sits behind every technology decision of this kind, and our comparison of in-house IT against managed support works through the staffing arithmetic in detail.

A note on platform licensing, because it changes the shape of the answer more than most architecture choices. Power BI Pro sits at roughly £11 per user per month at 2026 list prices and Premium Per User at roughly £19, both per named user; confirm current figures before you budget, as Microsoft has revised them. The alternative is a Fabric capacity, where an F64 or larger removes the need for every report consumer to hold a Pro licence — which is transformative for a business with eighty read-only viewers and punitive for one with twelve. F64 lists in the region of £5,000 to £7,000 a month depending on reservation, so the crossover sits at roughly four to five hundred viewers on Pro pricing. Below that, buy licences. The frequent mistake is moving to a capacity to obtain faster refresh and then discovering the capacity was justified on viewer count or not at all.

Streaming run costs deserve unpacking because they are made of several small meters rather than one line. Azure Event Hubs is charged by throughput unit and ingress events, Stream Analytics by streaming unit hour, and a continuously running consumer on Container Apps or Functions Premium has a floor cost whether data is flowing or not. Individually these look trivial — tens of pounds a month each. Together, with non-production environments included, a modest streaming pipeline lands between £750 and £2,900 a month before anybody has written a report. Keep everything in UK South or UK West and confirm current unit pricing directly, as these meters change more often than the architecture does.

Batch against streaming, side by side

The comparison below sets a scheduled or micro-batch approach against a genuine streaming architecture on the criteria that matter operationally rather than the ones that appear in vendor material. The highlighted column is the one that suits the large majority of UK SMEs, which is a statement about typical requirements and not a claim that streaming is wrong.

Two rows in that comparison are doing most of the work. Failure visibility is the practical argument against streaming for organisations without a data engineering function. A batch job that fails sends an email; somebody investigates before lunch. A streaming consumer that has fallen forty minutes behind, or has been quietly rejecting every message since a source column was renamed, looks exactly like a business in which nothing has happened. Dashboards showing a flat line are indistinguishable from a quiet Tuesday, and the discovery mechanism is usually a manager asking why a number looks wrong — days later. Streaming can be monitored properly, with consumer lag alerts and freshness objectives, but that monitoring is part of the build and it is the part most often cut when the budget tightens.

Reconciliation is the row that causes the most organisational damage and gets the least attention at design time. Finance figures depend on cut-off points: a day closes, a period closes, and the numbers inside the boundary stop moving. A streaming pipeline has no boundaries by design — it always reflects the latest state, including the credit note raised four minutes ago and the invoice that will be voided this afternoon. The result is a dashboard whose revenue figure never matches the accounting system, which is not a rounding problem but a definitional one. Once two systems disagree about revenue, the organisation stops trusting both, and rebuilding that trust takes considerably longer than building the pipeline did.

The honest summary is that streaming is the correct choice when a system must react, when a human is watching a screen continuously, or when the cost of a fifteen-minute lag is measurable in money. It is the wrong choice when the underlying requirement is that reports feel modern, when the real complaint is that the existing nightly job is unreliable, or when nobody has yet agreed what revenue means. The second case is by far the most common, and fixing reliability on a nightly batch costs a fraction of replacing it.

Which business functions genuinely need live data

This is the section to take into the requirements meeting. The three cards below sort the reporting that UK businesses actually commission into the refresh cadence each function genuinely warrants, based on whether a faster figure changes a decision that someone can act on. The badge records the justified cadence, not the requested one.

The first card has a property worth naming explicitly: almost nothing in it is a report. Fraud screening is a decision made by code inside a payment flow. Live stock availability is a field read by a website at the moment a customer adds to basket. Alarm conditions raise a notification. These are application features, and they should be built as transactional behaviour in the application or as event-driven logic alongside it — not as a dashboard that a person is expected to watch. Businesses that try to satisfy this card through their BI platform end up with something that is both slower than the application needs and more fragile than a dashboard should be. Where genuine event-driven logic is required, our guide to testing automated systems for reliability covers the discipline that keeps this kind of always-on component trustworthy.

The middle card is where most of the disappointed real-time requests actually belong, and the good news is that it is cheap. Everything in it is satisfied by a fifteen-minute incremental load, which is an ordinary engineering task on a Fabric or Data Factory pipeline and needs no message bus, no change data capture and no on-call rota. The distinction that matters in this band is whether the consumer is watching continuously. A despatch supervisor with the screen open all afternoon needs the auto-refresh to be genuine; a purchasing manager who checks stock three times a day does not notice the difference between five minutes and an hour.

The third card is the one people argue with, and the argument is almost always about status rather than utility. A finance director who asks for real-time management accounts is not usually describing a decision they would make differently at 11am. They are describing a frustration — that the numbers arrive late in the month, that they do not trust them, or that producing them takes three days of manual work. Those are real problems and none of them is a latency problem. Faster refresh on an unreliable, unreconciled, manually-assembled pack produces wrong numbers sooner.

One genuine exception sits inside the third card and deserves a mention: period end. During the two or three days of a month-end close, the useful cadence for a small number of finance reports rises sharply, because the whole point of those days is iterative correction and re-checking. The sensible pattern is not a permanently faster pipeline but a manual refresh button on a handful of models, available to the finance team during close. It costs almost nothing to provide and removes most of the pressure for an architectural change.

What a refresh strategy project actually looks like

The timeline below describes a reporting refresh engagement for a UK business of around a hundred to two hundred staff, delivering a reporting store, a defined refresh cadence per report, and the monitoring that keeps it honest. It assumes a single primary line-of-business system plus finance, which is the common shape.

The sequencing in that timeline is the part to defend under pressure. Correctness at a slow cadence first, then monitoring, then selective acceleration. The common failure is to invert it — to build the fast pipeline first because it is the interesting engineering, and to discover in week ten that the numbers were never reconciled and nobody agreed what an active customer was. Fast wrong data is more dangerous than slow wrong data, because its apparent immediacy discourages the sanity check that a twelve-hour-old figure invites.

Where UK businesses stand on reporting practice

The figures below are what we find when assessing reporting estates at the point a business is asking about real-time data. They describe the starting position, and the reason they appear in a guide about refresh strategy is that almost every row represents a problem that faster data makes worse rather than better.

The gap between the 42 per cent reading a replica and the 58 per cent that do not is the most immediate technical risk in this list. Reporting queries run against a live transactional database compete with the application for locks and resources, and the symptom is not a slow report — it is an order entry screen that times out at 4.45pm while somebody refreshes a year-to-date sales analysis. The fixes are well established and cheap: a readable secondary in an availability group, an Azure SQL geo-replica, snapshot isolation so readers do not block writers, or simply moving reporting to a copy loaded overnight. Anyone contemplating DirectQuery against production should treat this as a prerequisite rather than a refinement, because live query mode multiplies the number of queries the source receives by every filter click a user makes.

The 17 per cent alerting on staleness rather than job failure is the row that most often explains a loss of confidence in reporting. Job-failure alerting catches the pipeline that crashed. It does not catch the pipeline that ran successfully and loaded zero rows because a source credential expired, a view was renamed, or a high-water-mark comparison silently matched nothing. That failure serves yesterday’s numbers with today’s confidence, and it can persist for days. The check is trivial — assert that the maximum timestamp in the target table is within the agreed threshold, and alert if it is not — and it should exist for every pipeline regardless of cadence.

Read the 71 per cent at the top against the 33 per cent with an agreed definition of revenue and the tension is clear: most organisations broadly trust reports that have never been reconciled against a definition anybody wrote down. That trust is provisional, and it collapses the first time two reports disagree in front of a board. Accelerating the pipeline before closing that gap simply means the disagreement surfaces faster.

How often is real-time actually justified?

Of the reporting requirements that reach us explicitly specified as real-time or live, the proportion where the decision mapping supports a genuine sub-hourly need is small and remarkably consistent across sectors.

Eleven per cent is not an argument that stakeholders are wrong to ask. It is an observation about vocabulary. When a sales director asks for real-time sales figures, the request that survives translation is usually one of three things: they want the figures to be correct, they want them without waiting three days for somebody to build a spreadsheet, or they want to stop hearing a different number from every department. All three are legitimate and none is about latency. The word real-time is simply the available shorthand for “better than what I have”.

The practical consequence is that the requirements conversation should never start with a refresh frequency. It starts with the decision, the owner and the time of day. The frequency is an output of that conversation, and when it is derived rather than requested, roughly nine requirements in ten resolve to a cadence that existing platform licences and a scheduled pipeline already support — which means the budget can go on correctness, reconciliation and monitoring instead of infrastructure.

The remaining eleven per cent are real and should be built properly. The mistake in the other direction is to dismiss a genuine live requirement because most are not — a business losing sales because its website oversells stock has a problem that a nightly load cannot solve at any price. Identifying that minority accurately, and building it as a narrow, well-owned piece of engineering rather than as a general-purpose streaming platform, is the whole of good practice here.

The 12-point refresh strategy checklist

Work through these before committing to any architecture. Each one changes the answer, and every one is cheaper to establish now than to discover in month four of a build.

1. Map the decision, not the report. For every report, name the person who acts, the action they take, and the earliest hour of the day they could take it. Any requirement that cannot survive this question does not have a latency need behind it.
2. Agree the definitions in writing first. Revenue, margin, active customer, open order, completed job. One owner per definition. No refresh cadence compensates for two departments meaning different things by the same word.
3. Retire before you build. Audit which reports were actually opened in the last quarter. A third to a half of a typical estate is dead, and carrying it forward into a new platform multiplies the cost of every subsequent change.
4. Separate reporting from the operational database. A replica, readable secondary or loaded reporting store. Reporting queries against production will eventually collide with the business, and the collision always happens at the busiest hour.
5. Check what each source can actually support. Reliable modified-date columns, rowversion, change tracking or CDC availability. A source with no dependable change marker cannot be loaded incrementally at any cadence, regardless of what the reporting tool can do.
6. Handle deletions explicitly. A high-water-mark incremental load never sees a deleted row, so deleted records persist in the reporting store indefinitely. Decide now whether you use soft deletes, change tracking, or a periodic full reconciliation to catch them.
7. Reconcile against the ledger and document how. Row counts and values, source against target against the accounting system, for a defined period. Write the reconciliation as a re-runnable query, not a one-off spreadsheet check at go-live.
8. Put a visible timestamp on every report. “Data as at 03:14 today”. This is the highest-value hour of work in any reporting project and it is omitted from three quarters of estates.
9. Alert on staleness, not only on failure. Assert that the newest record is within the agreed threshold. This catches the successful job that loaded nothing, which is the failure mode that erodes trust fastest.
10. Check the licence implications of the cadence you want. Eight refreshes a day on Power BI Pro, forty-eight on Premium Per User or a capacity. Establish whether a capacity is justified on viewer numbers before justifying it on refresh frequency.
11. Assign a named owner and a freshness objective per pipeline. State the acceptable lag explicitly — for example, sales data no more than 20 minutes old during business hours — so that performance is measurable rather than a matter of opinion.
12. Consider the data protection scope of every copy. A pipeline that replicates personal data into a second store expands your record of processing activities, your retention obligations and your breach surface. Copy only the fields the reports need, keep the pipeline and store in UK South or UK West, and apply a retention rule to the reporting copy as well as to the source.

Scoring your reporting estate before you change the cadence

The gauge below shows where a typical UK business scores when we assess its reporting estate at the point it starts asking about real-time data. The scale measures readiness to benefit from faster data: definitions agreed, sources reliable, reconciliation in place, monitoring present, ownership assigned. A high score means faster refresh will deliver value. A low score means faster refresh will deliver the existing problems sooner.

A score in the high thirties has a consistent composition, and the composition is more instructive than the number. Data availability scores well — the records exist, the systems hold them, and somebody can produce a figure on request. Report coverage scores moderately, because most businesses have accumulated something for every function even if it is a spreadsheet. Definitional agreement, reconciliation and freshness monitoring score badly, and report ownership scores worst of all, because reports are typically created by whoever needed them and then inherited by nobody.

What makes a low score here different from most maturity benchmarks is that the cheap remedies are also the valuable ones. Publishing a timestamp is an hour. Writing a staleness assertion is an afternoon. Agreeing what revenue means is a two-hour meeting that nobody wants to chair. Retiring dead reports is a morning and it makes everything afterwards faster. Together these might move a score from thirty-eight to sixty-five for a few thousand pounds, and a business at sixty-five will get real value from raising cadence on the handful of reports that justify it. A business at thirty-eight that spends eighty thousand pounds on a streaming pipeline has bought a faster route to the same disagreements.

There is a specific trap for organisations scoring below about thirty-five. At that level the reporting estate is usually a set of spreadsheets maintained by two or three capable people, and the numbers are right because those people check them. Replacing that with an automated pipeline at any cadence removes the human check without necessarily replacing it with a mechanical one, and accuracy can genuinely fall in the first months of a well-intentioned modernisation. The reconciliation step in the checklist exists precisely to prevent that, and it is the step most often dropped when a project runs late.

Common mistakes when choosing a refresh strategy

These are the errors we see repeatedly. Each produces an architecture that looks defensible at sign-off and causes trouble within a year.

• Treating real-time as a requirement rather than a translation. When a stakeholder says live data they usually mean correct, timely and consistent. Building a streaming pipeline in response answers a question that was not asked and leaves the actual complaint — three different revenue figures — entirely untouched.
• Pointing DirectQuery at the production database. Live query mode sends a query to the source for every visual and every filter interaction. On a transactional database serving the business this produces lock contention at exactly the wrong time of day. If you want live query, point it at a replica and tune the model, or do not use it.
• Building incremental loads without handling deletions. A modified-date watermark cannot see a row that no longer exists, so cancelled orders and deleted records live on in the reporting store for ever. The report slowly diverges from reality in a way that looks like a data quality problem and is actually a design omission.
• Alerting on job failure and calling it monitoring. The damaging failure is the job that succeeds while loading nothing. Without a staleness check on the target table, that state is invisible and can persist for days while everyone reads stale numbers with full confidence.
• Accelerating before reconciling. Fast wrong numbers are more dangerous than slow wrong numbers, because their immediacy suppresses the instinct to sanity-check. Correctness at a slow cadence, then monitoring, then selective acceleration — in that order, every time.
• Letting one live requirement dictate the whole architecture. A single wallboard needing fifteen-second refresh does not justify rebuilding thirty nightly reports as streams. Build the demanding artefact separately with its own narrow data path and leave the rest alone.
• Ignoring the licence ceiling until late. Discovering in user acceptance testing that the eight-refresh Pro limit cannot deliver the agreed hourly cadence forces either a rushed capacity purchase or a climbdown on the requirement. Check the platform ceiling during design, not during testing.
• Copying personal data wholesale into the reporting store. A pipeline that lifts entire customer tables because it was easier than selecting columns expands your GDPR footprint, your retention obligations and your breach exposure for no analytical benefit. Reporting rarely needs full postal addresses, dates of birth or free-text notes.

Mini case study — a Bristol plumbing and heating merchant

A builders’ and plumbing merchant with six branches across the South West, 140 staff, and a trade counter alongside a growing online business. The board had approved a data project on the basis of real-time dashboards after the sales director and the operations director spent a quarter arguing about branch performance figures that did not agree. The existing arrangement was a nightly extract from the trade ERP into a SQL Server reporting database, feeding eleven Power BI reports built over four years by three different people, plus around fifteen spreadsheets maintained by the finance team.

The decision mapping in week one produced an uncomfortable result. Of thirty-one artefacts in the inventory, fourteen had not been opened in the previous quarter. Of the seventeen that had, exactly two had a decision behind them that needed data fresher than the following morning: the despatch progress screen against the carrier collection at 4pm, and online stock availability, which was oversell-prone because the website read a stock figure refreshed nightly while the trade counter sold the same units during the day. Everything else — branch performance, margin analysis, aged debt, purchasing, the board pack — resolved to nightly or weekly. The revenue disagreement turned out to have nothing to do with timing: one report counted an order at despatch and the other at invoice, and nobody had written either definition down.

The delivered solution reflected that mapping rather than the original specification. The nightly load was rebuilt against agreed definitions and reconciled to the ledger; every report gained a visible “data as at” stamp and a staleness alert; fourteen dead artefacts were retired. Despatch progress became a fifteen-minute incremental load. Online stock availability was moved out of reporting entirely and rebuilt as an application-level reservation check against the ERP, because a dashboard was never the right mechanism for preventing an oversell. Finance received a manual refresh button for use during month-end close.

We came in asking for live dashboards and left with two things that update during the day and a lot of things we agreed to stop producing. The argument we actually wanted to end was about definitions, and that cost us a two-hour meeting rather than the six-figure platform we had budgeted for.

The commercial shape is worth recording. The approved budget had assumed a streaming platform at around £95,000 to build. Delivered cost was £31,000 including the application-level stock reservation work, with annual run cost of roughly £2,800 on top of existing licences, and no new capacity purchase because the justified cadences sat inside the Power BI Pro refresh ceiling. Oversell incidents on the website fell to near zero, which was the only measurable commercial problem in the original list and the one item that genuinely needed live data.

Two things did not go smoothly, and both are typical. Retiring the fourteen unused artefacts met more resistance than any technical decision, because people who had requested a report years earlier experienced its removal as a loss regardless of usage data; the resolution was a three-month archive rather than deletion, after which nobody asked. And the fifteen-minute despatch load initially used a modified-date watermark that could not see cancelled lines, so cancelled orders sat on the screen until a full reconciliation was added — exactly the deletion-handling omission described in the checklist, found in the first fortnight of live running.

At a glance — batch, micro-batch and streaming

If you take one row from that table, take the last one. Every other factor is a consequence. The refresh cadence of a report is not a property of the data, the platform or the fashion in reporting architecture; it is a property of the decision the report exists to support. Derive it from that and the architecture follows, usually more cheaply than expected.

How Cloudswitched approaches reporting refresh

We start with the report inventory and the decision mapping rather than the platform, because that exercise is what determines the cost of everything afterwards. From there we build the reporting store, agree and document the measure definitions with finance and operations, reconcile the output to the accounting system, and add the timestamps and staleness alerting that make the numbers usable. Cadence is raised selectively on the reports whose mapping justifies it, and genuine live requirements are built as narrow, separately-owned components rather than by converting the whole estate.

Frequently Asked Questions

What is the difference between real-time and batch reporting?

Batch reporting runs a scheduled job that reads data from source systems at a fixed time, transforms it and writes it to a reporting store — typically nightly, so reports show a position that is eight to fourteen hours old. Real-time or streaming reporting inverts this: the source emits each change as it happens, through change data capture or application events, and a continuously running consumer keeps the reporting state current within seconds. Between them sits micro-batch, an incremental load every five to fifteen minutes, which is where most requirements described as real-time are genuinely satisfied at a fraction of the cost.

How often should a business intelligence dashboard refresh?

Derive it from decision latency rather than choosing a frequency. For each report, identify who acts on it, what they do, and the earliest point in the working day they could do it. If a despatch manager reviews figures at 8.30am, a load completing at 3am is effectively live. In our assessments around 70 per cent of reports are properly served by a nightly refresh, around 27 per cent by hourly or micro-batch, and about 4 per cent genuinely need sub-minute data — and most of that 4 per cent is automated action inside an application rather than a dashboard at all.

How many times a day can Power BI refresh data?

A semantic model in a workspace backed by Power BI Pro licences supports eight scheduled refreshes per day, roughly two-hourly across a working day. Premium Per User and capacity-backed workspaces raise that to forty-eight, or every thirty minutes. Anything faster requires a different mechanism: DirectQuery, which sends live queries to the source on every interaction, a push or streaming dataset, or Direct Lake on a Fabric capacity. The jump from two-hourly to live is therefore an architectural change with a licensing consequence rather than a setting you adjust.

Is real-time reporting worth the cost for an SME?

For the specific functions that need it, yes; as a general reporting strategy, rarely. A streaming pipeline costs roughly five to ten times the build effort of the equivalent batch load, adds several thousand pounds a year in run cost, and creates a permanent operational responsibility that needs a named owner. Where a business loses money to a fifteen-minute lag — overselling stock, unscreened fraudulent payments, an unmanaged call queue — that is justified. Where the underlying complaint is that reports are late, inconsistent or untrusted, the same budget spent on reconciliation, definitions and monitoring produces a much better outcome.

Will reporting slow down our main business system?

It can, and this is the most common self-inflicted problem in SME reporting. Analytical queries run against a live transactional database compete for locks and resources with the application, and the symptom is usually an order entry or booking screen timing out during the busiest hour rather than a slow report. The standard remedies are a readable secondary in an availability group, a geo-replica on Azure SQL, snapshot isolation so readers do not block writers, or loading a separate reporting store overnight. Anyone considering DirectQuery should treat a replica as a prerequisite, because live query mode multiplies source queries by every filter click a user makes.

Why do our dashboards show different numbers from our accounts?

Usually because of definitions and cut-off points rather than refresh timing. One report counts an order at despatch and another at invoice; one includes intercompany sales and another excludes them; one reflects credit notes raised this morning while the ledger closed the period last night. Streaming makes this worse because it removes the stable boundary that finance depends on — a figure that always shows the latest state can never agree with a period that has closed. Fix it by writing down one definition per measure with a named owner, then reconciling the reporting store against the accounting system as a re-runnable query.

What is micro-batch and when should we use it?

Micro-batch is a scheduled incremental load running every five to thirty minutes, using a modified-date column, rowversion or change tracking to move only what has changed. It suits anything where somebody is working from a screen through the day — despatch progress against a carrier cut-off, service desk SLA watch, intraday sales against target, warehouse pick rates. It needs no message bus, no change data capture and no on-call rota, and the latency it delivers is indistinguishable from live for any human decision process. For most UK SMEs it is the correct destination for requirements that arrive labelled real-time.

How do we stop stale data going unnoticed?

Do two things. Put a visible “data as at” timestamp on every report so readers can judge the figures themselves, and alert on staleness rather than only on job failure — assert that the newest record in each target table is within an agreed threshold and raise an alert when it is not. Job-failure alerting catches the pipeline that crashed but not the one that ran successfully and loaded zero rows after a credential expired or a view was renamed. That second failure serves yesterday’s numbers with today’s confidence and is the fastest way to lose trust in a reporting estate.

Does a real-time pipeline create GDPR obligations?

Copying personal data into a second store for reporting purposes extends your processing to that store: it belongs in your record of processing activities, it needs a lawful basis and a retention rule of its own, and it forms part of your breach surface. The practical controls are data minimisation — select only the fields the reports genuinely need rather than replicating whole customer tables — keeping the pipeline and store in a UK region such as UK South or UK West, applying retention to the reporting copy as well as the source, and pseudonymising where reports only require aggregates. The ICO’s guidance on minimisation and storage limitation applies to analytical copies exactly as it does to operational systems.

What does a reporting refresh project cost and how long does it take?

For a UK business of a hundred to two hundred staff with one main line-of-business system plus finance, a properly reconciled nightly reporting store with a report pack and monitoring is typically eight to twelve weeks and £8,000 to £20,000 at UK day rates. Converting a selected handful of loads to micro-batch adds perhaps £10,000 to £20,000. A genuine streaming pipeline is a different proposition — three to six months, £45,000 to £120,000, and an ongoing ownership commitment. Establishing which of those you need is a one-week decision-mapping exercise and it is the highest-return week in the whole programme.

Who should decide the refresh strategy?

The cadence per report belongs to the business function that acts on it, and the architecture belongs to whoever is accountable for running it afterwards. In practice a short exercise chaired by someone with an independent view works best, because the person who will build the pipeline and the person who asked for real-time both have a stake in the answer. Where there is no internal data function, this is exactly the kind of decision a fractional technology advisor is suited to — our comparison of a virtual CIO against a project IT consultant sets out the difference between the two engagement models.

How do we know whether the investment worked?

Agree the measures before you start, because they cannot be reconstructed afterwards. The useful ones are time from period end to a signed-off management pack, number of reports with a documented owner and definition, freshness objective met as a percentage of days, count of unreconciled measures closed, and reduction in manually maintained spreadsheets. Report usage matters too — a dashboard nobody opened is a cost regardless of its refresh rate. The general approach to building a defensible baseline before a technology investment is covered in our guide to measuring Microsoft 365 Copilot ROI.

Related reading

More guidance on data, systems architecture and technology decisions for UK businesses:

• Network documentation: what to record and why it matters
• Headless CMS versus traditional CMS: choosing an architecture
• Virtual CIO versus IT consultant: which engagement model fits
• In-house IT versus managed IT support: a cost comparison
• Microsoft 365 Copilot ROI: measuring what the licences return