AI Feature Prototyping: A UK Business Guide to Validating AI Product Ideas Before Committing Engineering Budget in 2026

Almost every expensive AI failure in a UK business starts the same way: somebody demonstrated something impressive in a chat window, everybody in the room agreed it was obvious, and a six-figure engineering budget was committed before anyone asked whether the underlying model could do the job reliably, at volume, on the organisation’s own messy data. AI prototype validation is the discipline that sits between those two moments - a short, deliberately cheap piece of work whose only purpose is to find out, before the money is spent, whether the idea survives contact with reality.

This guide sets out a process a real UK business can run. It covers how to build a throwaway prototype in days rather than months; how to test feasibility against your own data instead of a vendor demo; how to discover the failure modes that only appear at scale; how to estimate what a production build will actually cost once you include evaluation, monitoring and the unglamorous engineering around the model; and how to set go/no-go criteria in advance so that the decision to build is made against evidence rather than enthusiasm. The order matters. Prototyping comes first because it is the cheapest way to be wrong.

What AI feature prototyping actually means

An AI prototype is not a smaller version of the product you intend to build. It is an instrument for answering a specific question, and it is designed to be thrown away. That distinction is the single most useful thing to hold on to, because almost all of the waste in early AI work comes from teams building a fragile first version, discovering it half-works, and then being unable to abandon it because it already looks like a product.

The question a prototype answers is narrow: can a model, given realistic inputs from this organisation, produce an output of sufficient quality, often enough, at a cost and latency the business can live with? Everything else - the interface, the authentication, the audit trail, the integration with the finance system - is deliberately excluded. A prototype that takes four weeks because someone insisted on a login screen has failed at its job, which was to be fast.

It is worth separating three terms that get used interchangeably and mean quite different things. A demo shows that something is possible once, usually on a hand-picked example, and proves almost nothing. A prototype tests a hypothesis against a representative sample and produces a measurement. A pilot puts a real system in front of real users under controlled conditions and tests adoption and operational fit. They come in that order, they cost roughly ten times more at each step, and skipping from demo straight to build is where the money goes. The framing throughout this guide treats LLM feature feasibility as something you measure, not something you assess by intuition.

There is also a scoping point that catches teams out repeatedly. Modern general-purpose models are extremely good at tasks that look like language and extremely inconsistent at tasks that look like language but are actually arithmetic, lookup, or policy enforcement in disguise. “Summarise this contract” is a language task. “Tell me whether this contract breaches our procurement policy” is a policy task wearing a language costume, and it will behave completely differently under testing. A large part of prototyping is discovering which of the two you have actually been asked to build.

AI prototyping by the numbers - what UK teams are actually spending

The figures below reflect the shape of AI work in small and mid-sized UK organisations: the gap between what a validation exercise costs and what a committed build costs, and how often that gap is bridged on evidence rather than assumption. They are useful less as benchmarks to hit than as a sense of the asymmetry involved - the cost of finding out is small, and the cost of not finding out is not.

The asymmetry is the whole argument. A prototype that costs a low five-figure sum and kills a bad idea has returned its cost several times over, even though it produced nothing shippable. Organisations that struggle with AI investment are usually not the ones with a poor hit rate on ideas - a low hit rate is normal and expected. They are the ones with no cheap way to find out which ideas are the bad ones, so every idea is either dismissed on a hunch or built at full cost.

This is also why treating AI spend as part of a wider planning cycle matters. An organisation with a structured view of its technology commitments, of the sort covered in our guide to building an IT roadmap and technology strategy, tends to fund validation as a standing line item rather than as an exception that has to be argued for each time. That single budgeting change does more for AI success rates than any choice of model or framework.

Where AI features fail - the failure modes that appear after the budget is committed

Feasibility failures are not random. They cluster into a small number of recognisable patterns, and each one is detectable at prototype stage if you know to look for it. The distribution below reflects how often each pattern turns out to be the binding constraint when an AI feature does not make it into production in its intended form.

Data quality sits at the top of that list by a wide margin, and it is worth being precise about what that means, because “our data is bad” is too vague to act on. In practice it means one of four specific things. The data exists but is not accessible without an integration nobody has budgeted for. The data is accessible but inconsistent - the same field means different things in different years, or three systems hold competing versions. The data is consistent but lacks the labels needed to measure whether the model is right. Or the data simply does not contain the signal the feature requires, and no amount of model capability will extract information that was never recorded.

That last case is the most important and the most frequently missed. If a business wants to predict which customers will churn, and the only records held are invoices, then the interesting behaviour - support friction, usage decline, the unhappy phone call - is absent from the dataset. A model trained or prompted on invoices will produce confident output, and it will be confidently uninformative. A prototype catches this in days. A build discovers it in month five.

The second and third rows deserve to be read together. An AI feature that is 92% accurate sounds strong until you examine what the remaining 8% consists of and who absorbs the cost of it. If errors are evenly distributed, minor and easily spotted by the user, 92% may be excellent. If errors concentrate in exactly the unusual cases that most need attention - which is the normal pattern, since unusual cases are underrepresented in whatever the model learned from - then 92% accuracy can be worse than useless, because it trains users to trust a system that fails hardest precisely when the stakes are highest.

What an AI prototype costs - and what the full build costs if you skip it

Good AI development cost estimation starts by separating the four distinct stages of spend, because they are routinely collapsed into a single number and that single number is always wrong. The table below sets out what each stage typically involves for one AI feature in a UK small or mid-sized organisation, at 2026 rates.

Two lines in that table cause most of the budget surprises. The first is the annual run cost. AI features are not like a static web page that sits there once deployed; the models beneath them are updated by their providers, and an update that improves general performance can quietly degrade performance on your specific task. Without a maintained evaluation set, that degradation is invisible until a user complains. Budgeting 20–35% of build cost annually is not padding - it is the cost of the feature continuing to work.

The second is the production build range, which is wide for a reason. The model work is rarely the expensive part. What drives cost is everything around it: permissions that respect existing access controls, an audit trail detailed enough to answer a regulator or a customer, sensible behaviour when the model provider has an outage, and a support path for when output is wrong. Teams that have defined their support response expectations and SLA structure before building tend to scope this realistically. Teams that have not tend to discover an entire workstream in month four.

On inference economics specifically, the arithmetic is simple but frequently deferred until too late. Take the realistic per-call token usage measured during the prototype, multiply by expected daily volume, multiply by 250 working days, and then multiply by three to allow for retries, evaluation runs and the growth that follows a successful launch. A feature costing four pence per call sounds negligible until it is running twelve thousand times a day, at which point it is a £36,000 annual line item that nobody put in the budget. Measuring token consumption is a standard prototype output, not an afterthought.

Prototype-first versus build-first - comparing the two routes

The build-first route is not irrational. It is chosen because prototyping feels like delay, because a confident supplier has already scoped the work, or because the idea seems too obviously good to need testing. The comparison below sets out what each route actually buys, on the same feature.

The row that matters most is the last one. A weak idea taken down the build-first route rarely produces a clean failure, because by the time the problem is understood there is too much invested to stop. What happens instead is scope reduction: the ambitious feature becomes a modest one, the modest one ships, and it is used by nobody. The organisation records a delivered project and an unexplained absence of benefit. Prototype-first produces something more useful - an explicit decision, made early, with the reasoning written down.

It is also worth noting what prototype-first does not buy. It does not guarantee the build will succeed; plenty of things that work at prototype scale encounter integration and adoption problems later. It does not replace a pilot. And it does not remove the need for judgement about whether the feature is worth having at all, which is a commercial question a measurement cannot answer. What it buys is the elimination of one specific and expensive category of failure: building something the model was never going to be able to do.

Feasibility scoring - is your AI idea actually buildable with today’s models?

Before writing any code, most AI ideas can be triaged in an hour using the three dimensions below. The scoring is deliberately blunt. Its purpose is not precision but to surface the dimension on which the idea is weakest, because that is the thing the prototype should be designed to test first.

Read across the three cards rather than down them. An idea with strong task fit, weak data readiness and low error tolerance is worth prototyping, because the data problem is the only thing standing in the way and it is a solvable engineering question. An idea with poor task fit and high-risk error tolerance - a model applying eligibility rules to benefit entitlements, say - should not be prototyped as stated. It should be reshaped, usually by moving the decision into code and using the model only to prepare the inputs for it.

The third card is where governance enters and it should enter here, not later. If the output affects an individual in a meaningful way, the UK GDPR provisions on automated decision-making are engaged and a Data Protection Impact Assessment is likely to be required. That is not a reason to stop, but it is a reason to involve whoever holds data protection responsibility in week one rather than week twenty. The same applies to where the data goes: sending company records to a third-party model provider is a processing decision with contractual consequences, and the considerations mirror those we set out for Microsoft 365 Copilot data security, where the question of what the model can see is inseparable from the question of what your permissions model already allows.

The four-week AI prototype - a realistic timeline

The timeline below assumes one experienced engineer with part-time input from someone who understands the business process, which is the minimum viable team. The most common mistake is treating week one as a technical week. It is not; it is the week that determines whether the following three weeks measure anything meaningful.

Four weeks is the comfortable version. A focused feasibility spike answering a single narrow question can be done in five days, and for many ideas that is enough to establish that the answer is obviously no. The full four weeks earns its keep when the answer is likely to be a qualified yes, because then the detail of which parts work becomes the basis of the build scope.

Evaluation benchmarks - what accuracy levels real AI features reach

Expectation setting is half the battle. The figures below are the sort of accuracy levels well-implemented features reach on their own evaluation sets after a few iterations, measured against a human-agreed correct answer. They are not ceilings and they vary enormously with data quality, but they give a realistic sense of which tasks arrive near-solved and which require substantial engineering to become usable.

The pattern running down that list is a steady move from tasks where the answer is contained in the input towards tasks where the answer must be assembled from several places and checked against rules. Every step in that direction costs accuracy, and the useful engineering response is almost always to decompose rather than to push harder on a single call. A reconciliation feature that performs at 44% as one prompt will often reach the high eighties when the model is used to extract and normalise values while the arithmetic and rule-checking happen in ordinary code.

Note also what these numbers are measured against: a human-agreed correct answer. Establishing that agreement is frequently where the exercise becomes uncomfortable, because two experienced members of staff will often disagree about the right categorisation for fifteen or twenty percent of cases. When that happens, the model is not the limiting factor - the process definition is, and no AI feature will resolve an ambiguity the organisation has never settled for itself.

The go/no-go decision - what share of AI ideas survive validation

A validation process that approves everything is not validating anything. The figure below reflects the proportion of AI feature ideas that proceed to a full build in their originally requested form once they have been measured against a threshold agreed in advance.

The remaining two thirds do not simply disappear, and it would be misleading to present them as failures. In practice they divide into three groups of roughly similar size. Some are reshaped - the ambition is intact but the mechanism changes, most often by moving part of the work into deterministic code or narrowing the scope to the document types where extraction is reliable. Some are deferred, usually because the data required is not yet captured, which converts an AI project into a much cheaper data-collection project that makes the AI version viable a year later. Only a minority are stopped outright, and those are typically the ideas where the required information was never recorded anywhere.

An organisation seeing a 34% unchanged-proceed rate is running a healthy process. One seeing 90% is not testing hard enough, and its thresholds are almost certainly being set after results are known. One seeing 5% has a generation problem rather than a validation problem - ideas are arriving without any grounding in what the technology does, and the fix is upstream, in how candidate features are proposed in the first place.

The 12-point AI prototype validation checklist

Run through this before any engineering budget is committed. Each point corresponds to a question that, left unanswered, has a recognisable habit of reappearing later as an unplanned cost. An AI proof of concept that can satisfy all twelve is ready to be scoped as a build.

1. The feature is described as one input and one output. If it takes a paragraph to describe what goes in and what comes out, it is several features and should be split before testing.
2. A go/no-go threshold is written down and signed off. Accuracy level, maximum latency, maximum cost per call, agreed by the budget holder before any results exist.
3. A representative data sample has been extracted from live systems. Including the awkward records, not a curated set assembled to show the idea in its best light.
4. An evaluation set of 100–200 labelled examples exists. With correct answers agreed by someone who performs the task today, and the disagreement rate between reviewers recorded.
5. A baseline run has been measured before any optimisation. Straightforward prompt, capable general-purpose model, full evaluation set, results recorded.
6. Every failing case has been read and grouped by cause. Not sampled. The distribution of failures matters more than the headline accuracy figure.
7. Token consumption per call has been measured and annualised. At realistic volume, with headroom for retries, evaluation reruns and growth after launch.
8. The 95th percentile latency has been checked against the actual workflow. Not the median. Users experience the slow calls, particularly under concurrency.
9. The consequence of a wrong output has been documented. Who notices, how quickly, what it costs to correct, and whether a human reviews before anything is actioned.
10. The data protection position is understood. Lawful basis, DPIA requirement, whether special category data is involved, and what the model provider’s processing terms permit.
11. Behaviour when the model provider is unavailable has been decided. Queue, degrade to manual, or fail visibly - but decided deliberately rather than discovered during an outage.
12. The build estimate includes ongoing evaluation and monitoring. With the annual run cost stated as a figure, not described as minimal.

Scoring your readiness to commit engineering budget

Score one point for each checklist item satisfied, multiply by eight, and compare against the benchmark below. The gauge shows where organisations typically sit at the moment they are first asked to approve an AI feature build - which is to say, before any of this work has been done.

A score below 40 does not mean the idea is bad. It means the decision in front of the budget holder is not yet a decision but a guess, and the appropriate response is to fund a prototype rather than to approve or reject the build. Between 40 and 70, the significant unknowns are usually cost and failure distribution, and a focused two-week exercise will close them. Above 70, the remaining questions are typically about integration and adoption, which a prototype cannot answer and a pilot can.

The scoring is most useful as a conversation structure rather than a number. When a proposal scores 38, the productive question is not whether to proceed but which four or five checklist items are missing and what it would take to satisfy them. That reframes a yes-or-no argument about a large sum into a concrete, small piece of work - which is generally a much easier conversation to have with a board.

Common AI prototyping mistakes to avoid

The mistakes below are common enough to be predictable, and each one has the same underlying shape: an activity that feels like progress but does not reduce uncertainty about the question the prototype was meant to answer.

• Prototyping on curated examples. The five documents chosen to demonstrate the idea are, by construction, the five the idea handles best. Real accuracy is measured on a random sample of what actually arrives, including the scanned fax and the spreadsheet someone exported to PDF.
• Building the interface first. A polished front end makes the prototype harder to throw away and answers no feasibility question. Measure with a script; the interface belongs to the pilot.
• Optimising before establishing a baseline. Teams routinely spend a week on elaborate prompting before discovering that a straightforward prompt already met the threshold, or that the gap was never one prompting could close.
• Measuring accuracy without measuring failure distribution. A single percentage hides whether errors are harmless and evenly spread or concentrated in the highest-value cases. The second is disqualifying and the headline figure will not reveal it.
• Setting the threshold after seeing the results. The most expensive mistake on this list, because it converts validation into justification while leaving everyone convinced a rigorous process was followed.
• Ignoring the human comparison. If the existing manual process is 91% accurate and the model reaches 88%, the feature may still be worth building on cost or speed grounds - but that is a different business case, and it needs the human baseline measured rather than assumed to be perfect.
• Treating the prototype as version one. Prototype code is optimised for being changed quickly, not for being maintained. Carrying it into production imports every shortcut taken under time pressure into a system that will run for years.
• Deferring governance to the build phase. The DPIA, the processing terms and the permissions model rarely stop a good feature, but they routinely change its cost and timeline. Finding that out during the build turns a scoped project into a renegotiation.

A real-world example - validating a document extraction feature

A Leeds-based logistics business with 140 staff wanted to automate the processing of supplier delivery notes. Roughly 900 arrived each week across three channels - email attachments, a supplier portal, and photographs taken on drivers’ phones - and two members of the operations team spent most of their working week keying the details into the transport management system. The proposal presented to the board was a fully automated extraction pipeline, estimated at £110,000, on the strength of a supplier demonstration that had processed four clean PDFs flawlessly.

Rather than approve the build, the finance director funded a three-week prototype at £14,000, with a threshold agreed in advance: 90% field-level accuracy across all three channels, under four seconds per document, at less than three pence per document. An evaluation set of 180 delivery notes was assembled from a random fortnight of real traffic, labelled by the two people who did the keying.

The baseline run produced a result that changed the entire project. Across all documents the accuracy was 71%, comfortably below the threshold. But the failure analysis showed the number was an average concealing two different populations: the digital PDFs and portal exports, which made up 62% of volume, scored 96%; the driver photographs, which made up the remaining 38%, scored 31%, largely because of creased paper, poor lighting and handwritten amendments in the margin. No amount of prompt engineering moved the photograph figure above the low forties.

We were three weeks and fourteen thousand pounds away from approving a hundred-and-ten-thousand-pound build that would have automated the easy half of the job and quietly handed the hard half back to the same two people, with less context than they had before. The prototype did not tell us the idea was wrong. It told us which half of it was real.

The reshaped project was scoped at £38,000: automated extraction for the digital channels only, with photographs routed to a queue for manual entry and a supplier engagement programme to move the larger accounts onto the portal over the following year. Annual run cost was modelled at £4,800 on measured token consumption. Six months after launch the digital share of volume had risen to 74% as suppliers moved across, which improved the feature’s coverage without any further engineering - an outcome that was possible only because the scope had been drawn around what the technology could actually do rather than around what the original proposal assumed.

The instructive detail is the labelling exercise. When the two operations staff compared their labels, they disagreed on 11% of cases, mostly about how to record partial deliveries. That disagreement had been invisible for years and was flowing into the transport system as inconsistent data. Resolving it was a process improvement the business would have wanted regardless of whether any AI feature was ever built, and it surfaced only because someone sat down and wrote out what the correct answer was supposed to be.

Build, buy or wait - the three outcomes a prototype should be able to recommend

A validation exercise that can only recommend build or stop is missing the two most common correct answers. Once the measurement exists, four routes are usually on the table, and the prototype data points clearly at one of them.

Build custom is right when the task is specific to how your organisation works, the accuracy threshold is met, and no credible product does the same job. The signal from the prototype is a good baseline score combined with failure cases that are about your data conventions rather than general model limitations. Those are the cases retrieval and decomposition fix well, and they are also the cases an off-the-shelf product will never have been tuned for.

Buy is right more often than engineering teams like to admit. If the prototype shows that the task is generic - extracting fields from standard invoice formats, transcribing and summarising meetings, drafting routine correspondence - then a mature product will almost certainly beat a custom build on both cost and reliability, because its vendor has already absorbed the long tail of edge cases across thousands of customers. The prototype is still worth running, because it establishes the accuracy bar against which you evaluate vendors, and it gives you an evaluation set to test them with rather than relying on their own demonstration.

Reshape is the most common outcome and the hardest to reach without measurement. The delivery-note example above is the standard pattern: the feature works on a subset of inputs and not on the rest, and the right project is the one drawn around the subset with a deliberate route for everything else. Reshaping requires knowing the shape of the failures, which is exactly what the failure-mode analysis produces and what a headline accuracy percentage conceals.

Wait is legitimate and under-used. If the blocker is that the required data is not captured anywhere, then the useful project this year is a data-capture project, and the AI feature becomes viable once there is something to work with. If the blocker is model capability on a task that is improving quickly, a documented decision to re-run the same evaluation set in nine months costs almost nothing and is far better than either building against a limitation or abandoning the idea entirely. The evaluation set makes re-testing a half-day exercise rather than a fresh project.

Whichever route is chosen, the decision should be recorded with the evidence attached. Organisations that keep a short written record of why each AI idea was built, bought, reshaped or deferred stop relitigating the same proposals every budget cycle, and they build an institutional sense of which kinds of ideas tend to work in their particular context.

From prototype to production - what actually changes

Approving the build is not the end of the validation work; it changes its character. The prototype answered whether the model can do the job. Production engineering has to answer whether the system can keep doing it, under load, for years, while the models underneath it change without notice.

The most significant difference is that production needs a standing evaluation capability rather than a one-off measurement. The labelled set built during the prototype becomes a regression suite that runs on a schedule and on every change to prompts, retrieval configuration or model version. When a provider ships an update, the suite tells you within hours whether your specific task improved, held steady or degraded. Without it, the first indication of a problem is a user reporting that the feature has become unreliable, weeks after the change, with no way to establish when it started.

Second, production requires explicit failure behaviour at every layer. What happens when the provider returns an error, when a response exceeds the latency budget, when output fails validation, when the input is a document type nobody anticipated. Each needs a decided answer - retry, fall back to a smaller model, queue for human handling, or fail visibly with a clear message. The unacceptable option is the one that occurs by default, which is a silent partial result that looks like a real answer.

Third, permissions and audit become first-class concerns. The prototype ran against a data extract with the engineer’s access. Production must enforce that a user sees only what their role permits, which means retrieval has to be permission-aware rather than searching a single shared index. Getting this wrong is one of the more serious ways an AI feature can cause harm, because it can surface information across boundaries the organisation assumed were solid - the same class of problem that makes permission hygiene the central issue in any assistant deployment, as covered in our Copilot data security guide. The audit trail should record input, output, model version, prompt version and the identity of whoever acted on the result, which is what allows a decision to be explained months later.

Fourth, the interface has to communicate uncertainty honestly. A feature that presents a 78%-accurate extraction as settled fact will be trusted at face value and the errors will propagate. The same feature presenting the extraction as a draft for confirmation, with low-confidence fields visibly flagged, performs the same technical function while producing a materially better outcome. Interface decisions of this kind also carry accessibility obligations, and the considerations in our guide to WCAG accessibility compliance apply to AI-driven interfaces exactly as they do to any other - a confidence indicator conveyed by colour alone is not an accessible one.

Finally, production brings the security surface into scope. An AI feature accepting untrusted input - customer emails, uploaded documents, web content - is accepting instructions that may be crafted to manipulate the model into ignoring its intended behaviour. Treating model output as untrusted, constraining what downstream actions it can trigger, and validating structured output against a schema before it is used are baseline controls. Organisations already working to a recognised framework will find this maps onto existing practice, and the technical verification habits described in our Cyber Essentials Plus technical audit guide transfer directly to reviewing an AI feature before it goes live.

AI prototype validation at a glance

The key facts and figures from this guide, collected for reference.

How Cloudswitched approaches AI feature prototyping

Cloudswitched runs AI validation as a fixed-scope, fixed-price exercise with a defined end date and a written recommendation, deliberately separated from any build engagement. The work covers the sequence set out in this guide: agreeing the go/no-go threshold with whoever holds the budget, extracting a representative sample from live systems, constructing a labelled evaluation set with the people who do the task today, measuring a baseline, analysing every failure, modelling annual cost at realistic volume, and reviewing the governance and integration position. The output is a decision pack, and the recommendation may well be not to build.

Where a build follows, the evaluation set carries forward as the regression suite, and the integration, permissions and monitoring work is scoped against measured figures rather than estimates. Where it does not, the exercise has cost a fraction of the budget it prevented from being committed, and the organisation retains a labelled dataset and a clear written account of why the answer was no.

Frequently Asked Questions

How long should an AI prototype take?

Two to four weeks for a structured prototype that produces measured results, and three to five days for a feasibility spike answering a single narrow question. Anything longer than four weeks has usually stopped being a prototype and started becoming a product, which defeats the purpose. The constraint is useful: a hard time limit forces the work to stay focused on the feasibility question rather than drifting into interface design or integration. If a prototype genuinely cannot answer the question in four weeks, that is normally a sign the question is too broad and should be split into two or three narrower ones tested in sequence.

What does AI prototype validation cost in the UK?

A structured prototype typically runs to between £6,000 and £18,000, depending on how many data sources are involved and how much effort the labelled evaluation set requires. A shorter feasibility spike sits at £2,500 to £5,000. Set against a production build starting around £75,000 and an annual run cost of 20–35% of that, the validation spend is normally between 5% and 15% of the commitment it informs. The relevant comparison is not prototype cost versus zero, but prototype cost versus the expected cost of discovering the same information halfway through a build.

Can we prototype an AI feature without a data scientist?

For most current AI features, yes. The work is generally software engineering and evaluation design rather than model training: calling a capable general-purpose model, structuring prompts and retrieval, and measuring results against a labelled set. An experienced developer with good measurement discipline and access to someone who understands the business process covers the majority of cases. Specialist input becomes necessary when custom model training, fine-tuning on proprietary data, or genuinely statistical prediction work is involved - but those represent a minority of the AI features UK businesses actually want, and it is worth establishing through a prototype whether yours is one of them before hiring for it.

What accuracy is good enough for an AI feature?

There is no universal figure, which is why the threshold must be set per feature and before testing. The two questions that determine it are what the current process achieves and what a wrong output costs. If people perform the task at 91% accuracy today and errors are caught downstream, a model at 88% with faster turnaround may be a clear improvement in practice. If output feeds an automated process affecting customers, even 98% may be insufficient without human review. The distribution matters as much as the level: errors concentrated in the highest-value cases are disqualifying at any headline accuracy.

Should we use a general-purpose model or train our own?

Start with a capable general-purpose model in every case, because it establishes a baseline in days rather than months and frequently turns out to be sufficient. Custom training is justified when the baseline falls well short, when the gap is clearly about domain-specific patterns rather than prompt or retrieval design, and when enough labelled data exists to train on - typically thousands of examples rather than the hundred or two needed for evaluation. Most organisations that assume they need a custom model discover during prototyping that better retrieval, clearer task decomposition, or moving part of the work into deterministic code closes the gap at a fraction of the cost.

How do we stop the prototype becoming the production system?

Decide before starting that the code will be deleted, say so explicitly, and avoid building anything that makes deletion feel wasteful - particularly a polished interface. Keep the prototype in a separate repository with a clear name, and hand over the decision pack and the labelled evaluation set as the deliverables rather than the code itself. The pressure to carry prototype code forward is mostly a consequence of it looking finished; a prototype that is visibly a measurement script attracts no such pressure, while a working web interface with a login screen invariably does.

What data protection issues apply to AI prototyping?

Using live business data in a development context requires a lawful basis, and if personal data is involved a Data Protection Impact Assessment is likely to be needed before the feature reaches production - particularly where output materially affects individuals, which engages the UK GDPR provisions on automated decision-making. Sending data to a third-party model provider is a processing decision requiring appropriate contractual terms, and you should confirm whether inputs may be retained or used for training. Pseudonymising the prototype sample is often straightforward and removes most of the difficulty. Involve whoever holds data protection responsibility in week one; the ICO expects this position to be documented before deployment, not retrofitted afterwards.

How do we estimate the running cost of an AI feature?

Measure actual token consumption per call during the prototype rather than estimating from documentation, then multiply by realistic daily volume and 250 working days. Multiply the result by roughly three to allow for retries, evaluation reruns and volume growth after launch. Add the cost of maintaining the feature - re-running evaluations when the provider updates the model, adjusting prompts and retrieval, and handling incidents - which in practice runs to 20–35% of build cost annually. The inference bill is frequently the smaller half of the true running cost, and budgets that count only tokens tend to understate by a wide margin.

What if the prototype shows the idea is not feasible?

That is a successful prototype and should be recorded as one. In practice a negative result usually distinguishes between three situations: the feature works on a subset of inputs and should be reshaped around that subset; the required data is not captured, making the useful project a data-capture one; or the task is a poor fit for current models and the sensible action is to re-run the same evaluation set in nine to twelve months. Keep the evaluation set either way - it makes re-testing a half-day exercise rather than a fresh project, and it prevents the same proposal being argued from scratch at the next budget round.

How many AI ideas typically survive validation?

Around a third proceed to a full build in their original form. Of the remainder, most are reshaped into a narrower version that works or deferred until the underlying data exists, and only a minority are stopped outright. A process approving nearly everything is not testing seriously, and usually indicates thresholds being set after results are known. A process rejecting almost everything points to a problem further upstream in how candidate ideas are generated - features are being proposed without any grounding in what the technology reliably does, and the fix belongs at the proposal stage rather than the validation stage.

Do we need a prototype if we are buying an off-the-shelf AI product?

Yes, though a shorter one. The purpose changes from testing whether the task is possible to testing whether this particular product does it on your data. Build the labelled evaluation set exactly as described, then run the vendor product against it during the trial period. This replaces a demonstration curated by the supplier with a measurement against your own inputs, and it gives a directly comparable score across several products. It also frequently reveals that the product handles the common cases well and the awkward ones poorly, which is precisely the information needed to negotiate scope, pricing and what the manual fallback process needs to cover.

Who should be involved in the go/no-go decision?

At minimum the budget holder, someone who performs the task today, and whoever will be accountable for the feature once it is live. The budget holder signs the threshold in advance so the decision is made against a standard rather than an impression. The practitioner defines what a correct output actually is, which is the foundation of the evaluation set and frequently surfaces process ambiguities the organisation has never resolved. The future owner raises the operational questions - support, monitoring, what happens when it is wrong - that determine whether the feature survives its first year in production.

Related reading

Further guides covering the planning, security and delivery context around AI feature work.

• Microsoft 365 Copilot Data Security: A UK Guide
• IT Roadmap and Technology Strategy for UK Businesses
• Cyber Essentials Plus: The Technical Audit Explained
• Website Accessibility Compliance and WCAG for UK Business
• IT Support SLAs and Response Times: A UK Business Guide

Validate before you commit

The decision to build an AI feature is easier to make well when it is made against measurements rather than impressions. A short, scoped prototype produces those measurements - accuracy against your own data, the shape of the failures, cost per call at real volume, and the governance position - in time for them to inform the decision rather than explain it afterwards.