Good LLM integration architecture is what separates an AI feature that keeps working for years from one that becomes a maintenance burden within months. The pattern we see most often in UK businesses is easy to recognise. A developer adds a call to a large language model directly inside an existing application — a few lines that send a prompt to one provider’s API and paste the answer into a screen or a database field. The demo works, the business is impressed, and the same approach is copied into a second feature, then a third.
Six months later, the problems arrive together. The provider retires the model version the prompts were tuned for, and outputs change subtly. Prices change, or a cheaper model appears, but switching means editing code in a dozen places. Answers that used to parse cleanly start arriving in slightly different formats and break downstream processing. Nobody can say how much the AI features cost per month, which prompts are in use, or why a customer received a strange answer last Tuesday. The AI was never the hard part; the coupling was.
This guide starts with why tight coupling to a single model provider is the most common architectural mistake, then covers the integration patterns that age well: an abstraction layer between your applications and the models, retrieval augmented generation versus fine-tuning for grounding AI in business data, synchronous versus asynchronous integration for latency-sensitive workflows, and connecting AI to legacy systems that were never designed to be called by anything external. It is written for technical leads, IT managers and business owners commissioning AI work, and assumes no particular provider.
What LLM integration architecture means
Integration architecture is the set of decisions about how AI capabilities connect to the rest of your systems: where model calls are made from, how prompts and outputs are managed, how business data reaches the model, how results flow back into applications and processes, and how the whole arrangement is monitored, secured and changed over time.
It matters because large language models behave differently from the software most businesses are used to integrating. They are probabilistic rather than deterministic, so the same input can produce different outputs. They are offered by a fast-moving market in which models are released, improved, repriced and retired frequently. They can be slow by the standards of normal application calls, sometimes taking several seconds or more. And they are only as useful as the context they are given, which usually means connecting them to business data that lives in other systems.
Each of those characteristics argues for the same thing: a deliberate separation between the business logic of your applications and the specifics of whichever model you happen to be using today. That separation is the core of an architecture that ages well.
A simple test of any AI integration: is there exactly one place in your systems through which all model calls pass? If the answer is yes, changing providers, adding logging, controlling costs or redacting sensitive data is a change in one place. If calls are scattered across applications, each of those becomes a project.
Why tight coupling is the most common mistake
Coupling describes how much one part of a system depends on the internal details of another. AI system coupling becomes a problem when applications depend directly on one provider’s API, one model’s behaviour and one set of prompt wording, with no layer in between to absorb change.
Models change underneath you
Providers retire model versions on published schedules and release new ones that behave differently. A prompt carefully tuned for one model may produce longer, shorter or differently structured answers on its successor. When calls are embedded directly in application code, every upgrade means finding, editing and retesting each one.
The market moves faster than your release cycle
Prices fall, new models outperform old ones on particular tasks, and different providers lead in different areas. A business locked to one provider through scattered code cannot take advantage of cheaper or better options without substantial rework, and has no fallback if that provider has an outage.
Prompts become hidden business logic
Prompts embedded in code are effectively business rules written in natural language, yet they are rarely versioned, reviewed or tested like other business rules. When behaviour changes, nobody can easily see which prompt changed, when or why.
Outputs are parsed hopefully
Direct integrations often take free text from a model and parse it with assumptions about format. Small variations in output then break downstream processing in ways that are hard to trace.
Nobody can see what is happening
Without a central point, there is no consistent logging of prompts, responses, latency, errors or cost. Investigating a complaint, controlling spend or proving what data was sent to a third party becomes difficult.
Security and data controls are duplicated or missing
Each direct integration needs its own handling of API keys, sensitive data redaction and access control. In practice, some do it well and some do not do it at all.
Direct model calls versus an abstraction layer
The comparison shows the practical differences between bolting model calls into each application and routing them through a shared layer.
Direct calls in each application
Provider SDK and prompts embedded in application code
Abstraction layer
Applications call an internal AI service, which calls the models
The abstraction layer does not need to be elaborate. For many SMEs it is a small internal service or module with a clear interface, sitting between applications and whichever providers are in use. Its value comes from being the single place where model-specific details live, so that the rest of the business can treat AI as a stable capability rather than a moving target.
Scoring three integration approaches
The score grid compares three common approaches to AI API integration patterns. A high badge means the approach performs well on that dimension.
Note that the last row is scored differently: high means high dependency, which is a drawback. Direct calls are quickest to start but weakest on everything that matters after launch, and they tie you closely to one provider. An internal service takes a little longer to establish but scores well across the board. Third-party orchestration platforms and frameworks can accelerate development and provide useful features, but they swap dependency on a model provider for dependency on a platform; if you use one, it is still worth keeping your own thin interface in front of it.
AI integrations in UK businesses — the numbers
The figures below are indicative, drawn from AI integration reviews we carry out for UK businesses. They are not a formal survey, but the pattern is consistent across sectors.
The second figure is the one that most often surprises business owners. Rework rarely happens because the original idea was wrong. It happens because a model version is retired, a cheaper model becomes attractive, output formats drift, or the feature needs connecting to more data, and the original integration was not designed to accommodate any of these. The fourth figure explains why problems are often discovered by users rather than by the team: without evaluation, nobody knows when quality changes.
What goes wrong after launch
The chart shows the problems most often reported in directly coupled AI features within their first year of production use. Most features experience more than one, so the figures add up to more than 100%.
Almost every item on the chart is an architectural problem rather than a model problem. Format drift is solved by structured outputs and validation in one place. Model changes are absorbed by an abstraction layer and caught by evaluation. Cost attribution and investigation depend on central logging. Timeouts point to the choice between synchronous and asynchronous integration. Stale answers point to how business data reaches the model. The rest of this guide takes each in turn.
The abstraction layer: what it should do
An abstraction layer is an internal service or module that every application uses to reach AI models. Applications ask it to perform a task — summarise this document, classify this enquiry, extract these fields — and it handles everything model-specific. A well-designed layer typically takes on the following responsibilities.
A stable, task-based interface
Applications call named capabilities such as “classify support ticket” with defined inputs and outputs, rather than sending raw prompts to a named model. This keeps business code independent of which model or provider fulfils the request.
Model routing and configuration
Which provider and model serve each capability is configuration, not code. Different tasks can use different models — a fast, inexpensive model for simple classification, a more capable one for complex reasoning — and changing a model is a configuration change followed by evaluation, not a code release across several applications.
Prompt management
Prompts live in one place, are versioned like code, and are reviewed and tested before changes go live. Each logged request records which prompt version produced it, so behaviour changes can be traced.
Structured outputs and validation
Wherever possible, the layer asks models for structured output in a defined format, validates it against the expected structure and rules, and handles failures — retrying, falling back or flagging for review — before anything reaches an application. Downstream systems receive predictable data, not hopeful free text.
Reliability
Consistent timeouts, retries with sensible limits, and fallbacks to an alternative model or a non-AI path when a provider is slow or unavailable. Rate limits are handled centrally rather than surprising each application separately.
Logging, cost and usage tracking
Every request is logged with the capability, prompt version, model, latency, token usage and cost, plus a reference to the calling application and, where appropriate, the user. This makes cost attribution, investigation and capacity planning straightforward. Logs that contain personal data need the same protection and retention rules as any other personal data.
Data protection and security controls
Redaction or minimisation of sensitive data before it leaves your environment where appropriate, management of provider keys in one secure place, and enforcement of which applications may use which capabilities. Our guide to securing AI tools and LLMs covers the wider security picture.
Evaluation hooks
The ability to run a set of test cases against a capability whenever its prompt or model changes, and to sample production outputs for review. This is what turns a model upgrade from a leap of faith into a measured decision.
None of this needs to be built at once. A first version that provides a task-based interface, central configuration, structured output validation and logging delivers most of the value. Reliability features, redaction and evaluation can follow as usage grows.
Integration practices in place
The progress bars show how often each practice is in place across the production AI integrations we review in UK businesses. Figures are indicative.
AI integration practices in place
The basics that any developer would add — keys out of code, some error handling — are common. The practices that make an integration maintainable over time — a single layer, central logging, versioned prompts, cost tracking, evaluation and fallbacks — are each present in a minority of integrations. They are also the practices that would have prevented most of the problems in the earlier chart.
Grounding AI in business data: retrieval augmented generation versus fine-tuning
A general-purpose model knows nothing about your products, policies, customers or contracts. For most business uses, that knowledge has to be supplied. There are two broad ways to do it, and choosing the right one has large consequences for cost, accuracy and maintenance.
Retrieval augmented generation
Retrieval augmented generation, usually shortened to RAG, finds the relevant information at the moment a question is asked and includes it in the prompt. Documents, records or knowledge base articles are prepared in advance — typically split into sections and indexed so that relevant passages can be found by meaning as well as by keyword — and the most relevant are retrieved for each request. The model then answers using that supplied context.
RAG has several advantages for business use. Information stays current, because updating the source updates the answers without retraining anything. Answers can cite the sources they used, which helps users check them and builds trust. Access controls can be applied at retrieval time, so people only receive answers based on documents they are allowed to see. And because knowledge sits outside the model, switching models does not lose it.
Fine-tuning
Fine-tuning adjusts a model’s behaviour by training it further on examples. It is good at teaching a model a consistent style, format or specialised task — classifying documents into your categories, writing in a particular house style, following a complex output structure reliably. It is not a good way to teach a model facts that change, because updating knowledge means retraining, and fine-tuned models generally cannot cite where an answer came from.
Fine-tuning also increases coupling. A model fine-tuned on one provider’s platform usually cannot be moved to another, and when the base model is retired, the fine-tuning may need repeating.
Choosing between them
For most business questions about facts, documents and records, retrieval augmented generation is the better starting point. Fine-tuning is worth considering when the problem is behaviour rather than knowledge, when prompt design and examples have been exhausted, and when volumes justify the effort. Many mature systems use both: retrieval for knowledge, and fine-tuning or careful prompting for format and style.
Where RAG goes wrong
RAG is only as good as its retrieval. Common problems include source documents that are out of date or contradictory, sections split in ways that separate a question from its answer, retrieval that finds superficially similar but irrelevant passages, and no access control, so that confidential documents surface in answers to people who should not see them. Investing in clean, well-owned source content and in testing retrieval quality usually matters more than the choice of model.
Building an AI integration that lasts: a twelve-week outline
The timeline sets out a realistic sequence for a first production AI feature built on an architecture that will support the second, third and tenth. Timings assume a validated use case; our guide to AI feature prototyping covers validating the idea first.
The abstraction layer appears in week two, not as a later refactor. Building it alongside the first feature costs a few extra days; retrofitting it after several features have been built with direct calls typically costs weeks. Every subsequent feature then starts with logging, validation, evaluation and model choice already in place.
AI integration maintainability — where most UK businesses sit
Combining coupling, observability, data grounding, reliability and evaluation gives an indication of how maintainable an organisation’s AI integrations are. The gauge reflects a typical UK business with one to five AI features in production, built without a deliberate architecture.
A score in the mid-thirties usually reflects features that work but are fragile: direct calls to one provider, prompts in code, free text parsed with assumptions, little logging, no evaluation and no fallback. Each feature was built sensibly in isolation; together they form a set of independent dependencies on a fast-changing market.
Introducing an abstraction layer, moving existing features onto it, adding structured outputs, central logging and a basic evaluation set typically lifts the score into the seventies. Most of the effort goes into the first migration; later features inherit the benefits at little extra cost.
The cost of building it right versus fixing it later
The table compares indicative costs of building architectural elements from the start with the typical cost of retrofitting or living without them. Figures are broad ranges for UK SMEs with a small number of AI features, using external developers at typical UK rates; actual costs depend heavily on complexity and existing systems.
| Element | Cost to build in from the start | Typical cost of retrofitting or doing without | Who feels it |
|---|---|---|---|
| Abstraction layer with model configuration | £3,000 to £10,000 alongside the first feature | £8,000 to £30,000 to migrate several features later | Development budget, delivery timelines |
| Structured outputs and validation | A few days per capability | Repeated incidents when formats drift; data clean-up | Operations, downstream systems |
| Central logging and cost tracking | £1,000 to £4,000 | Unexplained bills; inability to investigate complaints | Finance, customer service |
| Evaluation test set and process | £2,000 to £6,000 initially, then ongoing upkeep | Quality regressions found by users after model changes | Customers, reputation |
| Fallbacks and asynchronous processing | £2,000 to £8,000 depending on workflows | Outages and timeouts that stop business processes | Staff, customers |
The pattern is familiar from other areas of software: the cheapest time to build a foundation is before anything depends on it. The figures also show why AI integration architecture is a business decision, not just a technical one. The costs of doing without fall on operations, finance, customer service and reputation, not only on the development team. Our guide to custom AI development covers budgeting for AI projects more broadly.
Synchronous versus asynchronous integration
Language model calls are slow compared with most application operations. A database query returns in milliseconds; a model call can take several seconds, longer for complex tasks or long documents, and occasionally much longer when providers are busy. How you handle that latency shapes both the user experience and the reliability of the system.
Synchronous integration
The application calls the model and waits for the answer before continuing. This suits interactive features where a person is waiting and expects a response: a chat assistant, drafting a reply, suggesting a classification while someone fills in a form. Streaming the response as it is generated makes waits feel shorter. Synchronous integration needs firm timeouts and a sensible behaviour when they are reached, such as offering to retry or continuing without the AI suggestion, so that a slow provider does not freeze the application.
Asynchronous integration
The application places a request on a queue and carries on; the AI work happens in the background, and the result is delivered when ready, by updating a record, notifying a user or triggering the next step in a process. This suits work where nobody is waiting at a screen: processing incoming documents, classifying a batch of emails overnight, enriching records, generating reports. It handles provider slowness and rate limits gracefully, allows automatic retries and makes it easy to process large volumes without overwhelming either the provider or your systems.
Choosing between them
The question to ask is whether a person needs the result to continue what they are doing right now. If yes, synchronous with streaming and timeouts. If no, asynchronous. Many problems in early AI features come from making background work synchronous — for example, processing an uploaded document while the user waits on a loading screen — when it could have been queued and the user notified when it was done.
Designing for failure in both
Either way, design for the model being unavailable. Interactive features should degrade gracefully, so the underlying task can still be completed without AI. Background processes should retry with limits and route items that keep failing to a person rather than losing them. A fallback to an alternative model, configured in the abstraction layer, covers many provider outages without anyone noticing.
Controlling cost through the integration layer
AI costs are usually charged by usage, typically by the amount of text sent and received, so they grow with adoption and can surprise a business that has no visibility. The integration layer is the natural place to control them.
- Match models to tasks. Simple classification or extraction often works well on smaller, cheaper models; reserve the most capable models for tasks that genuinely need them. Evaluation sets make it possible to test whether a cheaper model is good enough.
- Send less. Long prompts, entire documents and verbose instructions all add cost. Retrieval that supplies only the relevant passages, and prompts trimmed to what matters, reduce cost and often improve answers.
- Cache where appropriate. Identical or near-identical requests, such as summaries of the same document or answers to common questions, can be cached rather than recomputed. Some providers also offer reduced pricing for repeated prompt content.
- Use batch processing for background work. Where results are not needed immediately, some providers offer lower prices for batch requests processed over a longer period.
- Set budgets and alerts. Track spend per capability and per application, and alert when usage departs from the norm, so that a loop calling the model repeatedly or an unexpectedly popular feature is spotted in hours rather than at month end.
Each of these is a change in one place when calls pass through a single layer, and a project spread across several applications when they do not.
How many AI features can survive a provider outage
Across the production AI features we review, the share with no fallback when the model provider is slow or unavailable — no alternative model, no non-AI path and no queue to retry later — is high.
The figure is indicative rather than a formal survey. Provider outages are not frequent, but they do happen, and slowdowns at busy times are more common still. When an AI feature sits in the middle of a business process — triaging incoming enquiries, processing orders, extracting data from invoices — a provider problem without a fallback stops the process. With an abstraction layer, adding an alternative model is a configuration task; with asynchronous processing, work simply waits and retries; with a non-AI path, staff can carry on manually. Each is inexpensive compared with a stopped process.
Legacy system AI integration
Some of the most valuable AI use cases involve systems that were never designed to be called by anything external: an ageing ERP, a bespoke database application built fifteen years ago, a line-of-business package with no modern interface, or a mainframe-era system at the heart of operations. Legacy system AI integration is often where projects stall, and where architectural discipline pays off most.
Reading from legacy systems
The safest starting point is read-only access. Options, roughly in order of preference, include:
- An existing interface. Many older systems have an API, export function or reporting interface that is underused. Check before building anything new.
- A reporting copy of the database. Replicating data to a separate database or warehouse lets AI features read freely without loading the live system or risking changes to it.
- Scheduled exports. Regular file exports into a staging area are unglamorous but robust, and often sufficient where data does not need to be real-time.
- Change capture or events. For near-real-time needs, capturing changes as they happen and publishing them for other systems to consume.
Writing back to legacy systems
Writing AI outputs into a legacy system carries more risk. Prefer writing through whatever official interface exists, so that the system’s own validation and business rules apply. Where none exists, consider whether AI outputs can be placed in a staging area for a person or a controlled process to apply, rather than writing directly to the database. Direct database writes that bypass application logic can corrupt data in ways that are hard to detect.
Wrap the legacy system once
Rather than connecting each AI feature to the legacy system separately, build a small integration service that exposes the data and actions you need through a clean, modern interface. AI features and other applications then use that service, and the legacy system’s quirks are contained in one place. When the legacy system is eventually replaced, only the wrapper changes.
Screen automation as a last resort
Where there is truly no interface, robotic process automation that drives the user interface can bridge the gap. It is fragile, because any change to screens can break it, and it should be treated as a temporary measure while a better route is found.
Keep a person in the loop for consequential actions
When AI outputs will change records in core systems — customer accounts, orders, financial entries — a review step before changes are committed is usually wise, at least until the feature has a proven track record. Our guide to AI agent reliability testing covers how to build confidence before removing human review.
The 12-point AI integration architecture checklist
The checklist covers the architectural essentials for AI features that will still be maintainable in two years.
- Route every model call through one layer. No direct provider calls from application code.
- Expose task-based capabilities. Applications request tasks, not models.
- Make model choice configuration. Different tasks can use different models; switching is not a code change.
- Version and review prompts. Store them centrally and record which version produced each output.
- Use structured outputs and validate them. Downstream systems receive predictable data.
- Log requests, latency, usage and cost. With protection appropriate to any personal data in the logs.
- Ground answers with retrieval where facts matter. Keep source content current and apply access controls at retrieval.
- Reserve fine-tuning for behaviour, not knowledge. And weigh the extra coupling it creates.
- Choose synchronous or asynchronous deliberately. Queue work nobody is waiting for.
- Plan for failure. Timeouts, retries, alternative models and non-AI paths.
- Wrap legacy systems once. Prefer read-only access and official interfaces for writes.
- Maintain an evaluation set. Run it whenever prompts or models change, and sample production outputs.
Prompts, retrieval sources and evaluation sets are living assets. Each needs a named owner who keeps it current, approves changes and reviews quality. Architecture makes change possible; ownership makes sure change happens deliberately rather than by accident.
Case study: three features, one layer
A UK insurance broker with around 120 staff had built three AI features over eighteen months: a tool that summarised incoming client emails for account handlers, a document extractor that pulled key fields from policy schedules, and an internal assistant that answered staff questions about procedures. Each had been built by a different developer, and each called the same model provider directly from its own application, with prompts embedded in code.
Problems surfaced within weeks of each other. The provider announced the retirement of the model version the extractor relied on; testing on the replacement showed it returned dates in a different format, which broke the import into the broker’s policy administration system. The procedures assistant was giving answers based on documents that had been superseded months earlier, because its knowledge had been pasted into the prompt at launch. Monthly AI costs had tripled with no way of telling which feature was responsible, and an account handler’s complaint about a misleading email summary could not be investigated because nothing had been logged.
The broker commissioned a small internal AI service. All three features were moved onto it over six weeks. Each became a named capability with versioned prompts, structured outputs validated against defined formats, central logging with cost per feature, and a fallback model. The procedures assistant was rebuilt using retrieval from the broker’s document management system, with access controls matching existing permissions and citations to source documents. The extractor moved to an asynchronous queue, with items failing validation routed to an administrator rather than imported. An evaluation set of real examples was built for each capability.
When the model change came into effect, the extractor was switched to the replacement model after evaluation showed equivalent accuracy, with no application changes. Cost reporting showed the email summariser was responsible for most of the increase; moving it to a smaller model cut its cost substantially with no measurable loss of quality. A fourth feature, built later, took a fraction of the time of the first three.
“Each of the original features worked. What we did not have was anything holding them together. Now a model change is a decision we make with evidence, not an emergency, and I can tell the board exactly what each feature costs.” — Head of Operations, UK insurance broker
The lesson is not that the original developers did poor work. Each feature made sense on its own. The problem was the absence of a shared layer, which turned three reasonable features into three separate dependencies on a market that changes every few months.
Common AI integration mistakes
The mistakes below account for most of the rework we see in AI integration projects.
- Calling providers directly from application code. Couples every feature to one provider and one model.
- Embedding prompts in code without versioning. Behaviour changes become untraceable.
- Parsing free text with assumptions. Small format changes break downstream systems.
- Pasting knowledge into prompts. It goes stale; retrieval keeps answers current.
- Fine-tuning to teach facts. Expensive to update and tied to one provider.
- Making background work synchronous. Users wait, and timeouts break processes.
- No fallback. A provider problem stops the business process.
- Writing directly into legacy databases. Bypasses business rules and risks corruption.
- No logging or cost tracking. Bills cannot be explained and complaints cannot be investigated.
- Changing models without evaluation. Quality regressions are discovered by users.
A working demo proves that a model can do the task. It says nothing about whether the integration can survive model changes, scale, failures, audits or a second and third feature. Before a demo becomes a production feature, decide deliberately how it will connect to the abstraction layer, business data and existing systems, rather than shipping the demo code.
Evaluation and observability: knowing when quality changes
Traditional software either works or fails visibly. AI features can degrade quietly: slightly less accurate extractions, summaries that miss important points, answers that drift in tone. Without deliberate evaluation and monitoring, these changes are discovered by users, often long after they began.
Build an evaluation set early
For each capability, collect a set of real inputs with known good outputs or clear criteria for judging them: documents with correctly extracted fields, enquiries with correct classifications, questions with approved answers. A few dozen well-chosen examples covering common and awkward cases are enough to start. Run the set whenever a prompt, model or retrieval source changes, and compare results before releasing the change.
Choose measures that match the task
Extraction and classification can be scored automatically against expected values. Open-ended outputs such as summaries are harder; a combination of human review of samples and automated checks for specific requirements — required facts present, no prohibited content, correct format — usually works. Some teams also use a second model to grade outputs against criteria, which can be useful if its judgements are checked against human review.
Monitor production
Central logs make it possible to watch latency, error rates, validation failures, cost and volumes over time, and to alert when they change. Sampling a proportion of production outputs for regular human review catches quality changes that metrics miss. A simple way for users to flag poor outputs adds a valuable signal.
Close the loop
Add problematic real cases to the evaluation set, so that each failure becomes a test that future changes must pass. Over time the evaluation set becomes one of the most valuable assets in the system, because it captures what good looks like for your business.
Data protection and where your data goes
This section is a general overview, not legal advice. Take advice on your specific circumstances and check current guidance from the Information Commissioner’s Office.
Every model call sends data to a provider. Architecture determines how much data is sent, what kind, to whom and with what records, which makes it central to meeting data protection obligations.
- Know what is sent. Central logging and a single integration point make it possible to state exactly what data each capability sends to which provider.
- Minimise it. Send only the data needed for the task. Redact or pseudonymise personal data where the task does not require it.
- Check provider terms. Business and enterprise terms from major providers commonly address whether data is retained or used for training, and where it is processed. Terms differ between consumer and business offerings and change over time, so check the current position for the specific service you use.
- Consider processing location. Some providers offer processing in particular regions. Whether that matters depends on your data, contracts and risk assessment.
- Assess the risk. Where AI processing of personal data is likely to result in high risk to individuals, a data protection impact assessment is generally required, and it is good practice for significant new uses even where it is not strictly required.
- Be transparent. Privacy information should reflect the use of AI processing where it involves personal data, and people should be able to understand significant decisions made with AI assistance.
An abstraction layer makes each of these easier because controls are implemented and evidenced in one place. Scattered direct calls make them difficult to evidence at all.
Tools, agents and connecting models to actions
Increasingly, AI features do more than produce text. They call tools: look up an order, search a knowledge base, create a ticket, update a record. Agentic systems chain several such steps together to complete a task. This raises the stakes for integration architecture, because a model with access to tools can take actions in your systems.
The same principles apply with more force. Tools should be defined as narrow, well-documented capabilities exposed through your integration layer, not as broad access to underlying systems. Each tool should enforce its own permissions and validation, regardless of what the model asks for. Actions with real consequences should require confirmation, either from a person or from deterministic business rules. Every tool call should be logged alongside the model request that triggered it.
Emerging standards for describing tools to models, such as the Model Context Protocol, can make it easier to expose the same tools to different models and applications, which supports the goal of avoiding lock-in. They are worth considering when designing tool interfaces, though the underlying disciplines of narrow permissions, validation and logging remain your responsibility. For a comparison of the main model providers themselves, see our guide to GPT, Claude and Gemini for business integration.
Who should own the AI layer
An abstraction layer is a small piece of software with an outsized role, so its ownership deserves thought. Someone needs to maintain it, keep provider integrations current, manage configuration, review prompt changes and keep evaluation sets up to date.
For businesses with an in-house development team, the layer usually belongs with that team, with clear documentation and at least two people who understand it, so that knowledge does not rest with one developer. For businesses relying on an external partner, it is important that the layer, its configuration, prompts and evaluation sets are owned by the business and held in its own repositories and accounts, so that the arrangement does not become a different kind of lock-in. Ask any partner building AI features how model calls are structured, where prompts live, how changes are evaluated and what you would receive if the relationship ended.
Business ownership matters too. Prompts encode business rules and tone; retrieval sources encode business knowledge; evaluation sets encode what good looks like. Each needs a business owner as well as a technical one, typically the person responsible for the process the AI supports. Getting this right is what keeps AI features aligned with the business as both change.
At a glance: AI integration architecture
| Decision | Approach that ages well | Approach that creates maintenance problems |
|---|---|---|
| Where model calls are made | One abstraction layer for all applications | Direct provider calls in each application |
| Application interface | Task-based capabilities | Raw prompts to named models |
| Model selection | Configuration per capability, chosen on evaluation | Hard-coded in application code |
| Prompts | Central, versioned, reviewed | Embedded in code, untracked |
| Outputs | Structured and validated | Free text parsed with assumptions |
| Business knowledge | Retrieval from current, access-controlled sources | Pasted into prompts or fine-tuned in |
| Fine-tuning | For behaviour and format where justified | To teach facts that change |
| Latency handling | Synchronous with streaming for interactive; queued for background | Users waiting on background work |
| Failure handling | Timeouts, retries, alternative models, non-AI paths | No fallback |
| Legacy systems | Wrapped once; read-only first; official interfaces for writes | Direct database writes from each feature |
| Observability | Central logs with usage, cost and prompt version | One monthly bill and no records |
| Quality | Evaluation sets run on every change; production sampling | Users discover regressions |
| Tools and actions | Narrow, permissioned, logged; confirmation for consequential actions | Broad system access granted to the model |
How Cloudswitched approaches AI integration
We build AI features on an architecture designed to outlast the model in use today. Every engagement starts with the capability and how its quality will be judged, then puts a lightweight abstraction layer in place alongside the first feature: task-based interfaces, model choice as configuration, versioned prompts, structured outputs with validation and central logging of usage and cost.
We ground answers in your data using retrieval from authoritative, access-controlled sources, reserving fine-tuning for cases where behaviour rather than knowledge is the problem. We choose synchronous or asynchronous integration according to whether a person is waiting, design fallbacks so provider problems do not stop your processes, and connect to legacy systems through a single, contained integration service with read-only access first and official interfaces for anything written back.
For businesses that already have AI features built with direct calls, we can review the current state, introduce an abstraction layer and migrate existing features onto it with evaluation sets that make future model changes a measured decision. No architecture removes every risk in a fast-moving field, but separating your business logic from the specifics of any one model keeps you in control as the market changes.
Want AI features that stay maintainable?
Talk to us about AI integration architecture, retrieval over your business data, legacy system integration or reviewing an existing AI build.
Talk to an AI Development SpecialistFrequently Asked Questions
What is LLM integration architecture?
The design of how large language models connect to your applications, data and processes: where calls are made, how prompts and outputs are managed, how business data reaches the model, how results flow back, and how the arrangement is monitored, secured and changed.
Why is calling a model provider directly from application code a problem?
It ties each feature to one provider and model. When models are retired, prices change or better options appear, every feature needs code changes. It also scatters prompts, logging, cost tracking and data controls across applications.
What is an AI abstraction layer?
An internal service or module through which all applications access AI models. Applications request tasks; the layer handles model selection, prompts, output validation, retries, fallbacks, logging and data controls in one place.
What is retrieval augmented generation?
A technique that finds relevant information from your documents or data when a question is asked and includes it in the prompt, so the model answers using current business knowledge. It keeps answers up to date and allows sources to be cited.
Should we use RAG or fine-tuning?
For questions about facts, documents and records, retrieval augmented generation is usually the better starting point. Fine-tuning suits teaching consistent behaviour, style or format, and is less suitable for knowledge that changes. Many systems use both.
When should AI integration be asynchronous?
Whenever nobody needs the result immediately to continue what they are doing, such as processing documents, classifying batches or enriching records. Queuing the work handles slow responses, rate limits and outages gracefully.
How do we connect AI to legacy systems?
Start with read-only access through an existing interface, a reporting copy of the database or scheduled exports. Wrap the legacy system once in an integration service. Write back through official interfaces where possible, with human review for consequential changes.
How do we avoid lock-in to one AI provider?
Route all calls through an abstraction layer, keep model choice in configuration, store knowledge in retrieval sources rather than fine-tuned models, use structured outputs, and maintain evaluation sets so you can compare providers on evidence.
How do we know if a model change has affected quality?
Run an evaluation set of real examples against the new model or prompt before switching, monitor production metrics and validation failures, and sample outputs for human review after the change.
Is it safe to send business data to AI providers?
It depends on the data, the provider’s current terms and your controls. Minimise and redact what is sent, use business terms that address retention and training, consider processing location, assess the risk for personal data, and take advice where needed.
How much does an abstraction layer cost?
For an SME, building a first version alongside the initial feature typically adds a few thousand pounds and a week or two. Retrofitting one after several features have been built with direct calls usually costs considerably more.
Can AI features take actions in our systems?
Yes, through tools that the model can call. These should be narrow, permissioned and logged, with confirmation required for consequential actions, so the model can only do what you have explicitly allowed.
Related reading
More guidance for UK businesses from the Cloudswitched blog:
- Microsoft 365 Copilot agents and custom agents
- Cyber Essentials scope: deciding what is in and out of your certification
- Website security hardening: protecting your site from common attacks
- IT governance for UK SMEs: board reporting and risk registers
- Network change management: making changes without outages
Build AI that survives the next model change
Cloudswitched designs AI integrations with a clean abstraction layer, grounded answers, sensible failure handling and contained legacy connections, so your AI features keep working as the market moves.
Talk to an AI Development Specialist
Cloudswitched
London-based managed IT services provider offering support, cloud solutions and cybersecurity for SMEs.
