Most companies have an AI strategy by now. Very few have an LLM strategy, and many don’t see why they would need one.
The model gets treated as a purchase. Someone picks a provider, signs an enterprise agreement, and the real work moves on to use cases, pilots and adoption. The model sits underneath everything as a utility, like electricity.
That is a reasonable instinct, and it may be the most expensive assumption in your AI programme.
Where the time goes
Look at how AI time is spent in a typical large organisation and a clear pattern shows up. Most of it goes to five places.
Use cases. Workshops, idea backlogs and prioritisation matrices. Which processes could AI improve? Which business line goes first?
Governance. Risk frameworks, acceptable-use policies, model risk reviews and AI Act readiness. In regulated industries this work is necessary, and it takes a lot of time.
Agentic architecture. Choosing agent frameworks, designing orchestrators, and working out how agents plan, hand off to each other and call tools. It’s the most interesting design problem in the field right now, so it draws the best architects.
Building and integration. Teams connect agents to core banking systems, CRMs, document stores and legacy APIs. They build MCP servers, wrap old services and fight with authentication, data mapping and error handling.
Monitoring and refining. Once something is live, teams track its output quality, adjust prompts, add guardrails, handle edge cases and keep improving. Many teams underestimate this step, because it never really ends.
Why that could be a mistake
Your costs are set by the model choice, and they grow with success
A pilot on a frontier model is cheap. The same workload at production volume, across every channel and country, is not. Inference cost grows with usage, so the more successful your AI programme is, the bigger the bill for any architecture decisions you skipped.
Much of that work never needed a frontier model. NVIDIA Research estimates that 40–70% of calls in common agent frameworks could be handled by small language models. Serving a 7B model is 10–30x cheaper than serving a 70–175B one. Independent routing benchmarks such as RouteLLM keep about 95% of frontier quality at a fraction of the cost.
A caveat is in order. Sending every request to your most expensive model is a strategy too, just one that nobody chose on purpose.
Right-sizing often improves quality
The usual objection is that smaller models mean worse answers. The evidence says otherwise, at least for well-scoped tasks.
Checkr, a background-check company, moved a classification task from GPT-4 to a fine-tuned Llama model. Accuracy went from about 88% to 97%, and latency dropped from around 15 seconds to under 0.15 seconds. Salesforce put five specialist models in front of the frontier model in Agentforce. Safety accuracy rose from 95% to 99%, topic drift halved, and intent classification became about 55x faster.
The model market changes faster than your contracts
The best model today probably won’t be the best in 18 months. New models come out every few months, prices drop, and providers change their terms, retire older versions and shift their roadmaps. You should expect to change models several times.
The trouble is how most AI applications are built. Each team connects directly to one provider: its API, its request format, its way of calling tools. Multiply that across dozens of applications and agents, and the provider is now built into your codebase. Changing models is no longer a configuration change. It becomes a migration project across every team, followed by retesting and a new round of approvals. So most organisations don’t switch, even when a better or cheaper option exists. Their AI strategy ends up following one vendor’s roadmap, pricing and terms, whether they chose that or not.
The fix is an architecture pattern banks already know well. Think of a payment switch. Your channels don’t connect to each card network directly. They connect to the switch, and the switch decides where each transaction goes. An LLM gateway does the same for models. Applications and agents send every request to one internal layer. That layer decides which model handles the request, and it also manages authentication, fallback if a model fails, data screening, and cost and quality tracking for each model.
The banks that are furthest ahead have built exactly this:
- Goldman Sachs runs several models, both commercial and open-source, behind one internal gateway that handles routing and compliance checks.
- Revolut built a central LLM gateway with mandatory fallback chains and monitoring for each model. It runs its financial-crime workload on open-weight models at about a third of the cost of the alternative, handling up to 1.2 million tickets a month. Because of the gateway, the products using that workload didn’t need to change.
The takeaway: your ability to change models is not decided when you pick a model. It is decided when you design the layer between your applications and the models. With a gateway in place, a model change is a routing decision. Without one, it becomes a programme of work.
Sovereignty is an architecture decision, not a policy
For banks and insurers, where data goes matters as much as what the model can do. HSBC is deploying self-hosted Mistral models on its own systems. Wells Fargo uses small models on its own infrastructure to strip personal data before anything reaches a cloud model.
You can’t fix this with a policy document after the fact. Whether sensitive data can ever leave your perimeter depends on where your models run and how requests are routed, and that is decided at the LLM layer.
What an LLM strategy actually covers
An LLM strategy doesn’t mean building your own foundation model. It means making a few decisions on purpose instead of by default:
- Portfolio. Which kinds of model you use (frontier, mid-size, small specialist, or none) and which kinds of work go to each.
- Hosting. Which workloads run on public cloud APIs, which on private cloud, and which on-premises, graded by how sensitive the data is.
- Routing and gateway. One control layer between applications and models, handling authentication, routing, fallback, PII screening, caching and cost tracking.
- Evaluation. Your own test sets, so you can swap a model based on evidence rather than a vendor’s benchmark.
- Exit plan (if applicable). How long it would take to replace your main model provider. If the honest answer is “a year,” that is a risk worth recording.
The bottom line
Companies that pour their time into use cases, governance, architecture, integration and monitoring, but never make a deliberate choice about their models, will still end up with an LLM strategy. It just won’t be one they chose. They’ll discover it later in three ways: rising invoices, how hard it is to change vendors, and sensitive use cases they can’t implement because their models run in the wrong place.
Leave A Reply