Denshin / Blog / AI
Open-weight models: when self-hosting makes sense
An honest cost and control analysis of running open weight models yourself. What open weight really means, why the licence matters, the genuine reasons to self-host, the costs teams underestimate, the managed inference middle path, and a decision rule that starts with a hosted API and measurement.
Denshin Engineering · Engineering Team · 27 August 2026 · 7 min read
Self-hosting a model is one of those decisions that feels like control and often turns into a second product to maintain. Sometimes it is exactly right: a data residency clause, an air gapped site, steady high volume, a latency budget you cannot meet over the public internet. Sometimes it is a GPU sitting idle at night while a small team learns inference serving instead of shipping features. This is the honest version of the analysis, including the costs that do not appear in the spreadsheet until month three.
Open weight, open source and API hosted
Open weight means the trained parameters are published, so you can download them and run inference yourself. Open source, used strictly, would also mean an OSI approved licence and enough of the training recipe and data to reproduce the result, which most released model weights do not provide. API hosted means the weights stay with a vendor and you buy inference by the token. The three get used interchangeably in conversation, and the difference decides what you are actually allowed to do.
Several families publish weights, including Meta's Llama, Mistral, Qwen and DeepSeek, and the exact set and their terms change as new releases land, so check what is current rather than trusting a list in an article. The relevant point is that "you can download it" and "you can use it for anything" are different statements.
Read the actual licence, not the blog post announcing it
Licence terms on published weights vary widely and some carry real restrictions: acceptable use policies, limits tied to the size of your user base, naming or attribution requirements, restrictions on using outputs to train other models, or non commercial and research only terms. Two models described in the same news article as "open" can carry very different obligations.
Practical advice, and this is not legal advice: before you build on a set of weights, open the licence file that ships with them, read it, and if you are shipping to clients or in a regulated sector have someone qualified read it too. Record the licence and the version of the weights in the same place you record other dependencies. Terms have changed between releases from the same family, so a decision made a year ago does not carry forward automatically.
Genuine reasons to self-host
- Data residency and compliance. A contract or a regulator requires data stays in a jurisdiction or inside your own network, and no hosted vendor offers that in the region you need.
- Air gapped or on-premise deployment. Some environments simply have no route to a public API. That is a constraint, not a preference, and it decides the matter.
- Very high steady volume. Owned or reserved capacity beats per token pricing once utilisation is genuinely high and genuinely constant. The key word is steady: this is the same fixed capacity versus usage based arithmetic as serverless math, when Lambda is cheaper than a server, and it turns on utilisation rather than on peak throughput.
- Latency control. Colocating the model with your application removes a network hop and, more importantly, removes a queue you do not control. If you have a hard interactive budget, this can be the deciding factor.
- Freedom from deprecation. Weights you hold do not get retired on someone else's schedule. For a long lived product with a validated pipeline, being able to keep the exact same model for years has real value.
- Freedom to fine-tune and to shape the runtime. Full control over adapters, quantisation, sampling and inference internals, which matters when you have a narrow task and enough data, one of the cases in RAG versus fine-tuning.
Notice that four of those six are constraints rather than optimisations. That is the pattern: self-hosting usually earns its place because something forbids the alternative, not because it looked cheaper on a napkin.
The costs people underestimate
GPU capacity, and the hours it sits idle
Accelerator memory has to hold the weights plus the key value cache for every concurrent request, and the weights alone need roughly the parameter count multiplied by the bytes per parameter at your chosen precision, before any runtime overhead. Quantisation reduces that at some quality cost you have to measure rather than assume. Whatever hardware you land on, you pay for it while your users are asleep, so the number that decides the economics is average utilisation, not peak. Bursty traffic is the worst case for owned capacity and the best case for per token pricing.
Inference serving is its own discipline
Getting acceptable throughput means continuous batching, sensible cache handling, tuned concurrency, queueing, timeouts and backpressure, plus autoscaling that copes with slow starting nodes and scarce instance types. Mature serving stacks exist and are good. They are still infrastructure your team now owns, monitors and gets paged for.
Evaluation does not get cheaper
You still need an eval set, and now you need it more, because every change you make to quantisation, serving configuration or an adapter can move quality. Self-hosting adds knobs, and knobs need measurement. Budget for that from the start.
Security patching and access control
Model servers, runtimes, drivers and the container images around them all get CVEs. Someone has to track them and patch. You also own authentication, tenancy isolation, rate limiting, logging and abuse controls that a hosted API provided for you.
The engineering time that is not building your product
This is the largest and least visible cost. For a small team, the hours spent tuning a serving stack are hours not spent on the features customers asked for. We are a small studio and we weigh this heavily: infrastructure we run is infrastructure we maintain forever, and it competes directly with delivery.
Three options, honestly compared
| Hosted API | Managed open weight inference | Self-hosted |
|---|
| Time to first version | Hours | Hours to days | Weeks |
| Cost shape | Per token, scales to zero | Per token or per hour endpoint | Fixed capacity, plus people |
| Data path | Vendor terms apply | Vendor terms apply | Entirely yours |
| Deprecation risk | Vendor schedule | Lower, weights are portable | None |
| Ops burden | Minimal | Moderate | Substantial and permanent |
| Best when | Variable volume, small team | You want portability without a GPU fleet | A hard constraint or very steady volume |
The middle path most teams should look at first
Managed inference for open weight models sits between the two. You pick the weights, someone else runs the accelerators, and you pay per token or per endpoint hour. You keep most of the portability argument, because the same weights can be moved to another host or brought in house later, and you skip the fleet. It is also the cheapest way to find out whether an open weight model is good enough for your task before you commit capital or headcount to it. Offerings and prices in this space move quickly, so compare current options at decision time rather than relying on a comparison written earlier.
A decision rule you can apply this week
- Start on a hosted API unless a hard constraint forbids it. Ship the feature, learn what good output looks like, and build the eval set.
- Measure for a month. Requests per hour across the day, tokens in and out per request, peak to average ratio, and what a failure actually costs you.
- Check the constraints honestly. Is there a written residency or air gap requirement, or is it a preference someone expressed in a meeting? Constraints decide, preferences do not.
- Do the volume arithmetic with your real numbers. Compare your monthly API spend against the fully loaded cost of capacity plus the engineering time to run it, at your actual average utilisation. If the two are close, stay on the API, because the tie breaker is the maintenance you avoid.
- If you move, move one workload. Pick the highest volume, most stable, least latency sensitive task. Often it is classification, extraction or embedding rather than the customer facing chat, which is the same routing logic as choosing a smaller model where it wins.
- Keep the seam. Whatever you run, keep the provider behind your own interface so this decision stays reversible.
What to do next
Before the next planning session, get two numbers on paper: your current or projected monthly inference spend, and your average utilisation if you owned capacity sized for peak. If utilisation is low, the answer is a hosted API and the conversation is over. If it is high and steady, or if a contract requires residency, price the managed open weight option before you price a GPU fleet, because it usually gets you most of what you wanted for a fraction of the commitment. The broader cost discipline is in adding AI without setting money on fire.
If you want a second opinion on whether self-hosting is justified for your workload, talk to Denshin. We would usually rather help you avoid running a GPU fleet than help you build one, and we will say so plainly if the numbers point the other way.
Tags: Open Weight Models, LLM, Self-Hosting, AI Infrastructure, AI Costs
All posts · Work with Denshin