The (im)possibilities of Private AI

Three powers and trade-offs

To understand what private AI entails, it helps to look at how AI is distributed globally. There are roughly three forces that define the market.

The US is leading the way. American models like OpenAI, Claude, and Gemini are the most advanced and capable, but also the most expensive. They are offered via APIs and are widely accessible. But every query means that data leaves the organization and enters the Cloud Act jurisdiction. Those who do not see this as a problem immediately reap the benefits of the most advanced AI currently available. Those who do see it as a problem face a fundamental choice.

China has developed open-source models that can be hosted independently. Models like Qwen are highly efficient, available, installable in one's own data center, and queryable without data leaving the internal environment. This offers control and sovereignty but requires significant hardware investments. Furthermore, the model is on average about eight months behind the American frontier.

Europe has Mistral, a French provider and developer of AI models, which is solid in terms of architecture and falls entirely under European legislation. It is accessible via API and offers the most secure route for organizations concerned about compliance and data location. However, Mistral also lags behind the US and China, and as long as Europe does not substantially increase investment in AI development, that will not change anytime soon.

The core question is not which model is the smartest. The core question is what an organization wants to do with AI, what data is required for that, and what level of risk is acceptable.

The addictive nature of convenience

There is something seductive about working with large language models via an API. A few lines of code and intelligence is available. Dashboards are populated, notifications analyzed, reports generated. The barrier to entry is low. The result is impressive, and that is exactly what makes it dangerously easy.

The convenience of an API creates a reflex: if something can be solved with an AI query, that becomes the standard approach. What used to be a script that was built once and then ran 'for free' now becomes an ongoing API call with associated token costs. Those costs are small per query, but at scale, they quickly become substantial and opaque.

The question organizations ask themselves too rarely is: when is true intelligence needed, and when does a classic script or a smaller local model suffice? Not every notification, every batch process, or every automated task justifies a query to a frontier model. But the threshold to do so anyway is so low that the choice is rarely made consciously.

Moreover, no one checks the code that comes out anymore. It works, so why look? The lack of control and awareness is exactly where the vulnerability lies. Not just in terms of costs, but also in terms of data security and integrity.

Data is the real asset

As mentioned, the value of AI consists of two components: the model and the data. Anyone looking only at the model is missing half the equation. A smart model without organization-specific data provides generic output. A smart model fed with internal systems, customer data, process data, or business-critical information provides insights that are truly distinctive. That data is the crown jewel. And it is precisely those crown jewels that are exposed when external AI services are used without awareness.

This is not a theoretical risk. At municipalities, government agencies, and educational institutions, employees are installing AI clients on their laptops on their own initiative and feeding them all sorts of information. Contracts, policy documents, customer data, internal analyses. Not out of malice, but out of convenience and ignorance regarding what happens to that data. Private AI is therefore not a technical luxury. It is an organizational response to a real risk.

Private AI: attractive but not cheap

The promise of private AI is clear: the experience of a powerful language model, but fully under your own control, on your own infrastructure, without any data leaving the organization. That sounds like the ideal solution.

The reality is that this requires a serious investment. Large open-source models with more than 120B parameters require a massive amount of GPU memory to run. A consumer-grade GPU card typically has 12–32GB of memory. Larger data center cards go up to 141GB. The quality of the model is determined by, among other things, the available graphics memory: the more memory, the larger the model that can be loaded, the more spacious the context window, and the better the caching, which improves the model's response speed. A full-fledged, enterprise-grade private AI stack can easily cost hundreds of thousands of euros. The allure of the result makes that business case uncomfortable.

That does not mean there aren't well-thought-out, small-scale alternatives. A smaller local model deployed for a specific, well-defined use case is often a wiser choice than building a full AI factory. It is not about the largest infrastructure, but about the right infrastructure for the right requirement.

VCF 9 brings Private AI

VMware Cloud Foundation 9 introduces a so-called Private AI Practice: an integrated way to make GPU capacity available, 'slice' it, and link it to applications and containers. The platform can indeed help in setting up a sovereign AI environment on private infrastructure. But it is important to understand what VCF 9 does and does not solve.

VMware has traditionally been a virtualization platform. It partitions hardware: it turns large physical resources into smaller, logically separated units. That is useful for regular workloads. But large AI models work differently. They require as much contiguous GPU memory as possible. Partitioning hardware is counterproductive in that case: the more you virtualize, the less memory is available per model.

VCF 9 clearly has its place, then. That is in specific, well-defined use cases where a smaller model needs to run within an existing private cloud environment. Think of an organization that wants to automate a targeted AI task, keep the data private, and does not have the budget or the scale for a full-fledged AI factory. In that scenario, VCF 9 and its Private AI Practice can deliver a platform at the push of a button that unlocks GPUs, links applications, and manages containers. That is valuable functionality.

But for those who want to host large open-source models to realize a broad, Claude-like experience internally, VCF 9 is not the primary choice. For that, bare-metal GPU infrastructure with maximum available memory is the logical path. That has little to do with VMware. Technology is not an ideology. The right choice depends on the use case.

Scale determines the use case

There is a clear dichotomy in how organizations approach private AI. On one hand, there are organizations that want to realize the full AI experience internally. They want to run a large, powerful model on their own hardware, fully sovereign, with complete control over data and output. That requires serious infrastructure and a well-thought-out business model. The scale of the model determines the quality of the output, and quality has a price.

On the other hand, there are organizations with a specific, well-defined need. They want to automate one process, analyze one category of reports, or query one data source. A large model is unnecessary for that. A smaller, fine-tuned model running locally offers sufficient intelligence for the task, without the infrastructural burden of a full AI platform. And without the ongoing token costs of an external API.

Between these two extremes are SMEs and the mid-market: organizations that are thinking about their data but do not want to invest 'hundreds of thousands' in their own hardware. For them, an intermediate solution is the most logical route. Hosting with a party that already manages the data, offers a sovereign platform, and can unlock GPU capacity without the need to build an entire AI factory.

The question behind the question

Anyone thinking about Private AI is actually thinking about four questions at once.

• What do I want to do with AI and how intensive will the usage be?

• What data is going into it and how sensitive is it?

• Which model fits that use case, and how much loss in quality is acceptable?

• What infrastructure is required for that, and what are the realistic costs?

Only once those questions are answered does it make sense to discuss platforms, tooling, and architecture. VCF 9, bare-metal GPUs, Chinese open-source models, European APIs: these are all just means to an end. It is about which tool fits which goal.

Fundaments helps you weigh these options. Not based on a preference for a specific platform, but from years of experience with infrastructure for critical workloads and the conviction that architectural choices start with the question, not the technology.