Confidential Inference for Neoclouds: Stopping Weight Theft while Protecting Models and Data from Attackers and Insiders

AI inference protected by cybersecurity measures
Inference is where your AI is most valuable and most exposed. Confidential inference runs models inside hardware-isolated TEEs across CPU and GPU, so neoclouds can prove, not assert, that operators and attackers see only ciphertext.
https://www.anjuna.io/blog/confidential-inference-for-neoclouds-stopping-weight-theft

Inference is where your AI is most valuable and most exposed, because that’s when your model is running on previously unseen datasets to derive new learnings. Adversarial AI has pushed software defenses to their weakest point, and AI-driven zero-days now outrun the patch cycle. Enterprises, AI labs, ISVs and governments are moving AI to neoclouds for cost-efficient compute. For them, compromised infrastructure and model theft now top the risk list, slowing neocloud adoption.

The reason is simple. To run, a model loads its weights, customers’ prompts, keys and data into CPU and GPU memory, on hardware someone else runs. Anyone with admin access to that host sits one hop from your crown jewels: an operator, a contractor, or an attacker holding a stolen credential or an AI-built exploit.

Confidential inference closes that gap. The model runs in the neocloud inside a hardware-isolated trusted execution environment (TEE) that spans CPU and GPU. Weights and data effectively stay encrypted throughout the entire lifecycle, and attackers and even root users have no access to them - not limited access, or gated access - absolutely zero access. Cryptographic attestation proves this is true before models are loaded and decrypted into the TEE, given trusted configurations, keys and data to compute.

The silicon is ready: as of October 2026, NVIDIA supports confidential computing from a single H100 up to a full 72-GPU rack with B200s, B300s and the latest Vera Rubin chipset. The hard part is no longer whether the hardware is available. It is trust: how and who verifies the proof, who holds the keys, and who can pull them back; and managing this across multiple GPUs, nodes and rack-scale infrastructure. That is the problem Anjuna was built to solve. Confidential computing for the world's most valuable emerging companies, enterprises, model providers and AI-native businesses is prime-time ready. We’ve proven this in the world's most demanding enterprises. Now we bring this capability to mass AI adoption in neoclouds.

The risk: whoever controls the host controls your model

Encryption at rest and in transit stops at the moment of use. To serve a token, the stack must hold weights, prompts, outputs and keys in the clear. That is exactly what a privileged insider or a host-level attacker can read.

What is exposed while in use Who can reach it today What it can cost you
Model weights Host root, hypervisor, cloud and neocloud operators, firmware, unauthorized code. Your IP walks out the door; a competitor ships your model, an attack group abuses your AI capability for harm.
Prompts, RAG context, outputs As above, plus anyone or any unauthorized code that taps memory, or data going to storage. With a key pulled from memory, there’s access to sensitive data in encrypted storage. Regulated data breach; customer trust lost, inability to demonstrate to a company or regulators that sufficient controls are in place to withstand contemporary attacks in a resilient manner, which is a requirement for customer acquisition and modern risk frameworks like DORA and NIST AI RMF.
API keys, TLS keys, tokens As above. Lateral movement into every system the model touches, plus access to transmitted data, data systems, AI memory storage.
License terms The party running the hardware. Unmetered copies, unpaid usage, no way to revoke.

‍

The result is a standoff. Model vendors won't hand weights to infrastructure they can't verify. Enterprises won't send sensitive data to a model they can't inspect. Both sides default to "keep it in-house," and the cheapest, fastest GPU capacity goes unused for the workloads that matter most.

This is not hypothetical. Look at Fireworks AI. Fireworks is a hosted inference and fine-tuning platform with tens of thousands of customers. In five weeks it disclosed two incidents, and every step ran through infrastructure, not through the model (The Hacker News, TechCrunch):

  • Oct 6: Attackers using stolen internal credentials read environment variables, including API keys and tokens, inside customer evaluation jobs.
  • Oct 23: A third-party cloud provider hosting Fireworks R&D machines is breached.
  • Oct 24: Attackers pivot into Fireworks development machines and run their own workload, reaching billing data, datasets and incomplete model weights for some customers.
  • Nov 9–11: Fireworks widens its disclosure. Notion confirms Notion AI chat transcripts from fewer than 1% of its workspaces were exposed, and stops serving open-weight models through Fireworks.

Fireworks linked the attackers to a financially motivated group that resells stolen LLM access, a scheme known as LLMjacking. Note the pattern: stolen operator credentials, a compromised infrastructure provider, a rogue workload running beside customer data. One provider's breach became its customers' breach.

In a well-architected confidential inference design, that same intrusion meets ciphertext. Weights and datasets stay encrypted outside an attested TEE. Secrets release only to workloads whose measurements match policy, so a rogue workload fails attestation and gets no keys. The intrusion still happens; the weights, datasets and secrets stay out of reach - the attack is neutralized and meaningless. No regulated data is exposed, and weights are intact. The evidence proves this to regulators and investigators.

AI has turned this from a theoretical risk into an operational one. The time between disclosure and exploitation has collapsed. The Zero Day Clock project measured a mean time-to-exploit of minus 8 hours in July 2026, down from 21.5 days in 2025, and more than 80% of exploits seen in the wild now target zero-days (SecurityBrief). Meanwhile, the median enterprise took 43 days to fully remediate known-exploited vulnerabilities, up from 32 the year before (CSA research note). Anthropic and its Project Glasswing partners found more than 10,000 high- and critical-severity zero-days in their first month of searching (daily.dev summary).

The math is simple - and alarming. Attackers now move in hours; defenders patch in weeks. Patch updates to open source repos inform attackers of a window of opportunity, and exploitation is near instant. You cannot patch your way to safety. You have to make the host irrelevant. If a compromised hypervisor, a rogue admin or a stolen root credential exposes nothing, the zero-day under it stops mattering to your model.

Why neoclouds feel this first

Neoclouds win on price per GPU-hour and on getting the newest silicon first. AI labs, model providers and ISVs want that capacity. But when a buyer weighs a neocloud for a closed model or regulated data, the first questions are rarely about throughput. They ask:

  • Who on your staff has root on the host my model runs on?
  • How mature is your security program, and can you prove it?
  • After a successful attack, what controls are in place to prevent impact to my models?
  • If you are breached tomorrow, are my weights in scope?

A neocloud that answers with policies and SOC reports is asserting trust. A neocloud that answers with hardware attestation, checked by someone other than itself, is proving it. That difference decides which workloads it can win. After Fireworks, expect these questions on the first call, not the last.

Confidential inference turns "trust us" into a product feature. For the neocloud, that means:

  • Winning workloads it can't today. Premium closed models and regulated data that vendors and buyers won't put on shared infrastructure. That demand currently goes to the vendor's own hosted API or stays on-prem.
  • Differentiation beyond price. "Verifiably operator-blind" is something a neocloud can sell, and something a hyperscaler that also sells competing models struggles to claim with the same neutrality.
  • Smaller blast radius. If operators can't see weights or prompts, a breach or a rogue admin becomes a much smaller event, and the compliance story with enterprise customers gets simpler, as do investigations and audits post-incident.
  • No privacy tax. On current-generation GPUs, confidential mode runs close to native speed (details below), so customers no longer trade performance for control.

The same logic holds for the model provider or ISV on the other side. Confidential inference lets it ship weights to any customer's cloud, keep keys and license terms under its own control, and sell to regulated and sovereign buyers who will never call a hosted API.

What confidential inference actually does

Confidential inference protects three things at once: the model in use, the data in processing, and the keys that unlock both. It does it with hardware, and it proves it with cryptography, not paperwork.

The flow, start to finish:

  • Pick the cloud. The model vendor or enterprise chooses their neocloud, co-lo or on-prem site with confidential CPUs and GPUs.
  • Encrypt and measure. The model is encrypted before it leaves the owner, packaged with a confidential VM image, and fingerprinted so any change is detectable. The owner sets policy: region, term, instance count, what the model may connect to.
  • Launch in a TEE. The infrastructure provider boots the image on confidential GPUs. It runs the hardware; it never sees inside.
  • Verify, then release the key. The CPU and GPU each produce signed evidence of their firmware, mode and measurements. An independent verifier checks it against policy. Only then does the owner's key manager release the decryption key, straight into the TEE.
  • Serve inference, blind. Weights decrypt only inside the TEE. Prompts and outputs stay encrypted to everyone outside it, including the operator. The TEE keeps everything sealed yet fully functional.
  • Heartbeat: continue or revoke. The workload re-attests on a schedule. When drift or unauthorized change in firmware, code, policy, location or license terms is detected, the key is pulled, and workloads stop.

No one in that chain has to trust the operator. That is the point.

The hardware is ready. Here is what runs where.

Three GPU generations now support confidential mode, and each one moved the trust boundary outward: from one GPU, to one 8-GPU board, to an entire rack, and multi-nodes. Every combination needs a matching CPU TEE (Intel TDX or AMD SEV-SNP on x86, Arm CCA on Vera), and the CPU and GPU must both be in confidential mode to work.

Platform CPU TEE pairing Confidential boundary How models scale in one TEE Performance cost Tenant complexity (status Nov 2026)
NVIDIA Hopper, single GPU (H100 / H200 PCIe, NVL) Intel TDX (5th Gen Xeon+) or AMD SEV-SNP (EPYC Genoa+) 1 GPU per confidential VM; CPU–GPU traffic encrypted through bounce buffers One GPU, up to 141 GB; mid-size and quantized models Highest of the three generations on transfer-heavy paths; plan roughly 5–15% Low to medium · GA, widely offered
NVIDIA Hopper, 8-GPU HGX (Protected PCIe) Same Whole board to one VM; GPU-to-GPU NVLink not encrypted No safe pooled memory: GPUs run independent model copies, not confidential tensor parallelism Same as above Medium · GA, all-or-nothing allocation
NVIDIA Blackwell HGX (B200, B300) Same 1, 2, 4 or 8 GPUs per VM; NVLink hardware-encrypted Up to 8 GPUs pooled (about 1.4 TB on B200, 2.1 TB on B300); large-model inference with tensor parallelism NVIDIA cites near-identical throughput; plan roughly 1–8% Medium · GA; driver, VBIOS and cooling-SKU specific
NVIDIA RTX PRO 6000 Blackwell Server Same 1 GPU per VM; no NVLink 96 GB; FP4 lets some ~120B-parameter MoE models fit on one GPU Low Low · GA; lowest-cost entry point
NVIDIA GB200 / GB300 NVL72 (Grace + Blackwell) Grace has no Arm CCA, so no confidential VMs Whole rack via Confidential Bare Metal, announced Oct 21, 2026 72 GPUs in one NVLink domain; trillion-parameter serving Not yet independently published High · tenant must build and harden the host OS and full stack; whole-rack tenancy
NVIDIA Vera Rubin NVL72 (Vera + Rubin) Arm CCA on Vera One trust domain across 36 CPUs, 72 GPUs and NVLink; device I/O via PCIe IDE and TDISP Rack-scale, with CVM, container and bare-metal modes NVIDIA cites near-unencrypted performance at rack scale Medium to high · early availability; 100+ components to attest per rack
NVIDIA HGX / DGX Rubin NVL8 Intel Xeon 6 (TDX) 8 GPUs per node 8-GPU pooled inference Not yet published Medium · expected
AMD Instinct (MI300X, MI355X) + EPYC AMD SEV-SNP; SEV-TIO / TDISP on 5th Gen EPYC Building blocks shipping: device attestation, link security, trusted device I/O GPU-level confidential mode still maturing Not yet published High · verify per platform before committing

‍

Sources: NVIDIA Confidential Computing; NVIDIA Rubin platform blog; GMI Cloud on Confidential Bare Metal; AMD SEV; AMD MI355X brief; Helping Secure GPUs That Advance AI; overhead ranges: QSCompute.

What this tells a buyer:

  • Blackwell HGX is today's sweet spot. Encrypted NVLink is the step change: a large model can finally span 8 GPUs without plaintext on the interconnect, at close to native speed. The privacy tax is close to gone.
  • Hopper still works, with limits. It suits single-GPU models. Multi-GPU Hopper cannot pool memory confidentially, so large models either shrink or stay out.
  • Rack scale is here, but the trust surface explodes. One NVL72 rack means verifying dozens of CPUs, 72 GPUs, the switches and their firmware against reference values that the operator controls. Across racks, InfiniBand and Ethernet sit outside NVIDIA's protected domain, so cross-rack traffic needs its own attestation and encryption.
  • No two platforms attest the same way. TDX, SEV-SNP, Arm CCA and NVIDIA GPU evidence each have their own formats, services and reference values. A neocloud running three generations runs three trust stacks.

The hard part is trust, not silicon

A confidential GPU proves nothing on its own. It emits evidence. Someone has to decide whether that evidence is good enough to hand over a model worth hundreds of millions of dollars. That decision is where most confidential inference projects stall.

Trust question Why it is hard What it should be
Who verifies? A neocloud attesting its own hardware is circular. The model vendor can't take the operator's word for it. An independent verifier that neither the operator nor the tenant controls.
What counts as "good"? Evidence spans CPU, GPU, switches, firmware, drivers and the VM image. The operator updates firmware, so reference values move under you. Curated reference values per platform and version, updated as vendors ship, with policy that says which versions are acceptable.
Who holds the keys? If the operator's key service releases the model key, the operator can release it to itself. Keys stay in the owner's KMS or HSM and are released only into a TEE whose evidence matches policy.
Is launch-time proof enough? No. A workload that was clean at boot can be moved, cloned or left running past its license. Continuous attestation: heartbeat, re-check, and revoke on drift in firmware, location, term or instance count.
What does the model touch? A protected model wired to an untrusted RAG store, tool or client can still leak data through its inputs and outputs. Attested configuration for every connection: trusted RAG, trusted clients, controlled inputs and outputs.
How do you operate blind? No SSH, no host-side debugging, no operator access to logs. A failed node must re-attest before it rejoins. Encrypted telemetry and audit trails the owner can read and the operator can't.
What about new attacks on TEEs? Researchers publish new side-channel and firmware attacks on CPU and GPU TEEs every year. Policy that pins minimum security versions and revokes affected platforms the day an advisory lands.

‍

Each of these is solvable. Solving all of them, across TDX, SEV-SNP, Arm CCA and three GPU generations, is a real engineering program. Most neoclouds and model vendors don't want to build it. They want to ship models.

How Anjuna makes it work: prove, don't assert

Anjuna has spent years running unmodified, production workloads inside CPU and GPU TEEs for Fortune 25 financial institutions and the US Department of Defense. Anjuna Confidential Inference packages that experience into a neutral trust layer between the model owner and the infrastructure.

  • The neutral verifier. Anjuna sits between model owner and operator, so the claim "the operator can't see your weights" is checked by someone with no stake in the answer.
  • Encrypt, measure, package. The model is encrypted and measured before it ships. The owner's policy travels with it.
  • Attest, then release. Anjuna verifies CPU and GPU evidence together across Intel TDX, AMD SEV-SNP and NVIDIA confidential GPUs, then releases the key from the owner's own KMS or HSM, straight into the TEE.
  • Enforcement at every heartbeat. License term, instance count, region and usage are checked continuously. Breach a term and the key is revoked.
  • Govern what the model touches. Attested configurations control which RAG stores, tools and clients a model can connect to. Paired with Anjuna Overwatch, the same TEE boundary protects the governance layer for the agents that call the model.
  • One console, every fleet. Build, attest, deploy and monitor across neoclouds, hyperscalers and on-prem from one place, with full visibility across GPU fleets.

What each party gets:

Party What changes
Model vendors and AI labs Ship weights to any customer's cloud without giving them away. Reach regulated and sovereign buyers who will never call a hosted API.
Neoclouds Win closed-model and regulated workloads that won't run on shared infrastructure today. Sell "operator-blind" as proof, not a promise, with nothing to build.
Enterprises Run closed models on sensitive data, on the cheapest capable GPUs, without trusting the operator. Show regulators what ran, and where.

The window is now

Attackers move in hours. Patching takes weeks. The newest GPUs now protect models in use at close to native speed. The only missing piece is the trust that someone other than the operator can verify.

If you run models as a neocloud and customer acquisition is limited due to concern over weight theft or tampering, or your customers are demanding higher levels of security and privacy, get in touch. If you build models, ship them anywhere without giving them away: talk to us about a confidential inference strategy. The time is now.

Contact us at anjuna.io/demo

More like this
Get Started Free with Anjuna Seaglass

Try free for 30 days on AWS, Azure or Google Cloud, and experience the power of intrinsic cloud security.

Start Free