Author: Jeff, IOSG Ventures
Why Private AI is Necessary
On July 1st, Palantir CEO Alex Karp contributed a 20-minute interview on CNBC that some media called a “mental breakdown.” According to Karp, enterprises are paying premium tokens to cutting-edge labs while witnessing their IP flowing to model providers. He referred to this leakage as the transfer of alpha, occurring at the architectural level: each request sent to a closed-source model arrives at the provider's server in plaintext form. Just days before the broadcast, Palantir had announced a partnership with NVIDIA to run an open Nemotron model in client-controlled environments, accompanied by a nine-point AI sovereignty declaration. After the CNBC broadcast, PLTR jumped 8%.
For the past twenty years, companies have adopted cloud software based on trust at the protocol level, and it has worked. Each SaaS provider only sees slices of enterprise data, and most have little incentive to feed customer data back into their core products. Salesforce sees sales channels, Workday sees personnel, Jira sees development iterations, and AWS provides a foundation for storage and computing. However, today’s AI workflow advocates for the one-time upload of all assets, along with the structured context connecting various departments, to maximize productivity. Setting goodwill aside, upstream providers can now use this data to develop new features instead of letting it sit on servers gathering dust.
No one is slowing down; Anthropic's annual revenue reached $47 billion in May, a significant leap from the $9 billion at the end of 2025, while OpenAI surpassed 900 million weekly active users in February. Both companies completed new rounds of funding this spring, with valuations approaching $1 trillion, and are expected to go public with higher valuations. Years of privacy and IP allegations have not caused either company to lose momentum.
Some enterprises have already taken action. In February 2023, within three months of ChatGPT's release, major banks on Wall Street restricted its use. In May 2023, after Samsung engineers leaked chip source code into ChatGPT, the company completely banned generative AI. In response, OpenAI launched ChatGPT Enterprise in August of that year, promising not to use commercial data for training, along with a zero-data-retention (ZDR) protocol, which has since become a standard requirement for enterprise procurement.
But contracts only lock down company accounts. IBM found that by 2025, shadow AI (employees feeding company data into unapproved AI tools through personal accounts) will be involved in one-fifth of data breach events, with heavy shadow AI usage adding an average of $670,000 to breach costs. In a 2025 survey by security training company Anagram, four employees indicated they would be willing to violate AI usage policies to complete tasks faster.
Companies at least have the option to pay for an exit, ZDR contracts, non-training service tiers, and if you are a government or Palantir client, sovereign deployment options. However, for ordinary users like you and me, there remains a debate on whether privacy AI is important, until a court subpoena arrives at the door.
In May 2025, a court order forced OpenAI to retain even deleted consumer chat logs, and in November, a judge ordered the transfer of 20 million of these logs to the lawyers of The New York Times as evidence disclosure material. Then there are criminal cases: ChatGPT records entered evidence in the Palisades arson case, and a sworn statement in a Florida double homicide referenced a suspect's questions on how to dispose of a body. Sam Altman also admitted in a July 2025 interview that ChatGPT conversations are not protected by legal privilege, and in litigation, OpenAI “may be required to hand over” user chat records.
The point is that it’s not just criminals who need private conversations. Conversations between people and AI being recorded and subject to subpoenas presents a facet of surveillance that many users are unaware of. A survey by Kolmogorov Law in October 2025 of 1,000 American AI users found that 50% did not know these conversations could be subpoenaed, while two-thirds believed that such chats should receive the same protection as those with lawyers or doctors.
Self-hosting or running verifiable open-source models is rapidly catching up, but the strongest batch still lags the cutting-edge closed-source models by about four months in general capability. This puts enterprises and individuals engaged in tokenmaxxing at a crossroads: sacrifice a few months of model quality for this privacy, or continue uploading sensitive materials to Anthropic's servers, as competitors are leveraging this for productivity advantages.
Currently, there is no perfect solution on the market, and the report outlines various attempts by different parties to bridge the gap, observing how far cutting-edge intelligence verifiable privacy is from being delivered to enterprises and ordinary users.
How is Privacy Currently Achieved?
Privacy AI is not a singular engineering feat, but each mechanism in the current market addresses the same event: a prompt leaves your device, traverses the network, lands on the machine running the model, and then returns a reply. The differences between mechanisms lie in where plaintext exists along this path, who can read it, and what is used to verify the confidentiality of the reply.
Protocol-Level Privacy
At this level, besides you, others may read your plaintext prompt, and what happens next relies solely on a promise.
Contractual Zero-Retention is the enterprise version solution. The service provider knows who you are, processes your prompt, and promises not to retain it, enforced by contract and reputation.
Anonymous Proxies erase who you are but do not encrypt what you said. Downstream service providers process the plaintext according to their own policies. Terms vary by company; for example, Duck.ai (the chatbot product of DuckDuckGo) might negotiate deletion agreements with the model provider, while Venice directly lets users assume that the service provider will retain everything, but neither side can verify.
Every segment between machines runs on TLS, which only encrypts the pipeline, allowing the receiving party to read all information. Relays typically use Oblivious HTTP (RFC 9458) to separate this right to knowledge, functioning like a friend passing a note. The friend knows who passed it but cannot read the content; the recipient can read the content but does not know who wrote it. OHTTP became an IETF standard in January 2024, and many companies now run production traffic on OHTTP relays rented from Cloudflare and Fastly.
This is also the upper limit of privacy available when accessing closed-source models, due to a simple arithmetic problem. The cost of flagship-level training today is in the billions, and these labs’ valuations nearing trillions are betting on the exclusivity of model weights. The gap in model capabilities persists as long as the premium does, thus labs guard weight files as state secrets.
Meta has passively conducted this experiment. The LLaMA released in February 2023 was initially only open to researchers, but within a week, its weights leaked in seed form on 4chan. A week later, llama.cpp enabled even the smallest 7B model to respond locally on a MacBook, and three days later, Stanford managed to fine-tune a chat assistant, Alpaca, on the same model with less than $600. This leak reduced LLaMA's operational costs to electricity bills; anyone with the file could run it at home. In July 2023, Meta officially open-sourced LLaMA 2 with a commercial license excluding a clause on 700 million monthly active users. The weight ran, and the premium ran with it.
Cutting-edge labs theoretically could provide attestation (remote proof) for the inference of closed-source models, but attestation can only prove which segment of code read the prompt; it cannot prove what that code did with it. To determine if the server retained data, we need to audit the serving code and reconstruct it to the hash reported by the hardware. However, once the serving code is handed over, the labs also relinquish the batch processing and caching techniques that support their profit margins, techniques that will migrate to every future generation of models. Apple and Meta can provide remote attestation for the service stacks behind iPhone and WhatsApp because their profits derive from devices and advertisements, making the disclosure of service code nearly costless.
This is why the weights and service code of flagship models do not reach external operators. Without external operators, there is no third-party attestation; without attestation, verifiable privacy exists only on open-source models.
Structural-Level Privacy
Every mechanism in this category replaces trust commitments with evidence based on hardware, cryptography, or physics, but each must bear different costs for privacy upgrades, primarily that they can only run open-source models.
TEE (Trusted Execution Environment) Confidential Computing runs inference inside a hardware enclave (a sealed chamber on a chip that even machine operators cannot open), where the chip will sign an attestation detailing which model and which code were run.
Prompt is only sealed at the endpoint. A role capable of reading plaintext still resides along the pathway routed through proxies, and only the protocol prevents proxies from recording or leaking the transferred content.
E2EE (End-to-End Encryption) eliminates readable relays. The user device encrypts the prompt with the enclave’s key, with each hop carrying only a sealed envelope that only the enclave can unpack.
Trust lies on the client side. The code responsible for encrypting the prompt and verifying the attestation also has the ability to revoke this guarantee. Therefore, verifiable E2EE requires both proven enclaves as well as open and reproducible client-side code.
Compared to the simplicity of TEE, E2EE’s cost is the engineering burden, which slows down feature integration. E2EE turns proxies into blind messengers, making all functions relying on reading plaintext need to be rebuilt around client keys or only rebuilt within the enclave.
FHE (Fully Homomorphic Encryption, and its MPC variants) simply eliminate trusted parties. The server performs calculations on ciphertext inside a locked box it can never open, with the key remaining solely in your hands; MPC (Multi-Party Computation) splits the prompt into secret shares distributed among multiple parties, yielding similar effects unless all participants conspire.
The cost is speed. FHE inherently only performs addition and multiplication, so all non-linear steps required for transformers must be reconstructed at a high cost. Inference costs on ciphertext are 10,000 to 100,000 times that of plaintext; each token on small models can take seconds to minutes, while in unencrypted scenarios it takes only milliseconds.
Chips customized for encrypted computation are expected to narrow the gap, but the first prototype won’t complete demos until early 2026, and commercial versions will take several more years.
Local Inference outright removes this path. The model runs on your own hardware, with no proxies, no servers, no service providers, and no verification requirements.
The obvious cost is the expense and model capabilities. gpt-oss-120b scores about half of GLM-5.2 on the Artificial Analysis index, but weighs 65GB, exceeding the combined VRAM of two flagship gaming graphics cards on the market. Meanwhile, the full precision GLM-5.2 only runs on an 8-card data center node, requiring over $300,000 just for GPUs.
Nevertheless, beyond these structural limitations, the cost of placing inference in enclaves is decreasing. For single-card inference, benchmarks from enclave cloud service provider Phala show that enclave mode H100 throughput loss averages less than 7%, with near-zero loss on large models, primarily because the main cost lies in moving data into the chip, not in performing calculations within it. For multi-card inference, NVIDIA's next-generation GPU Blackwell now supports direct encryption of inter-chip traffic, while the older H100 can only achieve the same effect by routing through a CPU host at one-seventh of the bandwidth. NVIDIA’s own benchmarks on Blackwell show that a 397B model experiences less than 8% throughput loss in enclave mode. With these advancements, the performance penalties of privacy inference are no longer a decisive constraint.
In fact, enclaves themselves impose almost no additional operational costs on providers. Every H100 after 2023 comes with enclave mode built-in, with additional costs stemming from encryption's throughput loss rather than from extra chips. Currently, the rental price for confidential H100 SKUs on Azure is still $8.90 per hour, while without enclaves it is $6.98, representing a 27% markup compared to traditional cloud facilities. On the other hand, operators like Phala that specifically offer enclaves rent out confidential mode H100 starting at $3.80 per hour, lower than the price range of $3.99 to $4.29 for Lambda's ordinary SXM cards. For managed API solutions, NEAR AI offers endpoints with attestation priced at $0.15 per million tokens for input and $0.55 for output for gpt-oss-120b, matching Amazon Bedrock, Together, and Groq on plaintext routes. Even for models requiring multi-chip parallelism, NEAR AI prices are identical to Fireworks on GLM 5.2 and are 15% cheaper for inputs and 4% cheaper for outputs on larger Kimi K2.6.
Although these new privacy inference service providers may be burning profits to capture market share (this applies to any growth-seeking company in the market), the structural direction indicates that the cost of privacy is declining for both consumers and operators.
How do Open-Source Models Win?
Despite the compression of performance overhead, a visible gap still exists between cutting-edge models and SOTA open-source models, and an entity seeking to maximize productivity must still trust frontier labs not to steal its IP.
The gap persists, but AIA Labs under Bridgewater and Thinking Machines provided a case on June 30th: an open model fine-tuned with expert annotations outperformed cutting-edge models in both accuracy and cost.
In the study, the team fine-tuned Qwen3-235B using Tinker (Thinking Machines' hosted fine-tuning API service). They first purchased annotated data from vendors to train the first round, then passed the disagreement samples back to the company's investment personnel for re-annotation. Training utilized reinforcement learning (GRPO) and included three modifications: round-robin batching (each task takes turns producing a batch), CISPO loss (limiting how far a single answer can pull the model), and on-policy distillation (anchoring to the current best checkpoint to ensure the model does not learn from weaker replicas).
All tasks derived from the regular workflows of investment personnel: whether a news piece was significant to C-suite investment professionals, if a central bank document suggested future interest rate movements, or where template phrases in a document or email began. Scoring came from an independent testing set, where the cutting-edge models scored around 50% on simple prompts, and with expert prompts only reached 78.2%, below the 80% threshold set by investment personnel. In contrast, the fine-tuned Qwen achieved 84.7%, representing a reduction of 29.8% in errors compared to frontier optimal performance, while inference costs were 13.8 times lower.
https://thinkingmachines.ai/news/learning-to-replicate-expert-judgment-in-financial-tasks/
This case demonstrates that open-source models can win in both accuracy and cost, but the training process is still not private. The expert annotations used in the process are proprietary data from Bridgewater, passing through a third-party service Tinker, falling under the same trust tier as the ZDR protocol. The fund also rented computational power, with the entire training run occurring on machines it never controlled. Buyers wanting this formula without the burden of trust assumptions have very limited options today. Renting bare GPU clusters means the training process is readable by cloud operators. Purchasing a cluster resolves data holding concerns, but costs can skyrocket.
The route with attestation has just begun. In March, Workshop Labs and Tinfoil released Silo, a post-training stack running within the Tinfoil enclave on a single 8-card node, with keys solely controlled by the client. The article indicated that the attestation cost for the enclave involved an additional 11 minutes for two hours of training, and this stack could accommodate a trillion-parameter model (Kimi K2 Thinking) by freezing the base weights and only training a small adapter on top. The challenge lies in that reinforcement learning requires passing data back and forth among components, and moving data is precisely where enclave costs arise.
Less than a month after Silo's release, Workshop Labs was acquired by Thinking Machines, consolidating the parts needed to run the next Bridgewater-style RL loop within the enclave.
Privacy at the Harness Layer
There is also an issue that lies beyond all private inference mechanisms. These mechanisms manage the path from prompt to model, while every external tool invocation initiated by the agent opens a path that inference layers cannot reach. The recent trend in harness engineering amplifies this problem; every tool, memory, and data source surrounding the model represents another destination capable of reading its own workflow slice in plaintext. Calendar servers read schedules; database servers read queries. A fully local agent still needs to submit search terms in plaintext to a search engine for anything outside the training set, as the server cannot read plaintext and therefore cannot answer questions.
Mainstream solutions still default to the protocol layer. Companies like Runlayer and MintMCP control all tool traffic through a central gateway, obscuring personal identifiable information (PII) before requests exit. The gateway also decides which servers can receive traffic, keeping unvetted ones out, and logs each call’s destination and content for potential evidence. Even with independent audits (SOC 2), tool servers still need to read plaintext queries to respond, whether they retain copies depends on their retention terms, additionally multiplied by every tool in the harness. Furthermore, the gateway itself is an added party that relies on trust to read, rather than verify.
Structural-level solutions address the middle layer. For example, Phala directly hosts the MCP server within TEE, covering directories, wallets, code executions, and data sources. Users can verify privacy statements with one attestation rather than trusting operators. However, tools hosted in TEE inevitably have to pass queries in plaintext to service providers—the enclaves protect only the messenger, not the destination.
Only a few destinations have learned to answer without reading, but that applies only to structured queries. Apple provides private information retrieval for the iPhone, allowing caller numbers to check against spam lists without exposing the number; Microsoft employs the same approach for passwords in the Edge browser. MongoDB's Queryable Encryption allows clients to encrypt fields before they leave, enabling servers to perform equivalent and range matching based solely on the ciphertext.
But for open-ended searches, today’s best answers stop at trust—verifiable encrypted search has yet to emerge from the lab. Brave promises zero data retention on its own index of 40 billion pages (rather than Google’s), yet it still falls into the protocol layer. Exa constructed a neural index that embeds user keywords into semantics, ordering results based on semantic matches, but this embedding step still starts from plaintext on Exa's servers. MIT’s 2023 Tiptoe paper achieved ordering across 360 million webpages without exposing queries, but each search consumes significant server computation power, and ordering quality lags behind unencrypted searches. Apple's 2024 Wally paper reduced communication costs by up to 31 times by hiding real queries among decoys; however, this mathematical approach only becomes inexpensive at millions of concurrent queries, a scale that no private search system currently holds.
Encrypted search is feasible; it just hasn't reached a commercially viable performance and price level.
Outlook
The demand for private AI is rising. Venice AI recently surpassed 3.5 million registered users and a throughput of 1.3 trillion tokens per month, subsequently completing a new round of $1 billion Series A funding. Proton is its direct competitor, with its chat product Lumo reaching over 10 million users within a year of launch. In terms of infrastructure, Phala currently runs 2 to 3 billion tokens daily on OpenRouter. Duck.ai routes gpt-oss-120b and Gemma into Tinfoil's enclave, providing verifiable privacy beyond user proxies. This doesn't even take self-hosting into account; it may very well be the largest channel for private inference, after all, models run on one's own hardware, leaving no usage traces.
However, within the mainstream AI trend, privacy AI occupies merely a small part, and this gap will only close when frontier labs intentionally fulfill this need. In May, Google processed 32 trillion tokens across its entire product line; based on that calculation, Venice’s monthly throughput is roughly equal to Google’s in 18 minutes. Last November, Google launched Private AI Compute (PAC), placing some Gemini-driven functions into sealed TPU enclaves isolated from the company itself, with design independently audited by NCC Group. However, the issue is that PAC only covers a few Pixel functions like personalized recommendations and audio summaries, not hundreds of millions using Gemini applications. Google dares to hand the design over to auditors because these features monetize through devices and advertisements, not by selling tokens.
Current managed solutions are not perfect. Users looking to achieve maximum privacy through E2EE must wait for new features to be rebuilt in places unreadable by service providers. Private harnesses at the service layer still rely on protocols. Reasonably priced post-training outputs still require reliance on third-party suppliers for the best fine-tuning results. Self-hosting eliminates all service providers, but running the strongest open-source models locally may cost more than the building housing it.
Deficiencies aside, private AI is already a real and affordable option, and the remaining gaps are narrowing. For ordinary consumers, on Lumo and Venice, open model private chats under no-logs promises are free; subscriptions to Venice or Tinfoil costing $18 to $20 enclose the same chat within enclaves, and are no more expensive than a ChatGPT subscription. For enterprise workflows, endpoints with attestation are often cheaper than plaintext routes. Endpoints like NEAR's E2EE API can now bring encrypted context into enclaves, allowing memory, file uploads, and custom instructions to operate atop E2EE today. As for post-training with attestation, NVIDIA's upcoming Vera Rubin NVL72 will extend confidential computing from Blackwell’s 8-card nodes to 72-card racks, making cutting-edge RL loops more feasible without exposing IP.
However, key value captures lie beyond these levels of price compression. Privacy is nearly free where it already exists but has yet to cover mainstream agentic workflows. Operators focused on rental enclave sales hold a switch on standard chips, not a moat, whereas gateway protocols compete alongside traditional middleware. The defensible ground lies in that half of this report that has yet to be solved: training loops contained in enclaves, end-to-end sealed tool calls, and invisible index searches. Whoever first accomplishes one of these tasks will sell something that can’t be commodified through any price war. Capital chasing privacy AI should invest in the gaps, not the switch.
So, trust or verify? For tasks heavy on execution and agents, choose trust, because each tool invocation naturally delivers plaintext to destinations that enclaves cannot seal, while frontier models are priceworthy in such loops. As for high-level thinking that distinguishes a company from its competitors, choose verification. Strategies, planning, and judgments distilled from years of professional experience constitute precisely that alpha in contention. The way forward lies within the boundaries of company control, micro-tuning open-source models with these proprietary insights. In areas where a company’s alpha resides, finely tuned open models have already simultaneously outperformed cutting-edge models in accuracy and cost, and the infrastructure to build them in privacy environments is arriving one node at a time.
免责声明：本文章仅代表作者个人观点，不代表本平台的立场和观点。本文章仅供信息分享，不构成对任何人的任何投资建议。用户与作者之间的任何争议，与本平台无关。如网页中刊载的文章或图片涉及侵权，请提供相关的权利证明和身份证明发送邮件到support@aicoin.com，本平台相关工作人员将会进行核查。
