Project Lily: The Case That Just Made Verifiable AI a Trillion-Dollar Thesis

Project Lily never leaked a single byte of training data. It leaked something worse — the assumption that centralized AI could be trusted at all.

A California class action landed on OpenAI's doorstep this year, and the substance of it is almost comically mundane. The company, according to 404 Media's reporting, outsourced a slice of its human review pipeline — the unglamorous labor that makes ChatGPT sound less like a sycophantic autocomplete — to contractors who read real user conversations. The internal codename was Project Lily. Flowers. Soft. Innocuous. The kind of name you give a project you already know is going to be a PR problem.

I have been in this industry for twenty-one years, and I have watched the same movie four times now. A centralized operator makes a reasonable engineering decision, bolts on a trust assumption nobody can audit, and then acts surprised when the trust assumption is the thing that breaks. In 2017 it was ICO treasuries with no multisig. In 2020 it was DeFi protocols with reentrancy holes. In 2022 it was bridge validators with a single key. In 2024, it is a chatbot that reads your messages through a human being you were never told about.

The pattern is identical. The only variable is the asset class.

The Setup: What Project Lily Actually Was

Let me strip the legal language away and describe what happened in engineering terms, because the engineering terms are what matter.

OpenAI runs a training pipeline. The final stage of that pipeline — the stage that turns a raw autoregressive model into something that behaves like a helpful assistant — is called RLHF: reinforcement learning from human feedback. I will explain the mechanics in detail shortly, but the one-line version is this: you cannot produce a well-aligned language model without human beings making judgment calls about which of two (or four, in OpenAI's case) model responses is better. Those judgment calls get compiled into a reward model. The reward model shapes the final policy.

Project Lily was the human layer of that pipeline. Reviewers — recruited through a third-party staffing agency, which is a legally load-bearing detail I will return to — read live user conversations, summarized user intent, scored four model responses on a one-to-seven scale, and left written commentary. They also, per the complaint, saw what the filing describes as user memory summaries: condensed cross-session profiles that OpenAI began generating as part of its Memory feature rollout.

That is the entire scandal, from a technological standpoint. Human beings read your chats. You did not know. The chats were supposed to be anonymous, but they were not anonymized in any sense a cryptographer would recognize.

Now — here is where every lazy take on this story goes wrong. The lazy take says OpenAI was reckless, or greedy, or sloppy. The lazy take treats Project Lily as an execution failure. Project Lily was not an execution failure. It was a faithful implementation of the only architecture centralized AI currently knows how to build.

The company did what the method requires. The method is what is broken.

Core Analysis: The Structural Privacy Conflict Nobody Wants to Name

RLHF's Original Sin

Let me put on my cryptography hat for a moment, because the precise nature of the problem is being deliberately blurred in every mainstream writeup I have read.

RLHF, as formalized in the 2017–2020 literature that OpenAI itself pioneered, has three stages. First, supervised fine-tuning on demonstration data — humans write good answers, the model imitates. Second, reward modeling — humans rank pairs of model outputs, a separate network learns to predict those rankings. Third, policy optimization, usually via PPO, where the model is trained to maximize the reward model's score.

Stage two is the problem. Not stage two as implemented by OpenAI in 2024. Stage two as a concept. Stage two requires a distribution of human preference judgments over real, diverse, contextually rich prompts. And real, diverse, contextually rich prompts are, definitionally, prompts that real humans typed into a real product at a real moment of need. A prompt is not a synthetic benchmark. A prompt is the residue of a person.

You cannot get high-quality preference data from imagining what users would have asked. The entire point of RLHF is that it captures the tails of the distribution — the weird prompts, the edge cases, the culturally specific questions, the ones a research team in San Francisco would never have thought to write. The method's power and the method's privacy incursion are the same property. You cannot keep one and discard the other. That is not a policy choice. That is a mathematical identity.

So when OpenAI's public response — the one reported alongside the 404 Media piece — says the review work was essential to safety and alignment, that is not corporate spin. It is, narrowly, true. The problem is that the same sentence is also an indictment. If the work is essential and the work requires reading real conversations, then the privacy incursion is essential. The architecture has no room for a version of the product that is both well-aligned and private.

Hold that thought. I will come back to it. It is the hinge on which the entire decentralized AI thesis swings.

Anonymization Is Not a Law. It Is a Belief System.

The complaint's most embarrassing revelation, from a technical standpoint, is the memory summary issue. OpenAI's defenders will reach for the anonymization defense: sure, reviewers saw your conversations, but they never saw your name. The email address was stripped. The account ID was hashed. PII was filtered by automated systems before any human eyes touched the text.

I want to explain why that defense collapses, because I have been on both sides of this exact problem. In 2020, during the DeFi Summer, I spent three weeks as a part-time security advisor on an AMM called AeroSwap. My job was to stress-test the bonding curve against flash loan attacks. I found a reentrancy vulnerability in the liquidity withdrawal function — a classic, the kind of bug that lets an attacker re-enter the withdrawal call before the state update finalizes and drain the pool. We patched it three days before mainnet and saved about $15 million in TVL.

Here is the relevant part. The bug was not visible in the happy path. The function looked fine if you read it top to bottom and assumed callers behave. The vulnerability lived in the composition — in what happened when the function was called from inside itself, in a context the author never imagined. That is how almost all serious security failures work. Not at the level of the individual component. At the level of the context the component is dropped into.

Anonymization has the same pathology. A single conversation with the name stripped out is, genuinely, hard to attribute. But RLHF data is not consumed as isolated conversations. It is consumed as panels — as batches of related prompts scored together, alongside memory summaries, alongside timestamps, alongside topic clusters. And contextual inference is a re-identification primitive. Give me a conversation in which someone mentions they work in crypto, their employer's name (John), their time zone (implied by writing hours), and their recurring anxiety about a specific regulatory question, and I will find that person in a database of five million. I do not need their email. I need their fingerprints, and conversation style is a fingerprint.

This is why cryptographers distinguish pseudonymization from anonymization, and why the distinction is not academic hair-splitting. Pseudonymization replaces a direct identifier with a surrogate. It is reversible given auxiliary information, and auxiliary information is everywhere. Anonymization, in the strict sense, means the data cannot be linked back to an individual even with unlimited auxiliary information. True anonymization is extremely hard. For rich free-text natural language, it is arguably impossible with current techniques, because the text itself is the auxiliary information.

So when the automated PII filter ran, what did it actually catch? Explicit entities. Email addresses, phone numbers, social security numbers, credit card patterns — the things a regex or a named-entity recognizer reliably flags. What it did not catch, and structurally cannot catch, is implicit PII. The mention of a boss named John at a crypto company. The description of a medical symptom specific enough to identify. The complaint of a legal problem in a small jurisdiction. Every one of those sails straight through the filter, because a filter that flagged them would flag half of all useful prompts.

OpenAI's own statement, per the reporting, concedes that automated filters cannot intercept everything. Read that sentence again. It is not a bug report. It is an architectural confession.

The Consent Void

Everything above would be survivable if the consent story were clean. It is not. Per the filing, users were never clearly told that a human being might read their conversations. Not in the signup flow, not in the product UI, not in a prominent banner. The privacy policy used broad language about service improvement, which is the phrase every company reaches for when it wants to reserve maximum latitude, and which no court has ever read as an explicit disclosure of human review.

This is the weakest link in OpenAI's legal armor, and everyone involved knows it. Under California's CCPA and CPRA, and under common-law intrusion-on-seclusion theory, the presence of a robust, conspicuous, revocable consent mechanism is the difference between a defensible practice and an indefensible one. It is not a matter of whether the review was harmful in any concrete sense. It is a matter of whether the user had a reasonable expectation of privacy that the company failed to rebut.

I have spent the last year working at the seam between institutional finance and crypto — I partnered with a Swiss private bank to design decentralized custody for ETF-linked tokens, translating institutional risk requirements into smart contract logic. Let me tell you what that experience taught me about consent. Institutional compliance is not about doing the right thing. It is about producing an auditable record that you did the right thing. The record is the product. A bank that reads your data but documents its disclosure is legally safer than a startup that respects your data but writes a sloppy policy.

Project Lily had the technology and, apparently, not the record. That asymmetry is the whole case.

The Outsourcing Chain Is a Liability Laundering Machine

The staffing structure matters more than the reporting suggests. OpenAI did not hire these reviewers. It hired a staffing agency, which hired contractors, who may themselves be employed by a subcontractor. That is a four-layer chain, and every layer exists to do one thing: distribute responsibility so thinly that no single party can be held fully accountable.

I understand the incentive. I also understand why it backfires. From a privacy-litigation standpoint, a multi-layer outsourcing chain is not a shield. It is evidence of sophistication. Plaintiffs' counsel will argue that the structure was designed to evade direct-employment obligations and to create plausible deniability about data handling. Whether that argument wins is a jury question. But the optics are catastrophic, because the chain produces a second scandal on top of the first: the labor conditions.

Reported hourly rates for this class of work, across the industry, have been as low as $1.32 to $2 per hour in offshore review operations. I want to be precise: that figure comes from disclosures about comparable operations at other data-labeling vendors, not necessarily Project Lily specifically. But the economics of the sector are uniform enough that no one should pretend the number is anomalous. You are reading deeply intimate conversations on a wage that cannot support a life, with no clinical supervision, no mental-health support, and no meaningful ability to refuse a disturbing prompt.

That is the pipeline. That is the machine that produced the assistant you are talking to right now. I am not saying this to be provocative. I am saying it because the abstraction has to be punctured before any solution can be designed.

Why the Filters Cannot Be Fixed (And Why That's the Point)

Here is where I part company with the optimistic engineers. There is a widespread belief in the industry that Project Lily is a solvable problem — that better classifiers, better entity recognition, better differential-privacy noise injection, will eventually thread the needle between RLHF's data appetite and the user's reasonable expectation of privacy.

I do not believe that, and I want to explain why with the same rigor I would apply to a protocol audit.

Consider the security model of an RLHF data pipeline. The asset being protected is the linkability of a conversation to an identity. The adversary — in this case, a curious contractor or a motivated attacker with database access — has a set of capabilities: they see batches of conversations, memory summaries, and any metadata the pipeline retains. The defender applies anonymization and filtering.

The defender's tools are fundamentally lossy. Every filter that removes identifying information also removes signal, and RLHF data's entire value proposition is signal density. Tighten the filter, and you degrade the reward model. Loosen it, and you leak identity. There is no setting of the dial that satisfies both constraints, because the constraints are in direct tension.

This is not a temporary engineering immaturity. It is the same wall that has stopped traditional data anonymization research for two decades. The only methods that even theoretically satisfy strict anonymization guarantees for richly structured data — differential privacy with meaningful epsilon budgets, secure multiparty computation, fully homomorphic encryption — all impose costs that RLHF pipelines have historically been unwilling to pay. Differential privacy training can degrade model quality by five to twenty percent depending on the parameters. That is a catastrophic hit in a market where a one-point benchmark difference is a headline.

The uncomfortable conclusion: the centralized RLHF pipeline is not fixable at the margin. It is fixable only by changing what the pipeline is.

The Class Action Mechanics — and Why the Number Doesn't Matter

Let me talk about the litigation structure, because serious money is being misallocated here.

A California class action of this shape typically aggregates hundreds of thousands to millions of class members. Statutory damages under California privacy law can run from roughly $100 to $750 per consumer per incident. If you naively multiply, you get numbers in the hundreds of millions to low billions. That arithmetic is how plaintiff-side press releases generate headlines.

It is also almost entirely fictional. Real settlements land one or two orders of magnitude below the theoretical maximum. The Meta facial-recognition settlement, often cited in this context, was $650 million for a class of roughly 1.6 million people — about $400 per head in a case where the statutory ceiling was $5,000. That is the realistic texture of these resolutions. For Project Lily, the affected user base is a subset of paid users whose data actually touched the review pipeline, which is a far smaller population than the full ChatGPT user base. Reasonable settlement estimate: tens of millions, not billions.

So the direct financial hit is a rounding error against OpenAI's reported ~$157 billion valuation and its reported ~$5 billion annual burn. The company could write a check tomorrow and not notice.

The number is not the story. The number is never the story.

Three things matter more than the check. First, discovery. The class action opens a document-production process in which OpenAI's internal privacy practices, its handling of the memory-summary feature, and its vetting of the labeling vendors all become discoverable. That is the real risk. Not the damages — the disclosure.

Second, precedent. If this case establishes that human review of user conversations requires explicit, conspicuous, revocable consent, then every other frontier lab — Anthropic, Google, Meta, and every Chinese lab selling into regulated markets — inherits the same obligation. RLHF is not an OpenAI-specific method. It is the whole field's method. The industry has a shared single point of failure, and the lawsuit just put a spotlight on it.

Third, and most underappreciated: insurance and enterprise procurement. Large tech firms carry cyber-liability and D&O insurance that can absorb legal costs and settlements, unless the conduct is deemed willful, in which case coverage is voided. Meanwhile, enterprise buyers in finance, healthcare, and law have data-processing agreements that require transparency about subprocessors and human access. Project Lily, the moment it became public, became a line item in every enterprise DPA negotiation for the next eighteen months. That is the cost that does not appear in the settlement number, and it is the one that will persistently raise OpenAI's cost of doing business.

The Tension Nobody on the Crypto Side Wants to Admit

Now I have to do the thing I hate doing, which is turn the critique inward.

There is a reflexive reaction in the decentralization community to a story like Project Lily: see, this is why you need us. And the reflexive reaction is, in its broad strokes, correct. But it is being deployed with a smugness that the actual engineering does not yet support, and I want to be the guy who says so, because I have watched our side over-promise before.

Here is the honest state of the art.

Verifiable inference — proving that a model produced a given output from given inputs — is genuinely advancing. Zero-knowledge machine learning has moved from toy circuits to real deployments on smaller models, and trusted-execution-environment attestation is shipping today for inference on mid-sized models. But for a frontier model with hundreds of billions of parameters, proving inference in real time is still years away. The circuits are too large, the proving systems too slow, the cost structure not yet within a factor of ten of usable.

Federated learning — training across users' devices without centralizing the raw data — works beautifully for keyboard prediction and mostly fails for frontier-scale alignment, because the data heterogeneity across devices is brutal and the communication costs are enormous.

Decentralized data markets — compensating users for their data with cryptographic provenance — are the most promising structural fix, and also the one with the worst track record so far, because the compensation is always too small to matter and the provenance guarantees are always weaker than advertised.

I say all this not to undermine the thesis. I say it because I have been on the wrong side of an over-promised timeline, and I do not want to do it again. In 2017, at the peak of the ICO mania, I raised $4.2 million in forty-eight hours for a hybrid proof-of-work, proof-of-stake consensus layer called ZurichChain. I had a cryptography PhD and zero product experience, and I sold a story about decentralized sovereignty to retail investors who were tired of being excluded by banks. It worked. And then the market found out that a story is not a product, and the token did what tokens do.

I learned something in that sprint that I have carried ever since: narrative momentum is real capital, and it can be spent like real capital, but it cannot be redeemed like real capital. The whole crypto industry is sitting on a pile of spent narrative. Project Lily is a rare opportunity to redeem some of it by shipping something real. If we respond with slogans, we deserve the eye-roll we will get.

The Contrarian Angle: Decentralization Advocates Are Asking the Wrong Question

Here is where I want to push against my own tribe, hard, because the standard decentralization response to Project Lily is both obvious and strategically weak.

The standard response: centralized AI is a privacy catastrophe, therefore build decentralized AI, therefore users will migrate, therefore value accrues to decentralized infrastructure.

Every clause in that sentence is a hope, not a deduction. And the chain is only as strong as its weakest link, which is the third clause.

Consider the actual user. The user who is upset about Project Lily is upset because their conversations were read by a human being they did not authorize. Now ask: does that user, in that moment of upset, have any interest in running their own model, paying gas fees for inference, managing a wallet, or staking tokens to participate in a training collective? Almost certainly not. What that user wants is a product that behaves exactly like ChatGPT but does not read their chats. They want the convenience of centralization and the privacy guarantees of cryptography. They want to have it both ways.

And here is the uncomfortable truth I have been circling: most "decentralized AI" products have been designed for the crypto-native user, who is a rounding error of the market, and not for the privacy-anxious mainstream user, who is the actual prize. We keep building for the wrong customer and then blaming the customer for not showing up.

The real contrarian claim, the one that will annoy the true believers: the winning architecture is probably not "decentralized AI." It is hybrid AI with cryptographic verifiability layered onto otherwise centralized infrastructure. Trusted execution environments that attest to what happened to your data. Zero-knowledge proofs of policy compliance rather than of full inference. Federated learning where it works, plus auditable data-handling commitments where it does not. The decentralized movement's most valuable contribution to this problem may not be decentralization at all. It may be the tooling of verifiability — the same discipline that gave us trust-minimized bridges — applied to pipelines that remain operationally centralized.

I will go further. The most likely near-term winner in AI privacy is not a token. It is a compliance-and-attestation layer, sold to enterprises, that lets a bank or a hospital verify with cryptographic evidence that its AI vendor handles data as promised. That is an application of cryptography to institutional trust, which is precisely the terrain where I spent the last year of my career, and precisely the terrain where I have watched crypto-native teams consistently underinvest because it is not sexy.

There is a second blind spot I want to name, and it is a direct inheritance from the cross-chain wars. I spent 2022 at an interoperability lab, and I led a hackathon where we built cross-chain bridges in seventy-two hours and then wrote a report about how badly they broke. The report, titled with the phrase I still stand behind — the illusion of seamless interoperability — made one argument that applies directly here. Elegance in the protocol layer is worthless if the application layer fragments, and fragmentation is the default, not the exception.

The decentralized AI ecosystem is repeating this history in real time. There are dozens of compute networks, dozens of data-labeling DAOs, dozens of training-coordination protocols, and essentially none of them compose into a coherent pipeline a normal product team could adopt. The Cosmos ecosystem taught me this lesson in its purest form: IBC is the most elegant interoperability standard ever shipped, and its application layer is a balkanized mess, and the native token captured almost none of the value the chain produced. Elegant primitives, fragmented applications, misaligned value capture. I expect decentralized AI to walk the exact same path unless the builders consciously refuse to.

So my contrarian claim, stated plainly: Project Lily is a net positive for decentralized AI infrastructure if and only if the builders stop selling decentralization as an ideology and start selling verifiability as a compliance product. The customer is not a crypto native. The customer is a risk officer at a bank who needs to prove to a regulator that the AI vendor does not leak. That customer will pay. That customer does not care about the token.

The Infrastructure Layer Nobody Is Pricing In

One more angle before the takeaway, because I have not seen anyone frame it this way and I think it is the most actionable observation in this whole piece.

Data labeling, as an industry, is a large offshore labor market with almost no compliance infrastructure. The vendors — and the reporting names several of the biggest — built their businesses on the premise that labor is cheap and regulation is distant. Project Lily detonates that premise. If human review of sensitive data becomes a regulated activity requiring consent, audit trails, certification of the review environment, and minimum labor standards, then every data-labeling contract in the industry gets repriced.

I do not have a confident dollar estimate, and I will not fabricate one. But I can tell you the direction: the cost of human feedback goes up, the compliance burden goes up, and as a direct consequence, the economic case for synthetic data and for privacy-enhancing technologies improves. That is a genuine, non-narrative shift in capital allocation. It will take eighteen to thirty-six months to play out and it will not be linear. But it is real.

The comparison I keep returning to is medical ethics. Before institutional review boards existed, human-subjects research was a free-for-all, and the scandals that resulted — Tuskegee being the most infamous — forced a structural solution: independent ethical review, enforced standards of consent, documented oversight. The AI data pipeline is pre-IRB human-subjects research with a lighter regulatory touch. Project Lily is the kind of event that builds an IRB.

If that happens — if in five years a frontier model's training pipeline requires independent review-board certification the way a drug trial does — then the crypto industry's verifiability tooling is not a nice-to-have. It is the literal infrastructure of compliance. Proofs of policy adherence. Attested review environments. On-chain audit logs that a regulator can verify without trusting the operator. This is the moment when cryptography stops being a libertarian hobby and becomes the boring plumbing that regulators require.

And boring plumbing is where the durable money is. It is certainly where I would place my bets. Trust no one, verify everything, and — for once — be patient enough to let the verification layer mature before declaring victory.

Takeaway

The user who reads about Project Lily and feels betrayed is not going to be won over by a whitepaper. They are going to be won over by a product that feels exactly as convenient as what they use today and offers cryptographic proof that no human being, in any jurisdiction, at any wage, ever read their messages. That product does not fully exist yet. The proving systems are too slow, the decentralized training pipelines too fragmented, the compliance tooling too immature.

But the demand just became acute, and demand is the scarce resource.

The real question is not whether OpenAI survives this lawsuit — it will, and it will settle for a sum that barely registers on its balance sheet. The real question is whether the years of privacy-anxious engineering that decentralized AI has been doing in the dark are about to meet the one event that makes them a product. We built verifiable computation because we did not trust centralized operators. That conviction was dismissed as ideology for two decades. Project Lily may be the moment it stops being ideology and becomes the reason a risk officer signs a procurement contract.

We didn't need a whistleblower to tell us centralized AI has a trust problem. We needed the receipts. Lily just delivered them. The only open question is whether we will answer with slogans, and get the eye-roll we deserve — or with proofs, and get the market we have been claiming we wanted since 2017.