OpenAI slams court order to save all ChatGPT logs, including deleted chats

Lopez v. Apple (2024) seems to be a recent and useful example of this; my lay understanding is that Apple was found to have failed in its duty to switch from auto-deletion (even if that auto-deletion was contractually promised to users) to an evidence-preservation level of retention, immediately when litigation was filed.

https://codiscovr.com/news/fumiko-lopez-et-al-v-apple-inc/

https://app.ediscoveryassistant.com/case_law/58071-lopez-v-a...

Perhaps the larger lesson here is: if you don't want your service provider to end up being required to retain your private queries, there's really no way to guarantee it, and the only real mitigation is to choose a service provider who's less likely to be sued!

(Not a lawyer, this is not legal advice.)

Yes. That's how the US court system works.

Google can (and would) file to keep that data private and only the relevant parts would be publicly available.

A core aspect to civil lawsuits is everyone gets to see everyone else's data. It's that way to ensure everything is on the up and up.

If Amazon sues Google, a legal obligation to preserve all evidence reasonably related to the subject of the suit attaches immediately when Google becomes aware of the suit, and, yes, if there is a dispute about the extent of that obligation and/or Google's actual or planned compliance with it, the court can issue an order relating to it.

At Google's scale, what would be the hosting costs of this I wonder. Very expensive after a certain point, I would guess.

It can be just anonymised search history in this case.

OpenAI is the custodian of the user data, so they are responsible. If you wanted the court (i.e., the plaintiffs) to find specific infringing chatters, first they'd have to get the data from OpenAI to find who it is -- which is exactly what they're trying to do, and why OpenAI is being told to preserve the data so they can review it.

> So then the courts need to find who is setting their chats do be deleted and order them to stop.

No, actually, it doesn't. Ordering a party to stop destroying evidence relevant to a current case (which is its obligation even without a court order) irrespective of whether someone else asks it to destroy that evidence is both within the well-established power of the court, and routine.

> Or find specific infringing chatters and order OpenAI to preserve these specified users’ logs.

OpenAI is the alleged infringer in the case.

Under this theory, if a company had employees shredding incriminating documents at night, the court would have to name those employees before ordering them to stop.

That is ridiculous. The company itself receives that order, and is IMMEDIATELY legally required to comply - from the CEO to the newest-hired member of the cleaning staff.

> Time does not need user logs to prove such a thing if it was true.

No it needs to show how often it happens to prove a point of how much impact its had.

For the most part (there are a few exceptions), in the US lawsuits are not based on "possible" harm but actual observed harm. To show that, you need actual observed user behavior.

> Times can show that it is possible

The allegation is not that merely that infringement is possible; the actual occurrence and scale are relevant to the case.

> Fair use is the same doctrine that allows a school to play a movie for educational purposes without acquiring a license for the public performance of that movie. This is a pretty bad example, since fair use has been ruled to NOT allow this.

If AI companies don’t want the court headaches they should instead preemptively negotiate with rights holders and get agreements in place for the sharing of data.

Yeah, and the online radio providers argued that they don’t do anything shady, their service was basically just a very long antenna.

Anyway, the laws were not written with this type of processing in mind. In fact the whole idea of intellectual property breaks down now. Just like the early days of the internet.

https://www.copyright.gov/title17/92chap1.html#110 seems to this non-lawyer to be a specific carve out allowing movies to be shown, face-to-face, in non-profit educational contexts without any sort of license. The Fair Use Four Factors test (https://www.copyright.gov/title17/92chap1.html#107) isn't even necessary in this example.

Absent a special legal carve-out, you need to get judges to do the Fair Use Four Factors test, and decide on how AI should be treated. To my very much engineer and not legal eye, AI does great on point 3, but loses on points 1, 2, and 4, so it is something that will need to be decided by the judges, how to balance those four factors defined in the law.

> AI companies aren't seriously arguing that copyright shouldn't apply to them because "it's bad for business".

AI companies have, in fact, said that the law shouldn't apply to them or they won't make money. That is literally the argument Nick Clegg is using to ague that copyright protection should be removed from authors and musicians in the UK.

That's not entirely true. A lot of their briefing refers to how impractical and expensive it would be to license all the content they need for the models.

> allows a school to play a movie

No, it doesn’t. Play 10% of a movie for the purpose of critiquing it, perhaps.

https://fairuse.stanford.edu/overview/fair-use/four-factors/

Fair Use is not an a priori exemption or exception; Fair Use is an “affirmative defense” so once you have your day in court and the judge asks your attorney why you needed to play 10% of Priscilla, Queen of the Desert for your Gender Studies class, then you can run down those Four Factors enumerated by the Stanford article.

Particularly “amount and substantiality”.

Teachers and churches get tripped up by this all the time. But I’ve also been blessed with teachers who were very careful academically and sought to impart the same caution on all students about using copyrighted materials. It is not easy when fonts have entered the chat!

The same reason you or your professor cannot show/perform 100% of an unlicensed film under any circumstance, is the same basis that creators are telling the scrapers that they cannot consume 100% of copyrighted works on that end. And if the risks may involve reproducing 87% of the same work in their outputs, that’s beyond the standard thresholds.

I actually agree with your disagreement, and my answer is more scoped to a technical audience that has the know how base to deal with it.

I wish it was different and I agree that there’s a massive accountability hole with… who could it be?

Pragmatically it is what it is, self host and hope for bigger picture change.

Anything else is literally impossible, though.

If you send your neighbour nudes then they have your nudes. You can put in as many contracts as you want, maybe they never digitised it but their friend is over for a drink and walks out of the door with the shoebox of film. Do not pass GO, do not collect.

Conceivably we can try to control things like e.g. is your cellphone microphone on at all times, but once someone else, particularly an arbitrary entity (e.g. not a trusted family member or something) has the data, it is silly to treat it as anything other than gone.

Then your problem is with the US legal system, not this individual ruling.

You lose your rights to privacy in your papers without a warrant once you hand data off to a third party. Nothing in this ruling is new.

IP law was to promote human progress by giving financial incentive to create this IP knowing it was protected, and you could make money off it.

The court in question has no obligations to you at all.

The current legal stance in the US seems to be that you, not being a US person, have no particular legally protected interest in privacy at all, so you have nothing to complain about here and can’t even sue. The only avenue the EU would have to change that is the diplomatic one, but the Commission does not seem to care.

you aren't and they don't.

Roe v. wade is considered explicitly overruled, as well as considered wrongly decided in the first place, as of 2022 (Dobbs).

They also explicitly stated a constitutional right to privacy does not exist, and pointed out that Casey abandoned any such reliance on this sort of claim.

Griswold also found a right to marital privacy. Not general privacy.

Griswold is also barely considered good law anymore, though i admit it has not been explicitly overruled - it is definitely on the chopping block, as more than just Thomas has said.

In any case, more importantly, none of them have found any interesting right to privacy of the kind we are talking about here, but instead more specific rights to privacy in certain contexts. Griswold found a right to marital privacy in "the penumbra of the bill of rights". Lawrence found a right to privacy in your sexual activity.

In dobbs, they explicitly further denied a right to general privacy, and argued previous decisions conflated these: " As to precedent, citing a broad array of cases, the Court found support for a constitutional “right of personal privacy.” Id., at 152. But Roe conflated the right to shield information from disclosure and the right to make and implement important personal decisions without governmental interference."

You are talking about the former, which none of these cases were about. They are all about the latter.

So this is very far afield from a general right to privacy of the kind we are talking about, and more importantly, one that would cover anything like OpenAI chats.

So basically, you have a ~200 year period where it was not considered a right, and then a 50 year period where specific forms of privacy were considered a right, and now we are just about back to the former.

The kind of privacy we are talking about here ("the right to shield information from disclosure") has always been subject to a balancing of interests made by legislatures, rather than a constitutional right upon which they may not infringe. Example abound - you actually don't have to look any further than court filings themselves, and when you are allowed to proceed anonymously or redact/file things under seal. The right to public access is considered much stronger than your right to not want the public to know embarassing or highly private things about your life. There are very few exceptions (minors, etc).

Again, i don't claim any of this is how it is should be. But it's definitely how it is.

¯\_(ツ)_/¯ The supreme court overturned Roe v. Wade in 2022 and explicitly stated in their ruling that a constitutional right to privacy does not exist.

I understand what they mean. There's this great video [1] which explains it in better terms than I ever could. I've timestamped the link because it's quite long, but if you've got the time it's a fantastic video with a great narrative and presentation.

[1] https://youtu.be/Fzhkwyoe5vI?t=4m9s

"public interest" is a much more ambiguous thing than the written law

Nice find! Maybe this is a ploy by OpenAI to use API requests for training while blaming the courts?

My standard reply to such comments over the past year has been the same: you probably want to use Azure instead. A big part of the business value they provide is ensuring regulatory compliance.

There are multinational corporations with heavy presence in Europe, that run their whole business on Microsoft cloud, including keeping and processing there privacy-sensitive data, business-critical data and medical data, and yes, that includes using some of this data with LLMs - hosted on Azure. Companies of this size cannot ignore regulatory compliance and hope no one notices. This only works because MS figured out how to keep it compliant.

Point being, if there are business consequences, you'll be better off using Azure-hosted LLMs than running a local model yourself - they're just better than you or me at this. The only question is, whether you can afford it.

I've been poking around the medical / ehr LLM space and gently asking people how they're preserving privacy and everyone appears to be just shipping data to cloud providers based solely on a BAA. Kinda baffling to me, my first step would be to set up local models even if they're not as good, data breaches are expensive.

Yeah its an awkward position, as self-hosting is going to be insanely expensive unless you have a substantial userbase to amortize the costs over. At least for a model comparable to GPT-4o or deepseek.

But at least if you use an API in the same region as your customers, court order shenanigans won't get you caught between different jurisdictions.

And ironically because OpenAI is actually ClosedAI, the best self-hostable model available currently is a Chinese model.

OpenAI keeping the logs is the "you have no privacy" part. Anyone who inspects those logs can see what the users were doing. But now everyone knows they're keeping logs and they can't lie their way out of it. So, for your own legal safety, put it in your TOS. Then every user should know they can't use your service if they want privacy.

Yeah. You basically need cyberpunk style corporate extraterritoriality to get that particular benefit, of being able to tell governments to go screw themselves.

I don't think the suit would be against you preserving it, it would be against you falsely representing that you aren't preserving it.

A court ordering you to stop selling pigeons doesn't mean you can keep your store for pigeons open and pocket the money without delivering pigeons.

Almost all privacy policies are going to have a call out for legal rulings. For example, here is the Hackernews Legal section in the privacy policy (https://www.ycombinator.com/legal/)

> Legal Requirements: If required to do so by law or in the good faith belief that such action is necessary to (i) comply with a legal obligation, including to meet national security or law enforcement requirements, (ii) protect and defend our rights or property, (iii) prevent fraud, (iv) act in urgent circumstances to protect the personal safety of users of the Services, or the public, or (v) protect against legal liability.

Yes. If your agreement with the end user says that you won't collect and store data, you're responsible for it. If you can't provide it (even if due to a court order), you have to adjust your contract.

Your users aren't obligated to know that you're using open ai or other provider.

> If a court orders you to preserve user data, could you be held liable for preserving user data?

No, because you turn up to court and show the court order.

It's possible a subsequent case could get the first order overturned, but you can't be held liable for good faith efforts to comply with court orders.

However, if you're operating internationally, then suddenly it's possible that you may be issued competing court orders both of which are "valid". This is the CLOUD Act problem. In which case the only winning move becomes not to play.

No. It’s a legal court order.

This, however, is horrible for AI regardless of whether or not you can sue.

The problem ultimately isn't a technical one but a political one.

Point 1: Every company has profit incentive to sell the data in the current political climate, all they need is a sneaky way to access it without getting caught. That includes the combo of LLM provider and Escrow non-entity.

Point 2: No company has profit incentive to defend user privacy, or even the privacy of other businesses. So who could run the Escrow service? Another business? Then they have incentive to cheat and help the LLM provider access the data anyway. The government (and which one)? Their intelligence arms want the data just as much as any company does so you're back to square one again.

"Knowledge is power" combined with "Knowledge can be copied without anyone knowing" means that there aren't any currencies presently powerful enough to convince any other entity to keep your secrets for you.

No, because users don't care about privacy all that much, and for corporate clients discovery is always a risk anyway.

See the whole LIBOR chat business.

afaik only OpenAI is enjoined in this

Yes but it shifts all the value onto companies producing hardware and selling enterprise software to people who get locked into contracts. The market is significantly smaller # of companies and margins if they have to build value adds they won't charge for to move hardware.

None that I've worked for so I don't really track the statistics tbh.

We've always done self-hosted as old as things like gerrit and what not that aren't even really feature complete as competitors where I've worked.

Which gives you an opening for the excellent double contraction “shouldn’t’ve”

I care.

That used to be the case, but "shouldn't of" is definitely becoming more popular, even if it seems wrong. Languages change before our eyes :)

Who cares?

AFAIK, the actual trading algorithms themselves aren’t usually that far from what you can find in a textbook, their efficacy is mostly dictated by market conditions and the performance characteristics of the implementation / system as a whole.

Most (all?) hedge funds that use AI models explicitly run in-house. People do use commercial LLMs, but in cases where the LLMs are not run in-house, it's against the company policy to upload any proprietary information (and generally this is logged and policed).

A lot of the use is fairly mundane and basically replaces junior analysts. E.g. it's digesting and summarizing the insane amounts of research that is produced. I could ask an intern to summarize the analysis on platinum prices over the last week, and it'll take them a day. Alternatively, I can feed in all the analysis that banks produce to an LLM and have it done immediately. The data fed in is not a trade secret really, and neither is the output. What I do with the results is where the interesting things happen.

IIUC one reason is that prompts and other data sent to 3rd party LLM hosts have the chance to be funneled to 4th party RLHF platforms, e.g. Sagemaker, Mechanical Turks, etc. So a random gig worker could be reading a .env file the intern uploaded.

The plans scale down far enough that they can't possibly cover the cost of a private model-loaded-to-vram instance at the low end.

I get where it's historically coming from, but the combination of American courts having almost infinite discovery rights (to be paid by the losing party, no less, greatly increasing legal risk even to people and companies not out to litigate) and the result of said discoveries ending up on the public record seems like a growing problem.

There's a qualitative difference resulting from quantitatively much easier access (querying some database vs. having to physically look through court records) and processing capabilities (an army of lawyers reading millions of pages vs. anyone, via an LLM) that doesn't seem to be accounted for.

People need to read up on the LIBOR scandal. There was a lot of "wait why are my chat logs suddenly being read out as evidence of a criminal conspiracy".

Isn't it a risk even if they retain nothing? Likely less of a risk, but it's still a risk that you have no way to deep dive on, and you can still get "pwned" because someone broke into their servers.

Cannot emphasize this enough. If your psychologist’s records can be held for ransom, surely your ChatGPT queries will end up on the internet someday.

Do search engine companies have this requirement as well? I remember back in the old days deanonymizing “anonymous” query logs was interesting. I can’t imagine there’s any secrecy left today.

That's a pretty significant difference, though.

OpenAI (and other services) log and preserve your interactions, in order to either improve their service or to provide features to you (e.g., your chat history, personalized answers, etc., from OpenAI). If a court says "preserve all your user interaction logs," they exist and need to be preserved.

DDG explicitly does not track you or retain any data about your usage. If a court says "preserve all your users interaction logs," there is nothing to be preserved.

It is a very different thing - and a much higher bar - for a court to say "write code to begin logging user interaction data and then preserve those logs."

DuckDuckGo uses Bing.

It would be interesting to know how much Microsoft logs or tracks.

> but this would be like ordering phone companies to record ALL calls just in case some might become evidence later

That's not a good analogy. They're ordered to preserve records they would otherwise delete, not create records they wouldn't otherwise have.

No, it wouldn't be like that at all. Phone companies and telephone calls are covered under a different legal regime so your analogy is invalid.

That’s… not how discovery works

They will do it when they need the money and/or feel they have the leverage for precisely the same reason that 99% of people won’t care. It’s better to assume they’re just sitting on your data and waiting until they can get away with using it.

That's nice. How can a user verify whether they fully comply with those contracts?

They have every incentive not to, and no oversight to hold them accountable if they don't. Do you really want to trust your data is safe based on a pinky promise from a company?

> Or, the general populace just doesn't understand the actual implications.

That might've been the case in the first generations of ad-supported business models on the web. But after two decades, even non-technical users have understood the implications of "free" services.

IME talking to non-technical people about this topic, I can't remember the last time someone mentioned not being aware of the ToS and privacy policies they agree to, even if they likely hadn't read the legalese. Whereas the most common excuses I've heard are "I have nothing to hide", and "I don't use it often".

So I think you're underestimating the average person's tech literacy. I'm sure people who who still don't understand the implications exist, but they're in the minority.

It's like anonymizing your diary by erasing your name on the cover.

I don't dispute your example, but I suspect there is a non-zero number of cases that would not be so extreme, so obviously identifiable.

So, sure, no panacea, but .. why not for the cases where it would be a barrier?

https://en.wikipedia.org/wiki/K-anonymity

Not wanting to do something isn't the same thing as being unable to do something.

!define would

> Used to express desire or intent -- https://www.wordnik.com/words/would

!define cannot

> Can not ( = am/is/are unable to) -- https://www.wordnik.com/words/cannot

Even my 4-year-old M1 Pro can run a quantized Deepseek R1 pretty well. Sure, full-scale productizing these models is hard work (and the average "just-make-shovels" startups are failing hard at this), but we'll 100% get there in the next 1-2 years.

I put it LM Studio on an old gaming rig with a 3060 TI, took about 10 minutes to start using it and most of that time was downloading a model.

If you're dealing with ITAR compliance you should have experience with hosting things on-premises.

Yes. The past two companies I've been at have self-hosted enterprise LLMs running on their own servers and connected to internal documentation. There is also Azure Cloud for Gov and other similar privacy-first ways of doing this.

But also, running LLMs locally is easy. I don't know what goes into hosting them, as a service for your org, but just getting an LLM running locally is a straightforward 30-minute task.

I'm for hire, I'll do all that for any company that needs it. Email in profile. Contract or employee, makes no difference to me.

This hasn't been my experience. Pretty easy with AWS Bedrock

Deepseek isn't good enough? You need a beefy GPU cluster but I bet it would be fine until the large llama is better at coding, and I'm certain there will be other large models for LLM. Now if there's some new technology around the corner, someone might be able to build a moat, but in a surprising twist, Facebook did us all a favor by releasing their weights back when; there's no moat possible, in my estimation, with LLMs as it stands today. Not even "multi-model" implementations. Which I have at home, too.

Say oai implements something that makes their service 2x better. Just using it for a while should give people who live and breathe this stuff enough information to tease out how to implement something like it, and eventually it'll make it into the local-only applications, and models.

Yeah, shoot the messenger, that has always worked.

That's happening. Unmodified LLM outputs aren't copyrightable.

thanks for engaging.

> The linked tweet (whatever it's called)

"post" works for social media regardless of the medium; not an admonishment, an observation. Also, by the time i saw this, it was already an Ars link, leaving some comments with less context that i apparently didn't pick up on. I was able to make my observation because someone mentioned mastodon (i think), but that was an assumption on my part that the original link was mastodon.

So i asked the question to make sure it wasn't some bias against mastodon (or the fediverse), because I'd have liked to ask, "for what reason?"

This is why American cloud providers have legal entities outside of the US. Those have to comply with the law in the countries where they are based if they want to do business there. That's how AWS, Azure, GCP, etc. can do business in the EU. Most of that business is neatly partitioned from any exposure to US courts. There are some treaties that govern what these companies can and cannot send back to the US that some might take issue with and that are policed and scrutinized quite a bit on the EU side.

OpenAI does this as well of course. Any EU customers are going to insist on paying via an EU based entity in euros and will be talking to EU hosted LLMs with all data and logs being treated under EU law, not US law. This is not really optional for commercial use of SAAS services in the EU. To get lucrative enterprise contracts outside the US, OpenAI has no other choice but to adapt to this. If they don't, somebody else will and win those contracts.

I actually was at a defense conference in Bonn last week talking to a representative of Google Cloud. I was surprised that they were there at all because the Germans are understandably a bit paranoid about trusting US companies with hosting confidential stuff (considering some scandals a few years ago about the CIA spying on the German government a few years ago). But they actually do offer some services to the BWI, which is the part of the German army that takes care of their IT needs. And German spending on defense is of course very high right now so there are a lot of companies trying to sell in Germany, on Germany's terms. Including Google.

That's OpenAI's issue, not the court.