Let's talk about AI and end-to-end encryption

> You might even convince yourself that these questions are “privacy preserving,” since no human police officer would ever rummage through your papers, and law enforcement would only learn the answer if you were (probably) doing something illegal.

Something I've started to see happen but never mentioned is the effect automated detection has on systems: As detection becomes more automated (previously authored algorithms, now with large AI models), there's less cash available for individual case workers, and more trust at the managerial level on automatic detection. This leads to false positives turning into major frustrations since it's hard to get in touch with a person to resolve the issue. When dealing with businesses it's frustrating, but as these get more used in law enforcement, this could be life ruining.

For instance - I got flagged as illegal reviews on Amazon years ago and spent months trying to make my case to a human. Every year or so I try to raise the issue again to leave reviews, but it gets nowhere. Imagine this happening for a serious criminal issue, with the years long back log on some courts, this could ruin someones life.

More automatic detection can work (and honestly, it's inevitable) but it's got to acknowledge that false positives will happen and allocate enough people to resolve those issues. As it stands right now, these detection systems get built and immediately human case workers get laid off, there's this assumption that detection systems REPLACE humans, but it should be that they augment and focus human case workers so you can do more with less - the human aspect needs to be included in the budgeting.

But the incentives aren't there, and the people making the decisions aren't the ones working the actual cases so they aren't confronted with the problem. For them, the question is why save $1m when you could save $2m? With large AI models making it easier and more effective to build automated detection I expect this problem to get significantly worse over the next years.

> Yet this approach is obviously much better than what’s being done at companies like OpenAI, where the data is processed by servers that employees (presumably) can log into and access.

No need for presumption here: OpenAI is quite transparent about the fact that they retain data for 30 days and have employees and third-party contractors look at it.

https://platform.openai.com/docs/models/how-we-use-your-data

> To help identify abuse, API data may be retained for up to 30 days, after which it will be deleted (unless otherwise required by law).

https://openai.com/enterprise-privacy/

> Our access to API business data stored on our systems is limited to (1) authorized employees that require access for engineering support, investigating potential platform abuse, and legal compliance and (2) specialized third-party contractors who are bound by confidentiality and security obligations, solely to review for abuse and misuse.

The real threat here is going to be when AI expands from being applied to accelerate the work of individuals, to being applied to the control of organisations. And it will be tempting to do that. We all know the limitations of managers, management hierarchies, metrics, OKRs etc. It's easy to think of a CEO deciding that all the communications between their employees should just be fed into an AI that they can query . (Ironically that would be easier to enforce if everyone was remote). It's quite possible that it would enable more effective organisations, as the CEO and upper level management can have a better idea of what is really happening. But it will reduce the already tenuous belief of the powerful that their ordinary staff are real human beings. And it will inevitably leak out from private organisations, as the executive class see no reason why they shouldn't have the same tools when running the country as when running a corporation.

Advocates of mass surveillance like to point out that no human now needs to listen to your calls. But the real danger was never the guy in a drab suit transcribing your conversions from reel-to-reel tape. It was always the chief who could call for the dossier on anyone they were finding inconvenient, and have it looked at with an eye to making you never inconvenience them again.

The full consequences of mass surveillance have not played out simply because no one had the means to process that much ad-hoc unstructured data. Now they do.

> We are about to face many hard questions about these systems, including some difficult questions about whether they will actually be working for us at all.

And how. I'd lean towards no. Where we're headed feels like XKEYSCORE on steroids. I'd love to take the positive, optimistic bent on this, but when you look at where we've been combined with the behavior of the people in charge of these systems (to be clear, not the researchers or engineers, but c-suite), hope of a neutral, privacy-first future seems limited.

> Apple even says it will publish its software images (though unfortunately not the source code) so that security researchers can check them over for bugs.

I think Apple recently changed their stance on this. Now, they say that "source code for certain security-critical PCC components are available under a limited-use license." Of course, would have loved it if the whole thing was open source. ;)

https://github.com/apple/security-pcc/

> The goal of this system is to make it hard for both attackers and Apple employees to exfiltrate data from these devices.

I think Apple is claiming more than that. They are saying 1/ they don't keep any user data (data only gets processed during inference), 2/ no privileged runtime access, so their support engineers can't see user data, and 3/ they make binaries and parts of the source code available to security researchers to validate 1/ and 2/.

You can find Apple PCC's five requirements here: https://security.apple.com/documentation/private-cloud-compu...

Note: Not affiliated with Apple. We read through the PCC security guide to see what an equivalent solution would look like in open source. If anyone is interested in this topic, please hit me up at ozgun @ ubicloud . com.

> Who does your AI agent actually work for?

Yes. I made that point a few weeks ago. The legal concept of principal and agent applies.

Running all content through an AI in the cloud to check for crimethink[1] is becoming a reality. Currently proposed:

- "Child Sexual Abuse Material", which is a growing category that now includes AI-generated images in the US and may soon extend to Japanese animation.

- Threats against important individuals. This may be extended to include what used to be considered political speech in the US.

- Threats against the government. Already illegal in many countries. Bear in mind that Trump likes to accuse people of "treason" for things other than making war against the United States.

- "Grooming" of minors, which is vague enough to cover most interactions.

- Discussing drugs, sex, guns, gay activity, etc. Variously prohibited in some countries.

- Organizing protests or labor unions. Prohibited in China and already searched for.

Note that talking around the issue or jargon won't evade censorship. LLMs can deal with that. Run some ebonics or leetspeak through an LLM and ask it to translate it to standard English. Translation will succeed. The LLM has probably seen more of that dialect than most people.

"If you want a vision of the future, imagine a boot stepping on a face, forever" - Orwell

[1] https://www.orwell.org/dictionary/

The most depressing realization in all of this is that the vast treasure trove of data that we used to have in the cloud thinking it was not scannable even for criminal activity has now become a vector where we shall have thought police coming down upon us for simple ideas of dissent.

It's a good thing that encrypted data at rest on your local device is inaccessible to cloud based "AI" tools. The problem is that your average person will blithely click "yes/accept/proceed/continue/I consent" on pop up dialogs in a GUI and agree to just about any Terms of Service, including decrypting your data before it's sent to some "cloud" based service.

I see "AI" tools being used even more in the future to permanently tie people to monthly recurring billing services for things like icloud, microsoft's personal grade of office365, google workspace, etc. You'll pay $15 a month forever, and the amount of your data and dependency on the cloud based provider will mean that you have no viable path to ever stop paying it without significant disruption to your life.

Green, (the author), makes an important point: > a technical guarantee is different from a user promise. [...] End-to-end encrypted messaging systems are intended to deliver data securely. They don’t dictate what happens to it next.

Then Green seems to immediately forget the point they just made, and proceed to talk about PCC as if it were something other than just another technical guarantee. PCC only helps to increase confidence that the software running on the server is the software Apple intended to be there. It doesn't give me any guarantees about where else my data might be transferred from there, or whether Apple will only use it for purposes I'm okay with. PCC makes Apple less vulnerable to hacks, but doesn't make them any more transparent or accountable. In fact, to the extent that some hackers hack for pro-social purposes like exposing corporate abuse, increased security also serves as a better shield against accountability. Of course, I'm not suggesting that we should do away with security to achieve transparency. I am, however, suggesting that transparency, moreso than security, is the major unaddressed problem here. I'd even go so far as to say that the woeful state of security is enabled in no small part by lack of transparency. If we want AI to serve society, then we must reverse the extreme information imbalance we currently inhabit wherein every detail of each person's life is exposed to the service provider, but the service provider is a complete black-box to the user. You want good corporate actors? Don't let them operate invisibly. You want ethical tech? Don't let it operate invisibly.

(Edit: formatting)

The author helpfully emphasized the interesting question at the end

> This future worries me because it doesn’t really matter what technical choices we make around privacy. It does not matter if your model is running locally, or if it uses trusted cloud hardware — once a sufficiently-powerful general-purpose agent has been deployed on your phone, the only question that remains is who is given access to talk to it. Will it be only you? Or will we prioritize the government’s interest in monitoring its citizens over various fuddy-duddy notions of individual privacy.

I do think there are interesting policy questions there. I mean it could hypothetically be mandated that the government must be given access to the agent (in the sense that we and these companies exist in jurisdictions that can pass arbitrary laws; let’s skip the boring and locale specific discussion of whether you think your local government would pass such a law).

But, on a technical level—it seems like it ought to be possible to run an agent locally, on a system with full disk encryption, and not allow anyone who doesn’t have access to the system to talk with it, right? So on a technical level I don’t see how this is any different from where we were previously. I mean you could also run a bunch of regex’s from the 80’s to find whether or not somebody has, whatever, communist pamphlets on their computers.

There’s always been a question of whether the government should be able to demand access to your computer. I guess it is good to keep in mind that if they are demanding access to an AI agent that ran on your computer, they are basically asking for a lossy record of your entire hard drive.

The article hinges on a bad assertion that

> Apple can’t rely on every device possessing enough power to perform inference locally. This means inference will be outsourced to a remote cloud machine.

If you go look at Apple's site https://www.apple.com/apple-intelligence/ and scroll down, you get:

Apple Intelligence is compatible with these devices. iPhone 16 A18 iPhone 16 Plus A18 iPhone 16 Pro Max A18 Pro iPhone 16 Pro A18 Pro iPhone 15 Pro Max A17 Pro iPhone 15 Pro A17 Pro iPad Pro M1 and later iPad Air M1 and later iPad mini A17 Pro MacBook Air M1 and later MacBook Pro M1 and later iMac M1 and later Mac mini M1 and later Mac Studio M1 Max and later Mac Pro M2 Ultra

If you don't have one of those devices, Apple did the obvious thing and disabled features on devices that don't have the hardware to do it.

While Apple has this whole private server architecture, they're not sending iMessages off device for summarization, that's happening on device.

I heard that homomorphic encryption can actually preserve all the operations in neural networks, since they are differentiable. Is this true? What is the slowdown in practice?