Anyone who trusts an AI therapist needs their head examined

That wad the good thing about the old Eliza program: no cloud, no memory between conversations. Heck, barely any memory within the conversation at all, except the last one or two sentences.

That's the trouble though, isn't it? It costs money to get my head examined. I can talk to ChatGPT for free.

“therapy requires confidentiality”

That pretty much sums it for me. If I do use LLMs for ‘therapy’, I never login. Is that enough confidentiality? I don’t know

I actually work directly in this space, and am deeply involved in rolling out AI features for a mental health platform.

Note 1: This is a long comment mostly because I go into detail on a system design intended to mitigate privacy concerns for an AI-powered feature on a mental health platform. If you've got a sec, I'd love feedback on it!

Note 2: In this comment I'm using "AI" to refer to generative AI in general, and LLMs in particular.

First, I'll say that I broadly agree with the theme of the article: there are major privacy and efficacy concerns around the use of AI in healthcare. Those concerns are only heightened for mental health specifically. That said, I don't think they are insurmountable, and there are several premises in the article where I disagree -- hence my opposing conclusion :)

> There's a debate to be had about whether AI chatbots make good psychotherapists. This is not an area of my expertise, so I'm not going to weigh in on that debate.

It's not my area either, but the people I know and trust for whom it _is_ their area of expertise all seem to agree that AI is surprisingly effective in the role of a counselor or mentor.

> [...] and holy shit is the idea of a chatbot psychotherapist running on some Big Tech cloud a terrible idea.

Ah, there's the rub. I 100% agree that running therapy sessions through a model hosted and controlled by a third party is a terrible idea.

For my job, I'm addressing this by standing firm on my decision that no data that can be associated with a specific patient should ever be sent outside our infrastructure. Where possible, models should be open source (preferably "open weights" as well, since the term isn't well-defined for LLM models yet).

Fine-tuning may be done using data collected from other patients, but at no point should the model be trained on any text that can be tied back to an individual, even through inference or statistical means. I'm paying very close attention in particular to meeting transcripts, as those are both particularly difficult to have 100% confidence in their anonymity and extremely sensitive.

PII/PHI should only ever be made available to the model within the context of a single conversation. This was looking to be very expensive in practice, but token caching on AWS Bedrock seems so far to have improved things to the point where it's not a concern.

> Now consider the chatbot therapist: what are its privacy safeguards? Well, the companies may make some promises about what they will and won't do with the transcripts of your AI sessions, but they are lying.

I assure you that I am not lying - but you shouldn't trust me, in case I'm lying :). That's why while it's critical that privacy safeguards be in place, it's equally important that users have both visibility into and control over their data.

Patients must explicitly consent to data collection before any data collection is initiated.

Patients must explicitly consent to data collection every time a session transcript is creaed.

Patients must have a clear way to opt out of all data collection and AI functionality.

Patients must have a dashboard that clearly describes what data are stored, for what purpose, and for how long. This includes logs for the deletion of artifacts.

Patients must be able to completely remove their data from our systems without assistance. This includes everything from purging a single artifact all the way up to "remove every reference that I ever registered for an account"

> Of course they're lying! AI companies lie about [...]. But most of all, they lie about data.

Ah, that's fair. We're not an AI company doing healthcare; we're a healthcare company doing AI. Trust has always been a fundamental requirement in our line of work.

> What's more: they will leak your therapy sessions. They will leak them because they can't figure out how to prevent models from vomiting up their training data verbatim:

Yeah, this terrifies me. Here's how I'm addressing that:

Text transcripts are generated via Zoom's built-in cloud recording functionality

If video recordings must be created - I'm trying to find a way to prevent that - then they should be explicitly deleted immediately. There must also be monitoring in place to show aggregate statistics about our Zoom data artifacts, and alerting in place so that no recordings (audio, video, or text) ever remain on Zoom's system for more than a very short time. I'm aiming to set that time at 30 minutes. If the time limit is exceeded even for a single file, that will be considered a p1 alert that on-call staff must immediately investigate and resolve.

Text transcripts are processed using an appropriate tool; all personally-identifiable tokens (names, phone numbers, addresses, etc.) are replaced with generic placeholders. We used Microsoft's Presidio during our initial trial - while Presidio explicitly calls out that it is not intended to satisfy HIPAA requirements, that's not actually a requirement here.

Two artifacts come out of this process: a processed transcript and a JSON object containing tokens and their values.

Both are stored in s3, in separate buckets. Each bucket is encrypted using a dedicated KMS key. No one in our organization has the ability to read those buckets without manually retrieving the KMS key and adding the appropriate statements to their role's IAM policy doc. All access to those buckets outside the automated ingestion process itself is monitored and alarmed.

For both buckets, filenames are a hash of the zoom meeting ID, a salt (the patient's user ID, a UUID), and a pepper (a global salt used only for this purpose). This is intended so that associating either artifact type with a given user requires access to multiple systems: our codebase for the hash structure and algorithm, the pepper (an env var on running tasks; an SSM SecureString otherwise), the artifact itself, and multiple tables from our main production database.

For the JSON object, as it contains significantly more sensitive data, we have the additional step of symmetrically encrypting the values inside the object using a secret that is dynamically derived from the zoom meeting ID, the user's ID, a salt, a pepper, and one or more arbitrary additional values just to muddy the waters.

The intention here is to make it very difficult to associate these data by creating a rainbow table or similar. Even with access to all the data on all the various systems, it would still be very difficult to start with a processed transcript file and determine what user it belongs to. It would be harder still to do that with a sensitive JSON object. Producing the original unprocessed transcripts would mean having to meet both of those challenges.

On the other hand, given that you have access to all the necessary data and secrets, we can retrieve all stored data for a given user easily. Additional secrets are needed to actually read it, though! This means that we can respond to user requests for data removal, whether we're required to do so (GDPR) or simply believe it to be the right thing to do (a 100% transparent data dashboard; granular ability to review and delete individual artifacts).

-----

Now that I've typed all that out instead of spending that time actually building it, I would love feedback on the approach. I'm not an encryption or security expert - I have experience, but it's not my specialization - so by all means poke holes in it. It doesn't have to be a gaping security hole. Bear in mind the level of responsibility and care I'm trying to take here to ensure that I'm going well above and beyond the legal requirements; my primary goal is to prevent any data breach in the first place, but I see it as equally important to minimize the impact if I should fail to do so.

To put it another way: I'm confident that I can ensure that I'm meeting all the legal and regulatory requirements. I want to ensure that even a breach of PHI/PII that mandates reporting and notification under HIPAA would have no objective impact to our users. If you might be able to help with that, I'm all ears!

There's plenty of bad human therapists. It's hard to vet quality of care when you're in crisis and bad therapists can cause great harm.

Im pretty bullish on AI therapy agents, at least for mechanistic-ish stuff like CBT where emotional attachment to the therapist isn't super important.

TFA is complaining about cloud software, not AI therapists. Totally agree with that complaint. But LLMs can run locally, and cloud software can be wrapped in enough regulations to stay private.