School AI surveillance can lead to false alarms, arrests

"Are you thinking the outcome would have been much different if she threatened to kill Mexicans on Teams at work instead?"

Yeah, she'd probably get fired but law enforcement wouldn't get called because there was no specific or credible threat and the company doesn't have the mandatory reporting requirements that the school does.

The child in OP sounds to me like she thought she was making a bad-taste joke in a private forum, and was shocked when it promptly led to an entanglement with an unfeeling system who was looking over her shoulder. I’ve never threatened to kill anyone in my work DMs, but I’ve definitely written stuff that I wouldn’t post in public threads. I think we all to some degree use these “private” systems this way until it burns us, only then do we adjust. Privacy is much more a feeling than a technical reality.

In that sense, it isn’t “normal”, it’s just “something that’s happening in theory but eh maybe it only affects scary people or whatever idk”. I feel like this tolerance we’re developing for outside forces invading “private” spaces, nominally for these loose justifications of harm reduction, will be what _actually does_ make it normal.

Once it’s truly normal, and people think it’s what keeps them safe from mass shootings or whatever, it will be too late to get rid of it. I think fear and normalcy will motivate its spread to places beyond school chat platforms and Snapchat.

This is nefarious.

[flagged]

"can't really blame the school for jumping to conclusions"

Of course you can blame the school. They were too lazy to look at context and determine if the threat was real and credible. They took the determination of a complex tool as an unquestionable fact. The system supplies the fact that the user's account made the comment. All other facts need to be made by investigation. This statement provides a reasonable suspicion to investigate, but should not exhibit probable cause for an arrest as it requires a threat be credible, incite panick, etc per the specific terrorist threat law. This requires investigation and thought.

It's a threat of violence in context as well.

I mostly agree with you. But the same is true with your employer in the US. You do something stupid outside of work like an OUI and there will likely be consequences with employment - either current or future. Not terrible to internalize that early when the consequences are relatively forgettable.

Public schools get money from the government. That doesn't mean they're are the private arm of the government.

Public schools are not the US government. They're usually city or county.

AI has nothing to do with this. It’s trivial to build a keyword based system to detect someone threatening/joking that they will kill all the Mexicans at their school.

AI hype is at play here as well, not only in the breathless press releases from AI companies.

I have a problem with the degree of punishment but not that the student was punished.

A student said, “on Thursday we kill all the Mexico’s,” on a schools private communication platform. The school should correct that behavior.

Unfortunately they involved law enforcement. Thats where I see the problem. A better solution would be detention and informing the parents.

The fact that weapons exist, might make some people feel compelled to always shoot as much as possible.

Sure, these people exist. They are dangerous.

Adding more and more safety and control is always a self-defeating policy. It always ends with creating uncontrollable societal discord.

Same thing with unfettered capitalism, the systems only work if we continue to support said systems. When the rules break down, so do the desires of the collective to maintain said systems.

Yeah, it's a feature. If anything, its a basement bargain feature. For any large system, indoctrination early and often is the best way for systemic change.

We already see it with the modern surveillance state, post 9-11 the US citizenry has lost so many freedoms and if you ask random people on the street about it they would be perplexed. Hell, even my friends give me a bit of the "ahh so this is your conspiracy theory" look when I mention them. Growing up through 9-11 and the forever war was pretty dystopian, or at least a March into the dystopia's that I only read about in books.

My kid went to public school in Canada. He learned English pretty quickly and one day got suspended from school for a week. When I asked what had happened he said the other kids asked him to pronounce f.u.c.k. and then, after he complied ran to the teacher to say that he'd used a 'bad word'.

So, the principal, one Roman Peredun calls me up and says that my son used a bad word. I asked him what word. He wouldn't say it. So I asked how am I supposed to know how 'bad' my son is if you can't even repeat the word. He then spelled the word. I said, oh, 'fuck'. Yes, that's not in dutch however so he must have picked it up in your school. Peredun hung up and I sent my kid back to school the next day.

I can confirm this. Sent a 5yo to public school and they came home with the new saying "I don't give a fuck!". Not what I want a 5yo to learn. We have massive overreactions by the schools for comments like in the article, but they're culpable in this sort of behavior by creating a largely undisciplined environment for basic in-person behavior. Uncontrolled classrooms and busses lead to all sorts of problems because nobody gets punished on the low end. My kid got in trouble for something and the punishment was to play alone at recess. Really? I told the school I think they should have been in detention at recess, possibly for a couple days. Of course they got punished at home too, which I suppose doesn't happen in some households.

> The child was trained on harmful data, it's not surprised the child exhibited undesirable behaviours.

Your comment reduces children to entities that will behave as expected provided they get fed “good” data.

Humans are not LLMs.

Any system will have failures. The goal is to eliminate some of those failures through multiple checks. The automated system can help by bringing up possible issues that couldn't be monitored by humans. But then it should be humans reviewing these alerts for false positives. It should be much like using a security scanning tool. Findings will come up and you need someone to disposition them. Some will need no actions false positives or issues under your risk tolerance, but then others need appropriate responses. It seemes in the cases like the example, you have a intern running the scan tool and turning every finding into a high vulnerability because they don't know any better than to blindly trust the tool.

We always rely on humans being competent. If we forget that and treat everything as being correct then we are doomed.