Writing "/etc/hosts" breaks the Substack editor
369 comments
·April 25, 2025matt_heimer
mjr00
> I disagree with other posts here, it is partially a balance between security and usability.
And economics. Many people here are blaming incompetent security teams and app developers, but a lot of seemingly dumb security policies are due to insurers. If an insurer says "we're going to jack up premiums by 20% unless you force employees to change their password once every 90 days", you can argue till you're blue in the face that it's bad practice, NIST changed its policy to recommend not regularly rotating passwords over a decade ago, etc., and be totally correct... but they're still going to jack up premiums if you don't do it. So you dejectedly sigh, implement a password expiration policy, and listen to grumbling employees who call you incompetent.
It's been a while since I've been through a process like this, but given how infamous log4shell became, it wouldn't surprise me if insurers are now also making it mandatory that common "hacking strings" like /etc/hosts, /etc/passwd, jndi:, and friends must be rejected by servers.
swiftcoder
Not just economics, audit processes also really encourage adopting large rulesets wholesale.
We're SOC2 + HIPAA compliant, which either means convincing the auditor that our in-house security rules cover 100% of the cases they care about... or we buy an off-the-shelf WAF that has already completed the compliance process, and call it a day. The CTO is going to pick the second option every time.
mjr00
Yeah. SOC2 reminds me that I didn't mention sales as well, another security-as-economics feature. I've seen a lot of enterprise RFPs that mandate certain security protocols, some of which are perfectly sensible and others... not so much. Usually this is less problematic than insurance because the buyer is more flexible, but sometimes they (specifically, the buyer's company's security team, who has no interest besides covering their own ass) refuse to budge.
If your startup is on the verge of getting a 6 figure MRR deal with a company, but the company's security team mandates you put in a WAF to "protect their data"... guess you're putting in a WAF, like it or not.
sgarland
OS-level monitoring / auditing software also never ceases to amaze me (for how awful it is). Multiple times, at multiple companies, I have seen incidents that were caused because Security installed / enabled something (AWS GuardDuty, Auditbeat, CrowdStrike…) that tanked performance. My current place has the latter two on our ProxySQL EC2 nodes. Auditbeat is consuming two logical cores on its own. I haven’t been able to yet quantify the impact of CrowdStrike, but from a recent perf report, it seemed like it was using eBPF to hook into every TCP connection, which is quite a lot for a DB connection poolers.
I understand the need for security tooling, but I don’t think companies often consider the huge performance impact these tools add.
simonw
I wish IT teams would say "sorry about the password requirement, it's required by our insurance policy". I'd feel a lot less angry about stupid password expiration rules if they told me that.
cratermoon
Sometime in the past few years I saw a new wrinkle: password must be changed every 90 days unless it is above a minimum length (12 or so as best I recall) in which case you only need to change it yearly. Since the industry has realized length trumps dumb "complexity" checks, it's a welcome change to see that encoded into policy.
thayne
Not exactly that, but I've had the security team say "sorry about the password policy, we agree it is stupid and counterproductive, but it's required for compliance with X, and we need X to sell to some big customers."
betaby
> but a lot of seemingly dumb security policies are due to insurers.
I keep hearing that often on HN, however I've personally never seen seen such demands from insurers. I would greatly appreciate if one share such insurance policy. Insurance policies are not trade secrets and OK to be public. I can google plenty of commercial cars insurance policies for example.
simonw
I found an example!
https://retail.direct.zurich.ch/resources/definition/product...
Questionnaire Zurich Cyber Insurance
Question 4.2: "Do you have a technically enforced password policy that ensures use of strong passwords and that passwords are changed at least quarterly?"
Since this is an insurance questionnaire, presumably your answers to that question affect the rates you get charged?
(Found that with the help of o4-mini https://chatgpt.com/share/680bc054-77d8-8006-88a1-a6928ab99a...)
bigbuppo
The fun part is that they don't demand anything, they just send you a worksheet that you fill out and presumably it impacts your rates. You just assume that whatever they ask about is what they want. Some of what they suggest is reasonable, like having backups that aren't stored on storage directly coupled to your main environment.
The worst part about cyber insurance, though, is that as soon as you declare an incident, your computers and cloud accounts now belong to the insurance company until they have their chosen people rummage through everything. Your restoration process is now going to run on their schedule. In other words, the reason the recovery from a crypto-locker attack takes three weeks is because of cyber insurance. And to be fair, they should only have to pay out once for a single incident, so their designated experts get to be careful and meticulous.
tmpz22
This is such an important comment.
Fear of a prospective expectation, compliance, requirement, etc., even when that requirement does not actually exist is so prevalent in the personality types of software developers.
manwe150
You can buy insurance for just about anything, not just cars. Companies frequently buy insurance against various low-probability incidents such as loss of use, fraud, lawsuit, etc.
wvh
Having worked with PCI-DSS, some rules seem to only exist to appease insurance. When criticising decisions, you are told that passing audits to be able to claim insurance is the whole game, even when you can demonstrate how you can bypass certain rules in reality. High-level security has more to do with politics (my definition) than purely technical ability. I wouldn't go as far as to call it security theatre, there's too much good stuff there that many don't think about without having a handy list, but the game is certainly a lot bigger than just technical skills and hacker vs anti-hacker.
I still have a nervous tick from having a screen lock timeout "smaller than or equal to 30 seconds".
lucianbr
There should be some limits and some consequences to the insurer as well. I don't think the insurer is god and should be able to request anything no matter if it makes sense or not and have people and companies comply.
If anything, I think this attitude is part of the problem. Management, IT security, insurers, governing bodies, they all just impose rules with (sometimes, too often) zero regard for consequences to anyone else. If no pushback mechanism exists against insurer requirements, something is broken.
mjr00
> There should be some limits and some consequences to the insurer as well. I don't think the insurer is god and should be able to request anything no matter if it makes sense or not and have people and companies comply.
If the insurer requested something unreasonable, you'd go to a different insurer. It's a competitive market after all. But most of the complaints about incompetent security practices boil down to minor nuisances in the grand scheme of things. Forced password changes once every 90 days is dumb and slightly annoying but doesn't significantly impact business operations. Having to run some "enterprise security tool" and go through every false positive result (of which there will be many) and provide an explanation as to why it's a false positive is incredibly annoying and doesn't help your security, but it's also something you could have a $50k/year security intern do. Turning on a WAF that happens to reject the 0.0001% of Substack articles which talk about /etc/hosts isn't going to materially change Substack's revenue this year.
jimmaswell
This is why everyone should have a union, including highly paid professionals. Imagine what it would be like. "No, fuck you, we're going on strike until you stop inconveniencing us to death with your braindead security theater. No more code until you give us admin on our own machines, stop wasting our time with useless Checkmarx scans, and bring the firewall down about ten notches."
II2II
> If an insurer says "we're going to jack up premiums by 20% unless you force employees to change their password once every 90 days", you can argue till you're blue in the face that it's bad practice, NIST changed its policy to recommend not regularly rotating passwords over a decade ago, etc., and be totally correct... but they're still going to jack up premiums if you don't do it.
I would argue that password policies are very context dependent. As much as I detest changing my password every 90 days, I've worked in places where the culture encouraged password sharing. That sharing creates a whole slew of problems. On top of that, removing the requirement to change passwords every 90 days would encourage very few people to select secure passwords, mostly because they prefer convenience and do not understand the risks.
If you are dealing with an externally facing service where people are willing to choose secure passwords and unwilling to share them, I would agree that regularly changing passwords creates more problems than it solves.
Aeolun
> removing the requirement to change passwords every 90 days would encourage very few people to select secure passwords
When you don’t require them to change it, you can just assign them a random 16 character string and tell them it’s their job to memorize it.
620gelato
> jack up premiums by 20% unless you force employees to change their password once every 90 days"
Always made me judge my company's security teams as to why they enable this stupidity. Thankfully they got rid of this gradually, nearly 2 years ago now (90 days to 365 days to never). New passwords were just one key l/r/u/d on the keyboard.
Now I'm thinking maybe this is why the app for a govt savings scheme in my country won't allow password autofill at all. Imagine expecting a new password every 90 days and not allowing auto fill - that just makes passwords worse.
smeg_it
I'm no expert, but I did take a CISSP course a while ago. One thing I actually remember ;P, is that it recommended long passwords in in lieu of the number, special character, upper, lower ... I don't remember the exact wording of course and maybe it did recommend some of that, but it talked about having a sentence rather than all that mess in 6-8 characters, but many sites still want the short mess that I never will actually remember
vlovich123
While the password recommendation stuff is changing (the US government updating it guidelines last year), it’s generally best practice to not share passwords which itself implies using a password manager anyway which makes the whole “long passphrase” vs “complex” password moot - just generate 32 lowercase random characters to make it easier to type or use the autogenerated password your password manager recommends.
The long passphrase is more for the key that unlocks your password manager rather than the random passwords you use day to day.
mcoliver
entropy is stronger than complexity. https://xkcd.com/936/
paxys
"You never know..." is the worst form of security, and makes systems less secure overall. Passwords must be changed every month, just to be safe. They must be 20 alphanumeric characters (with 5 symbols of course), just to be safe. We must pass every 3-letter compliance standard with hundreds of pages of checklists for each. The server must have WAF enabled, because one of the checklists says so.
Ask the CIO what actual threat all this is preventing, and you'll get blank stares.
As an engineer what incentive is there to put effort into knowing where each form input goes and how to sanitize it in a way that makes sense? You are getting paid to check the box and move on, and every new hire quickly realizes that. Organizations like these aren't focused on improving security, they are focused on covering their ass after the breach happens.
chii
> Ask the CIO what actual threat all this is preventing
the CIO is securing his job.
reaperducer
the CIO is securing his job.
Every CIO I have worked for (where n=3) has gotten where they are because they're a good manager, even though they have near-zero current technical knowledge.
The fetishizing of "business," in part through MBAs, has been detrimental to actually getting things done.
A century ago, if someone asked you what you do and you replied, "I'm a businessman. I have a degree in business," you'd get a response somewhere between "Yeah, but what to you actually do" and outright laughter.
null
ryandrake
This looks like a variation of the Scunthorpe problem[1], where a filter is applied too naively, aggressively, and in this case, to the wrong content altogether. Applying the filter to "other stuff" sent to and among the servers might make sense, but there doesn't seem to be any security benefit to filtering actual text payload that's only going to be displayed as blog content. This seems like a pretty cut and dried bug to me.
rurp
This is exactly what I was thinking as well, it's a great Scunthorpe example. Nothing from the body of a user article should ever be executed in any way. If blocking a list of strings is providing any security at all you're already in trouble because attackers will find a way around that specific block list.
chrisjj
> This looks like a variation of the Scunthorpe problem[1], where a filter is applied too naively
No.
> aggressively
No.
>, and in this case, to the wrong content altogether.
Yes - making it not a Scunthorpe problem.
pmarreck
Correct. And a great example of it.
krferriter
I don't get why you'd have SQL injection filtering of input fields at the CDN level. Or any validation of input fields aside from length or maybe some simple type validation (number, date, etc). Your backend should be able to handle arbitrary byte content in input fields. Your backend shouldn't be vulnerable to SQL injection if not for a CDN layer that's doing pre-filtering.
immibis
Because someone said "we need security" and someone else said "what is security" and someone else said "SQL injection is security" and someone looked up SQL injections and saw the word "select" and "insert".
WAFs are always a bad idea (possible exception: in allow-but-audit mode). If you knew the vulnerabilities you'd protect against them in your application. If you don't know the vulnerabilities all you get is a fuzzy feeling that Someone Else is Taking Care of it, meanwhile the vulnerabilities are still there.
Maybe that's what companies pay for? The feeling?
pxc
WAFs can be a useful site of intervention during incidents or when high-severity vulns are first made public. It's not a replacement for fixing the vuln, that still has to happen, but it gives you a place to mitigate it that may be faster or simpler than deploying code changes.
patrakov
> If you knew the vulnerabilities you'd protect against them in your application.
Correction: it is not your application but someone else's Certified Stuff (TM) that you can't change, but which is still vulnerable.
gopher_space
If your clients will let you pass the buck on security like this it would be very tempting to work towards the least onerous insurance metric and no further.
ordersofmag
A simple reason would be if you're just using it as a proxy signal for bad bots and you want to reduce the load on your real servers and let them get rejected at the CDN level. Obvious SQL injection attempt = must be malicious bot = I don't want my servers wasting their time
chrisjj
> A simple reason would be if you're just using it as a proxy signal for bad bots
Who would be that stupid?
benregenspan
It should be thought of as defense-in-depth only. The backend had better be immune to SQL injection, but what if someone (whether in-house or vendor) messes that up?
I do wish it were possible to write the rules in a more context-sensitive way, maybe possible with some standards around payloads (if the WAF knows that an endpoint is accepting a specific structured format, and how escapes work in that format, it could relax accordingly). But that's probably a pipe dream. Since the backend could be doing anything, paranoid rulesets have to treat even escaped data as a potential issue and it's up to users to poke holes.
nijave
The farther a request makes it into infrastructure, the more resources it uses.
kiitos
> I disagree with other posts here, it is partially a balance between security and usability. You never know what service was implemented with possible security exploits and being able to throw every WAF rule on top of your service does keep it more secure. Its just that those same rulesets are super annoying when you have a securely implemented service which needs to discuss technical concepts.
I might be out of the loop here, but it seems to me that any WAF that's triggered when the string "/etc/hosts" is literally anywhere in the content of a requested resource, is pretty obviously broken.
schnable
I don't think so. This rule for example probably block attacks on a dozen old WordPress vulnerabilities.
kiitos
And a rule that denies everything blocks all vulnerabilities entirely.
A false positive from a conservative evaluation of a query parameter or header value is one thing, conceivably understandable. A false positive due to the content of a blog post is something else altogether.
stingraycharles
Yup. Were a database company that needs to be compliant with SOC2, and I’ve had extremely long and tiring arguments with our auditor why we couldn’t adhere to some of these standard WAF rulesets because it broke our site (we allow people to spin up a demo env and trigger queries).
We changed auditors after that.
spydum
sounds like your security policy is wrong (or doesnt have a provision for exceptions managed by someone with authority to grant them), or your auditor was swerving out of his lane. As far as I've seen: SOC2 doesn't describe any hard security controls - it just asks to evaluate your policy versus your implemented controls.
stingraycharles
You are absolutely correct, which is why we switched auditors. We use a third party to verify compliance of all our cloud resources (SecureFrame), and one of their checks is that specific AWS WAF rulesets are enabled on e.g. CloudFront endpoints. These are managed rulesets by AWS.
We disabled this check, auditor swerved out of his lane, I spent more several hours explaining things he didn’t understand, and things resolved after our CEO had a call with him (you can imagine how the discussion went).
All in all, if the auditor would have been more reasonable it wouldn’t have been an issue, but I’ve always been wary of managed firewall rulesets because of this reason.
RKFADU_UOFCCLEL
There's no "trade-off" here. Blocking IPs that send "1337 h4x0r buzzword /etc/passwd" in it is completely naive and obtrusive, which is the modus operandi of the CDN being discussed here. There are plenty of other ways of hosting a website.
julik
This is what surprises me in this story. I could not, at first glance, assume that either Substack people or Cloudflare people were incompetent.
Oh: I resisted tooth and nail about turning on a WAF at one of my gigs (there was no strict requirement for it, just cargo cult). Turns out - I was right.
netsharc
Reminds me of an anecdote about an e-commerce platform: someone coded a leaky webshop, so their workaround was to watch if the string "OutOfMemoryException" shows up in the logs, and then restart the app.
Another developer in the team decided they wanted to log what customers searched for, so if someone typed in "OutOfMemoryException" in the search bar...
PhilipRoman
Careless analysis of free-form text logs is an underrated way to exploit systems. It's scary how much software blindly logs data without out of band escaping or sanitizing.
ycombinatrix
Why would someone "sanitize" OutOfMemoryException out of their logs? That is a silly point to make.
teraflop
The point is not to sanitize known strings like "OutOfMemoryException". The point is to sanitize or (preferably) escape any untrusted data that gets logged, so that it won't be confused for something else.
owebmaster
An OutOfMemoryException log should not be the same as a search log
Error: OutOfMemoryException
Search: OutOfMemoryException
MortyWaves
Absolutely incredible how dense HN can be and that no one has explained. Obviously that isn’t what they are saying, they are saying it’s profoundly stupid to have the server be controlled by a simple string search at all.
skipants
I've actually gone through this a few times with our WAF. A user got IP-banned because the WAF thought a note with the string "system(..." was PHP injection.
Y_Y
Does it block `/etc//hosts` or `/etc/./hosts`? This is a ridiculous kind of whack-a-mole that's doomed to failure. The people who wrote these should realize that hackers are smarter and more determined than they are and you should only rely on proven security, like not executing untrusted input.
jrockway
Yeah, and this seems like a common Fortune 500 mandatory checkbox. Gotta have a Web Application Firewall! Doesn't matter what the rules are, as long as there are a few. Once I was told I needed one to prevent SQL injection attacks... against an application that didn't use an SQL database.
If you push back you'll always get a lecture on "defense in depth", and then they really look at you like you're crazy when you suggest that it's more effective to get up, tap your desk once, and spin around in a circle three times every Thursday morning. I don't know... I do this every Thursday and I've never been hacked. Defense in depth, right? It can't hurt...
hnlmorg
I’m going through exactly this joy with a client right now.
“We need SQL injection rules in the WAF”
“But we don’t have an SQL database”
“But we need to protect against the possibility of partnering with another company that needs to use the same datasets and wants to import them into a SQL database”
In fairness, these people are just trying to do their job too. They get told by NIST (et al) and Cloud service providers that WAF is best practice. So it’s no wonder they’d trust these snake oil salesman over the developers who asking not to do something “security” related.
zelphirkalt
If they want to do their job well, how about adding some thinking into the mix, for good measure? Good would also be,if they actually knew what they are talking about, before trying to tell the engineers what to do.
bombcar
I love that having a web application firewall set to allow EVERYTHING passes the checkbox requirement ...
CoffeeOnWrite
(I’m in the anti-WAF camp) That does stand to improve your posture by giving you the ability to quickly apply duct tape to mitigate an active mild denial of service attack. It’s not utterly useless.
vultour
A large investment bank I worked for blocked every URL that ended in `.go`. Considering I mostly wrote Golang code it was somewhat frustrating.
nickdothutton
See "enumerating badness" as a losing strategy. I knew this was a bad idea about 5 minutes after starting my first job in 1995.
tom1337
Well I've just created an account on substack to test this but turns out they've already fixed the issue (or turned off their WAF completely)
augusto-moura
How would that be hard? Getting the absolute path of a string is in almost all languages stdlibs[1]. You can just grep for any string containing slashes and try resolve them and voilá
Resolving wildcards is trickier but definitely possible if you have a list of forbidden files
[1]: https://nodejs.org/api/path.html#pathresolvepaths
Edit: changed link because C's realpath has a slightly different behavior
TheDong
The reason it's doomed to failure is because WAFs operate before your application, and don't have any clue what the data is.
Here is a WAF matching line: https://github.com/coreruleset/coreruleset/blob/943a6216edea...
Here's where that file is loaded: https://github.com/coreruleset/coreruleset/blob/943a6216edea...
It's loaded with '"@pmFromFile lfi-os-files.data"' which means "case-insensitive match of values from a file".
So yeah, the reason it can't resolve paths properly is because WAFs are just regex and substring matching trying to paper over security issues in an application which can only be solved correctly at the application level.
myflash13
It’s not hard, but I think that’s more computation than a CDN should be doing on the edge. If your CDN layer is doing path resolution on all strings with slashes, that’s already some heavy lifting for a proxy layer.
watusername
> How would that be hard? Getting the absolute path of a string is in almost all languages stdlibs[1]. You can just grep for any string containing slashes and try resolve them and voilá
Be very, very careful about this, because if you aren't, this can actually result in platform-dependent behavior or actual filesystem access. They are bytes containing funny slashes and dots, so process them as such.
Edit: s/text/bytes/
eli
Is a security solution worthless if it can't stop a dedicated attacker? A lot of WAF rules are blocking probes from off-the-shelf vulnerability scanners.
da_chicken
"It's technically better than nothing," is kind of a bizarre metric.
It's like not allowing the filesystem to use the word "virus" in a file name. Yes, it technically protects against some viruses, but it's really not very difficult to avoid while being a significant problem to a fair number of users with a legitimate use case.
It's not that it's useless. It's that it's stupid.
eli
Do you lock your front door?
ndsipa_pomu
It's merely security theater.
It reminds me of when airports started scanning people's shoes because an attacker had used a shoe bomb. Yes, that'll stop an attacker trying a shoe bomb again, but it disadvantages every traveller and attackers know to put explosives elsewhere.
geoffpado
“attacker had used a shoe bomb”
It’s even dumber than that. An attacker tried and failed to use a shoe bomb, and yet his failure has caused untold hours of useless delay for over 13 years now.
eli
Most ransomware attacks are opportunistic. They scan basically the whole internet for vulnerabilities and attack from there. It's usually not a skilled attacker targeting a specific company.
Ransomware is a huge and growing problem. Very different than airline security, where attacks are extremely uncommon. If planes were constantly getting blown up, and if a majority of those attacks started with a shoe bomb, then checking everyone's shoes would seem a lot more reasonable, no?
kevincox
IMHO the primary value for WAFs is for quickly blocking known vulnerabilities with specific rules to mitigate vulnerabilities while they are being properly patched. Ideally the WAF knows what software is behind it (example WordPress, Java app, ...) and can apply filters that may be relevant.
Anything else is just a fuzzy bug injector that will only stop the simplest scanners and script kiddies if you are lucky.
richardwhiuk
Every security solution can only stop a certain fraction of attacks.
mystifyingpoi
No one expects any WAF to be a 100% solution that catches all exfiltration attempts ever, and it should not be treated this way. But having it is generally better than not having it.
Macha
> But having it is generally better than not having it.
The problem is that generally you're breaking actual valid use cases as the tradeoff to being another layer of defense against hypothetical vulnerabilities.
Yes, discussing the hosts file is a valid use case.
Yes putting angle brackets in the title of your message is valid use case your users are going to want.
Yes putting "mismatched" single quotes inside double quotes is a thing users will do.
Yes your users are going to use backslashes and omit spaces in a way that looks like attempts at escaping characters.
(All real problems I've seen caused by overzealous security products)
simonw
"But having it is generally better than not having it."
I believe the exact opposite.
One (of many) reasons is that it can make your code less secure, by hiding your security mistakes from you.
If your WAF obscures escaping issues during your own testing and usage you could very easily let those escaping issues go unresolved - leaving you vulnerable to any creative attacker who can outsmart your WAF.
RamRodification
If you are in charge of testing code for escaping issues, and you do that through a WAF, you might not be very good at your job.
paxys
> But having it is generally better than not having it.
So is HN and every other site in the world insecure because it allows users to post "/etc/hosts" ?
null
null
mystifyingpoi
Maybe? I don't know nor care. Assuming that HN has a vuln with path traversal, a sanely configured WAF would block the traversal attempt.
rcxdude
Is it? The WAF is also now an attack surface itself, and I don't think WAFs have exactly proven themselves as something that meaningfully increases security. They certainly break things unpredictably, though.
wyager
> But having it is generally better than not having it.
Why? It obviously has an annoying cost and equally obviously won't stop any hacker with a lukewarm IQ
wavemode
No, that logic doesn't follow. If your application is so hopelessly vulnerable as to benefit from such naive filtering of the text "/etc/hosts, then your application is still going to be vulnerable in precisely the same ways, with just slightly modified inputs.
It is net zero for security and net negative for user experience, so having it is worse than not having it.
serial_dev
Net zero for security might be generous.
The way I assume it works in practice on a real team is that after some time, most of your team will have no idea how the WAF works and what it protects against, where and how it is configured… but they know it exists, so they will no longer pay attention to security because “we have a tool for that”, especially when they should have finished that feature a week ago…
simonw
"How could Substack improve this situation for technical writers?"
How about this: don't run a dumb as rocks Web Application Firewall on an endpoint where people are editing articles that could be about any topic, including discussing the kind of strings that might trigger a dumb as rocks WAF.
This is like when forums about web development implement XSS filters that prevent their members from talking about XSS!
Learn to escape content properly instead.
awoimbee
I'm in the position where I have to run a WAF to pass security certifications. The only open source WAFs are modsecurity and it's beta successor, coraza. These things are dumb, they just use OWASP's coreruleset which is a big pile of unreadable garbage.
serial_dev
Surprisingly simple solution
ZeroTalent
hire a cybersec person. I don't think they one.
blenderob
> This case highlights an interesting tension in web security: the balance between protection and usability.
But it doesn't. This case highlights a bug, a stupid bug. This case highlights that people who should know better, don't!
The tension between security and usability is real but this is not it. Tension between security and usability is usually a tradeoff. When you implement good security that inconveniences the user. From simple things like 2FA to locking out the user after 3 failed attempts. Rate limiting to prevent DoS. It's a tradeoff. You increase security to degrade user experience. Or you decrease security to increase user experience.
This is neither. This is both bad security and bad user experience. What's the tension?
myflash13
I would say it’s a useful security practice in general to apply WAF as a blanket rule to all endpoints and then remove it selectively when issues like this occur. It’s much, much, harder to evaluate every single public facing endpoint especially when hosting third party software like Wordpress with plugins.
crabbone
Precisely.
This also reminded me, I think in the PHP 3 era, PHP used to "sanitize" the contents of URL requests to blanket combat SQL injections, or perhaps, it was a configuration setting that would be frequently turned on in shared hosting services. This, of course, would've been very soon discovered by the authors of the PHP site and various techniques were employed to circumvent this restriction, overall giving probably even worse outcomes than if the "sanitation" wasn't there to begin with.
TRiG_Ireland
The days of addslashes() and stripslashes()!
SonOfLilit
After having been bitten once (was teaching a competitive programming team, half the class got a blank page when submitting solutions, after an hour of debugging I narrowed it down to a few C++ types and keywords that cause 403 if they appear in the code, all of which happen to have meaning in Javascript), and again (working for a bank, we had an API that you're supposed to submit a python file to, and most python files would result in 403 but short ones wouldn't... a few hours of debugging and I narrowed it down to a keyword that sometimes appears in the code) and then again a few months later (same thing, new cloud environment, few hours burned on debugging[1]), I had the solution to his problem in mind _immediately_ when I saw the words "network error".
[1] the second time it happened, a colleague added "if we got 403, print "HAHAHA YOU'VE BEEN WAFFED" to our deployment script, and for that I am forever thankful because I saw that error more times than I expected
simonw
Do you remember if that was Cloudflare or some other likely WAF?
SonOfLilit
First time something on-prem, maybe F5. Second time AWS.
Oh, I just remembered I had another encounter with the AWS WAF.
I had a Jenkins instance in our cloud account that I was trying to integrate with VSTS (imagine github except developed by Microsoft, and still maintained, nevermind that they own github and it's undoubtedly a better product). Whenever I tried to trigger a build, it worked, but when VSTS did, it failed. Using a REST monitor service I was able to record the exact requests VSTS was making and prove that they work with curl from my machine... after a few nights of experimenting and diffing I noticed a difference between the request VSTS made to the REST monitor and my reproduction with curl: VSTS didn't send a "User-Agent" header, so curl supplied one by default unless I added I think -H "User-Agent:", and therefore did not trigger the first default rule in the AWS WAF, "if your request doesn't list a user agent you're a hacker".
HAHAHA I'VE BEEN WAFFED AGAIN.
netsharc
+++ATH
pimanrules
We faced a similar issue in our application. Our internal Red Team was publishing data with XSS and other injection attack attempts. The attacks themselves didn't work, but the presence of these entries caused our internal admin page to stop loading because our corporate firewall was blocking the network requests with those payloads in them. So an unsuccessful XSS attack became an effective DoS attack instead.
darkwater
This is funny and sad at the same time.
mrgoldenbrown
Everything old is new again :) We used to call this the Scunthorpe problem.
kreddor
I remember back in the old days on the Eve Online forums when the word cockpit would always turn up as "c***pit". I was quite amused by that.
Too
This is actually a better solution, replacing dangerous words with placeholders, instead of blocking the whole payload. That at least gives the user some indication of what is going on. Not that I'm for any such WAF filters in the first place, just if having to choose between the lesser of two evils I'd choose the more informative.
Matumio
Not so sure. Imagine you have a base64 encoded payload and it just happens to encode the forbidden word. Good luck debugging that, if the payload only gets silently modified.
I suddenly understand why it makes sense to integrity-check a payload that is already protected by all three of TLS, TCP checksum and CRC.
greghendershott
See also: Recent scrubbing US government web sites for words like "diversity", "equity", and "inclusion".
Writing about biology, finance, or geology? Shrug.
Dumb filtering is bad enough when used by smart people with good intent.
netsharc
Huh, quick tell one Musk's DOGE l33t h4ck3ers about reverse proxies, and put all government sites behind one, that looks for those words and returns an error... Error 451 would be the most appropriate!
For bonus, the reverse proxy will run on a system infiltrated by Russian (why not Chinese as well) hackers.
odirf
It is time to add the Substack case to this Wikipedia article.
reverendsteveii
"I wonder why it's called Scunthorpe....?"
sits quietly for a second
"Oh nnnnnnnooooooooooooooo lol!"
petercooper
I ran into a similar issue with OpenRouter last night. OpenRouter is a “switchboard” style service that provides a single endpoint from which you can use many different LLMs. It’s great, but last night I started to try using it to see what models are good at processing raw HTML in various ways.
It turns out OpenRouter’s API is protected by Cloudflare and something about specific raw chunks of HTML and JavaScript in the POST request body cause it to block many, though not all, requests. Going direct to OpenAI or Anthropic with the same prompts is fine. I wouldn’t mind but these are billable requests to commercial models and not OpenRouter’s free models (which I expect to be heavily protected from abuse).
esafak
Did you report it?
petercooper
Not yet, other than on X, because the project's comms is oriented around Discord which involves some hoop jumping.
(Update: On the way to doing that, I decided to run my tests again and they now work without Cloudflare being touchy, so I'll keep an eye on it!)
(Update 2: They just replied to me on X and said they had fixed their Cloudflare config - happy days!)
jmmv
I encountered this a while ago and it was incredibly frustrating. The "Network error" prevented me from updating a post I had written for months because I couldn't figure out why my edits (which extended the length and which I assumed was the problem) couldn't get through.
Trying to contact support was difficult too due to AI chatbots, but when I finally did reach a human, their "tech support" obviously didn't bother to look at this in any reasonable timeframe.
It wasn't until some random person on Twitter suggested the possibility of some magic string tripping over some stupid security logic that I found the problem and could finally edit my post.
robertlagrant
> This case highlights an interesting tension in web security: the balance between protection and usability.
This isn't a tension. This rule should not be applied at the WAF level. It doesn't know that this field is safe from $whatever injection attacks. But the substack backend does. Remove the rule from the WAF (and add it to the backend, where it belongs) and you are just as secure and much more usable. No tension.
myflash13
I would say it’s a decent security practice to apply WAF as a blanket rule to all endpoints and then remove it selectively when issues like this occur. It’s much, much, harder to evaluate every single public facing endpoint especially when hosting third party software like Wordpress with plugins.
SonOfLilit
I don't agree. WAFs usually add more attack surface than they remove.
https://www.macchaffee.com/blog/2023/wafs/
Of course, Wordpress is basically undefendable, so I'd never ever host it on a machine that has anything else of value (including e.g. db credentials that give access to much more than the public content on the WP installation).
worewood
There is a tension, but it's between paying enough to developers to actually produce decent code or pay a 3rd-party to firewall the application.
marcosdumay
Again, there is no tension.
People will manage to circumvent the firewall if they want to attack your site. But you will still pay, and get both the DoS vulnerabilities created by the firewall and the new attack vectors in the firewall itself.
josephcsible
WAFs were created by people who read https://thedailywtf.com/articles/Injection_Rejection and didn't realize that TDWTF isn't a collection of best practices.
The people configuring WAF rules at CDNs tend to do a poor job understanding sites and services that discuss technical content. It's not just Cloudflare, Akamai has the same problem.
If your site discusses databases then turning on the default SQL injection attack prevention rules will break your site. And there is another ruleset for file inclusion where things like /etc/hosts and /etc/passwd get blocked.
I disagree with other posts here, it is partially a balance between security and usability. You never know what service was implemented with possible security exploits and being able to throw every WAF rule on top of your service does keep it more secure. Its just that those same rulesets are super annoying when you have a securely implemented service which needs to discuss technical concepts.
Fine tuning the rules is time consuming. You often have to just completely turn off the ruleset because when you try to keep the ruleset on and allow the use-case there are a ton of changes you need to get implemented (if its even possible). Page won't load because /etc/hosts was in a query param? Okay, now that you've fixed that, all the XHR included resources won't load because /etc/hosts is included in the referrer. Now that that's fixed things still won't work because some random JS analytics lib put the URL visited in a cookie, etc, etc... There is a temptation to just turn the rules off.