Let's Not Encrypt (2019)

I don't know what the official name for the phenomena is where you widely experience a huge problem, the market reacts to fix the problem almost completely, people who never experienced the problem in the first place because the world before them solved it begin to complain about the solution, people defending the solution are mocked by people who have no context, the solution is rolled back and all the people who did it are happy with their win for a brief moment in time, then the original problem comes back in force but all of the walls put up to tear down the original solution make it 1000x harder to fix.

I feel like there needs to be a name for this. For now, "Those who do not learn from history are doomed to repeat it." is the most apt I think.

Happens constantly when you're essentially born on 3rd base. Maybe that's the proper name. Born on 3rd Base Syndrome.

> suddenly things started breaking. We eventually tracked it down

Amateur level ... Around 2006, we enjoyed some clients complaining why information on our CMS was being duplicated.

No matter what we did, there was no duplication on our end. So we started to trace the actions from the from the client (inc browser, ip etc). And low and behold, we got one action coming from the client, and another from a different IP source.

After tracing back the IP, it was a anti virus company. We installed the software on a test system, and ... Yep, the assh** duplicated every action, inc browser settings, session, you name it.

Total and complete mimic beyond the IP. So any action the user did + the information of the page, was send to their servers for "analyzing".

Little issue ... This was not from the public part of our CMS but the HTTPS protected admin pages!

Sure, our fault for not validating the session with extra IP checks but we did not expect the (admin only) session to leak out from a HTTPS connection.

So we tried to see if they reacted to login attempts at several bank pages. O, yes, they send the freaking passwords etc. We tried on a unused bank account, o, look, it was duplicating bank actions (again, bank at fault for not properly checking the session / ip).

It only failed on a bank transfer because the token for authorization was different on their side, vs our request.

You can imagine that we had a rather, how to say, less then polite talks / conversation with the software team behind that anti-virus. They "fixed it" in a new release. Did they remove the whole tracking? Nowp, they just removed the code for the session stealing if the connection was secure.

O, and the answer to why they did it. "it a bug" (yea, right, your mimic a total user behavior, and its a "bug"). Translation: Legal got up their behinds for that crap and they wanted to avoid legal issues with what they did.

Remember folks, if its free your the product. And when its paid, you are often STILL the product. And yes, that was a paid anti-virus "online protection". And people question why i never run any anti-virus software beyond a off-line scan from time to time, and have Windows "online" protections disabled.

Companies just can not stop themselves from being greedy. Same reason why i NEVER use Windows 11... You expect if you paid for Windows, Office or whatever, to not be the product, but hey ...

Haha, yeah this kind of stuff made HTTP long polling requests over mobile pretty fun. IIRC, we ran HTTP over IMAP and POP3 ports for cases where port 80 was unreliable.

My ISP (Mediacom) appears to have a deal with certain websites to display service messages. The only two I've encountered it on is Amazon and Facebook but they are somehow able to insert a maintenance banner at the top of those two when downtime is anticipated or if I am near the end of my bandwidth quota. Haven't gotten any ads this way but they have the technology.

This was common back in the early 2010s with Indian ISPs as well, particularly the state-controlled BSNL.

Our app reports all of the runtime exceptions to the server. We had one years ago (maybe before 2008) that was caused by somebody's "toolbar" replacing a method like Element.appendChild with one that sometimes crashed.

This inspired me to add a list of all script tags to error reports.

The modern version of that is brave or ublock or screen reader extensions or spyware inserting JS or data attributes which leads to user complaints. We don't need ISPs hacking lines, people do it to themselves when they sign up to shady sms services on download sites.

And who now uses metro wifi «от оленевода»? Noone

We can do that and also use HTTPS to be more certain of it.

They can also just tell some CA to sign a certificate.

LE checks from multiple places, so you'd have to MITM all of them, which makes it seem rather challenging to actually pull off.

TOFU would allow the ISP to MITM every connection and then serve you ads. The ISP could simply provide their own cert to you.

> Simply using TOFU-certificates (Trust On First Use) would achieve this.

As a user how would I know if I should trust the website's public key on first use?

> using http-only for decades, never seen “injections”

This has to be a rage bait comment, but anyway, how do you expect 'injections' to show up on 'http-only' ?

"Don't mind us, we're just sitting in the middle of your traffic here and recording your logins in plaintext"

> 2. just change ISP

Not a viable option in a lot of places. Nor does anyone really even want to consider this possibility of their ISP being able to MITM something in the first place.

> just change ISPs

I sure love when decisions reduce themselves to single points of consideration by virtue of them being discussed in a heated internet forum thread

The problem with the horrible injections on the page can be solved very simply. If the information on a page is open, just pass that page openly and pass a checksum of page in the header. To prevent this sum from being tampered with, the server will encrypt it. Not the whole page, just the sum. You will save a lot of CPU time on server and on client, reduce CO₂ and so on

> Does it even work great for nerds?

No, but I was extending a charitable amount of credulousness :-)

Even amongst nerds I've seen a significant amount of key pair re-use in my time, both 1:n::dev:servers and sometimes even 1:n::organization:devs. The transport security is moot when the user(s) discard all precautions and best practices on either end.

Please let's not break something that works really well just to cater to those who don't know how to use the tools of their trade.

The platform engineering team at my big corp work simply disabled host key checking in the cloud tool Python script they wrote for all of us to log into our bastion hosts.

For prod.

ssh —-known-hosts-file=/dev/null

I think it's pretty reasonable to turn off the "yes i would like to accept this key" on first connect. Just scream if it ever changes. I get that they're expecting me to compare it to something out of band but nobody does that.

This is the sleight of hand being employed when folks suggest TOFU mechanisms. The problem with any communication boils down to trust. The modern web PKI has a bunch of complexity and a plenty of rough edges in how it handles resolving that trust. TOFU is then proposed as a simpler solution with none of those pesky rough edges, but it doesn't have the rough edges because it leaves all the hard parts as an exercise for the reader.

It's a bit like suggesting that AES-GCM has risks so we ought to just switch to one-time-pads.

> How do I make sure that if I should be trusting the website public key on first use? It doesn't seem like any easier to solve than PKI.

Usually such questions get replied to with a recommendation of implementing DNSSEC. Which is also obviously PKI and in many ways worse than WebPKI.

It's the usual hilarious flow of "HTTPS is dogshit, so here's the SSH fingerprint you should trust instead, served over HTTPS of course".

Well, I do! And Google doesn't get paid!

> otherwise they drop certificates from Chrome and this has happened.

As far as I know, all the CAs Google dropped, this was because the CA misbehaved and misissued certs or was obviously failing at their job. Also, all CAs google has removed from their root store have also been removed by mozilla (or weren't removed because mozilla never included them).

You're thinking of the CAB, which dictates which CAs are trusted. Google is a participant in that. The things they dictate are public and have to do with security requirements, not whether or not they pay Google money.

Also, if the CA runs the ACME check from five different validation servers that aren't all on the same continent, which Let's Encrypt does and all other CAs will be required to do in a couple years, then it is dramatically harder to simultaneously MITM them all. And if you really want to, you can use DNS-01 with DNSSEC, which means an attacker would have to be able to compromise DNSSEC on top of everything else.

> Web sites that do not monitor CT logs probably are vulnerable to well resourced attacks of this kind

How many web site owner really do that? I mean, even Cloudflare hasn't been running a tight ship in this regard[0] until recently.

[0]: https://blog.cloudflare.com/unauthorized-issuance-of-certifi...

They need to install their root certificate into your work machine's trust store. Which they can only do because they control the machine (or VPN software), and would not be possible for a regular machine.

This only works because you company's endpoints have been configured to trust the company's root CA. Which makes sense, because it's their device and their VPN.

I wonder if you could roll your own VPN tunnel that directly connects to your home internet IP and passes custom encrypted payload that your IT department cannot decode. Would they just drop the connection if they can't inspect what your are sending?

Yes, but if you use HSTS a regular browser will flag that. Perhaps your browser is also "MITM"d via a management policy? hehe :S

Certainly a MITM between a website and LE is less likely than a MITM between a user on a random public Wi-Fi network and the website, but I've often wondered why more attention hasn't been given to securing the domain validation process itself.

There are a lot of random internet routers between CAs and websites which effectively have the ability to get certificates for any domain they want. It just seems like such an obvious vulnerability I'm kinda shocked it hasn't been exploited yet. Perhaps the fact that it hasn't is a sign such an attack is more difficult than my intuition suggests.

Still, I'd be a lot more comfortable if DNSSEC or an equivalent were enforced for domain validation. Or perhaps if we just cut out the middleman and built a PKI directly into the DNS protocol, similar to how DANE or Namecoin work.

EV certs seem to have basically the same verification policies that CAs had for ordinary certificates back in the early 2000s (i.e., really not that much at all), so I am intrigued as to what the DV has to offer except "it's basically self-signed but with extra steps and the rest of the world will trust it".

> If someone is between a service and LE

There is always someone there: my ISP, my government that monitors my ISP, the LE's ISP, and the US government that monitors the LE's ISP.

It does kind of suck that Let's Encrypt is entirely funded by donations from corporations like Google and Facebook. If they pulled support what would happen? Would 92% of websites we visit get downgraded to http?[1]

Also his point that it "supplants better solutions" is inarguably true. The 2010s had lots of conversations about certificate transparency and CA changes that just don't happen today because the existence of Let's Encrypt made it so easy to put a cert-signed website online.

[1] of US firefox users: https://letsencrypt.org/stats/

Hey, I only read EmbarrassingFetish.com for the recipes section (I recommend their carrot cake.) I'm not into the rest of the stuff there and you can't prove it thanks to HTTPS.

More seriously, you are not wrong. Site Authentication is still a problem and actually the weakest part of HTTPS but it is also more of a people problem than a technical one. Nothing stops somebody from registering MyB4nk.com but at least HTTPS stops crooks spoofing MyBank.com exactly.

> but websites over HTTP still work fine

The best attack surfaces always do. If I'm a smart attacker, why would I impair your experience (at least, until I get what I want)? It's better to give you a false sense of security. There are, of course, dumber attacks that will show obvious signs. While many people do fall prey to such attacks from lapses in, or impairment to, their judgment, the smarter attacks hide themselves better.

The classical model of web security based around "important" sites and "sensitive" actions has been insufficient for decades. It was certainly wrong by the time the first coffee shop/airport/hotel wifi was created; by the time the first colocation provider/public cloud was created; by the time every visitor/student/employee of any library/university/company was given open Internet access; etc.

> If the traffic is not tamper-proof, no matter how "unimportant" it may seem, it presents the opportunity for manipulation. All it takes is one of the nodes in the path to be compromised.

It also ignores one really important fact that these pipes are not perfect, they do introduce errors into the stream. To ensure integrity we would still need to checksum everything and in a way that no eager router "fixes".

We want our bank statements to be bit-perfect, our family pictures not to be corrupted, so on and on.

So even if someone handwaves away all the reasons why we need encryption everywhere (which is insane), we would still need something very similar to TLS and CAs being used. Previous TLS versions have even had "eNULL" ciphersuites.

> I couldn't care less if someone wants to spend the time and effort to intercept and change pages from a blog I read.

I do not understand why so many people think having, say, zero-day exploits served to them is not a problem.

The blog is not the target; the unsecured connection is.

Approximately nobody is taking the time to hand craft a specific modification of some random blog. They develop and use tools that manipulate any packet streams which allow tampering, without the slightest concern for how (un-)important the source of those packets is.

I think I've never ran it as root since it came out by using the `webroot` method, where certbot just writes the challenges to a specified path it has access to and that's it.

I haven't experienced that, since I prefer acmetool.

That quote is the only thing you have to read of that article besides the headline.

The ironic observation about the page using an LE cert is fantastic; Browser mandates make the encryption discussion moot. If you don't use it, your argument literally won't load for a modern audience.

It speaks to the problem of digital decay. We can still pull up a plain HTTP site from 1995, but a TLS site from five years ago is now often broken or flagged as "insecure" due to aggressive deprecation cycles. The internet is becoming less resilient.

And this has real, painful operational consequences. For sysadmins, this is making iDRAC/iLO annoying again.

(for those who don't know what iDRAC/iLO are, it's the out-of-band management controller that let you access a server's console (KVM) even when the OS is toast. The shift from requiring crappy, insecure Java Web Start (JWS) to using HTML5 was a massive win for security and usability - old school sysadmins might remember keeping some crappy insecure browser around (maybe on a bastion host) to interact with these things because they wouldn't load on modern browsers after 6mo)

Now, the SSL/TLS push is undoing that. Since the firmware on these embedded controllers can't keep pace with Chrome's release schedule, the controllers' older, functional certificates are rejected. The practical outcome is that we are forced to maintain an old, insecure browser installation just to access critical server hardware again.

We traded one form of operational insecurity (Java's runtime) for another (maintaining a stale browser) all because a universal security policy fails to account for specialised, slow-to-update infrastructure... I can already hear the thundering herd approaching me: "BUT YOU NEED FIRMWARE UPDATES" or "YOU NEED TO DEPRECATE YOUR FIRMWARES IF NOT SUPPORTED".. completely tone-deaf to the environments, objectives and realities where these things operate.

No. The private key does not leave the server, you can't use the certificate without it.