Hidden Messages in Emojis and Hacking the US Treasury
81 comments
·March 14, 2025cogman10
gowld
Let's be clear: Beyond Trust is not a company that wrote a database-backed web app and made the all-too-common mistake of writing insecure code that tickled a bug in the database that allowed privilege escalation. Beyond Trust's is a company whose entire contribution is adding a security layer to prevent privilege escalation, and their solution here was to bypass Postgres's standard functionality and use this weird `psql` hack instead.
They had one job, and they failed at it. This amateur-level mistake should sink the entire company.
pjc50
Didn't happen when Crowdstrike broke all their customers.
The problem is that getting information security right is a matter of process control, which everyone hates, and so CEOs are absolute suckers for being sold a product which magically "adds on" security. This is like trying to buy "anti-lead-paint" rather than actually remove all your existing lead paint.
1oooqooq
but they got the right sales people to get to the IRS and they hired the "right" (wrong) certification company. so that's three jobs at least.
we now have one job to ask for accountability and will not do it.
somat
There is an interesting project https://github.com/Abstrct/Schemaverse where the whole game takes place entirely within a postgres database that the players connect directly to.
The author discovered quite a few... well lets just call them policy errors in postgres, but had a hard time filing bug reports, mainly because the response was usually an incredulous "why on earth would you even do that in the first place?.
But the author has fun with this, There is a trophy in the game that you can only get by putting your name in the trophy table.
0xbadcafebee
The only reason BeyondTrust implemented that was it wasn't untrusted user commands. They sanitized the data, so it should have been fine. The unfortunate problem was that the sanitizer didn't sanitize.
Systems are built on a set of expectations. Undermine the expectations and you undermine the system.
cogman10
> They sanitized the data, so it should have been fine.
This is a 101 rookie level approach to SQL or injection defense.
It's dumb for exactly the same reason why this is dumb
"SELECT * FROM foo WHERE bar=" + sanitize(userInput)
"SELECT * FROM foo WHERE bar=?"
bindParameter(1, userInput);
IF this company wanted to build dynamic queries, then the only correct way to do that is to limit input to only valid variables. IE "isValidColumnName(userInput)" before sending the request. And even then, you'd not use psql to do that.
You simply can't use a generalized sanitizer and expect good results.
voxic11
This is the article I link my developers whenever I see them making this mistake "Don’t try to sanitize input. Escape output." https://benhoyt.com/writings/dont-sanitize-do-escape/
Its been fairly effective at making them realize the fundamental mistake they are making. Quoting the key part:
> The only code that knows what characters are dangerous is the code that’s outputting in a given context.
> So the better approach is to store whatever name the user enters verbatim, and then have the template system HTML-escape when outputting HTML, or properly escape JSON when outputting JSON and JavaScript.
> And of course use your SQL engine’s parameterized query features so it properly escapes variables when building SQL
oefrha
The more you work in software the more you should realize the developers writing security-critical software (in this case the one writing that sanitizer) are often/usually as clueless as you are. The solution? Hard to say.
thaumasiotes
> The correct way to do something like this will always be parameterized input which looks something like this
> Why? Because [] the postgres protocol splits out the command and the data for the command in a way that can't be injected.
I'm not sure I'm comfortable with this. You can create a prepared statement and then pass user input to it as parameters, sure. https://www.postgresql.org/docs/17/sql-prepare.html
But who says that means your statement can't be injected? It would have to be true that the handling for EXECUTE statements is bug-free. Maybe it is bug-free. Or maybe it isn't. Maybe I can figure out just the right username to cause your prepared statement to have some undesirable side effects.
That wouldn't be SQL injection in the sense of putting hostile values into an SQL query in order to form a different SQL query, but it would be SQL injection in the sense of putting hostile values into an SQL query in order to accomplish nefarious goals via the database. The only real qualm I'd have about calling it "SQL injection" is that it wouldn't be portable across different database implementations; it would be more accurately described as "PostgreSQL injection".
If we feel entitled to assume that PostgreSQL's provided functionality to interpret strings that are provided to EXECUTE statements has no bugs, why aren't we also entitled to assume that PostgreSQL's provided functionality to interpret strings that are provided to the string escaper has no bugs? I don't really see the conceptual difference.
furstenheim
100%.
Btw, even using psql directly allows binding parameters https://www.postgresql.org/docs/current/app-psql.html
gcr
I’m starting to think that string-based languages like SQL which mix structure and content are a mistake.
Maybe future database systems will only accept queries serialized from protobuf, or JSON (output by a proper serializer)
Nijikokun
His argument is that you are building on the assumption that this is safe:
"SELECT * FROM foo WHERE bar=?"
bindParameter(1, userInput);
null
zoeysmithe
Just a guess but this looks like politically powerful dev culture overwriting cybersecurity culture, demanding, thus getting an exception from management for 'productivity' and 'being agile.'
I dont think we appreciate how much of a wild west things are with the incredible mix of hugely complex and powerful tools available trivially to developers and the concept of "move fast, break things."
Especially as corporate sees devs like they see salesmen (big moneymakers who deserve perks, exceptions) and top-down security culture as a cost center.
The other buried ledes are that postgres allows emojis (not sure if that's intended but it works) and that you can just run system commands and scripts directly from postgres cli. I imagine a lot of eyes are going to be on new hardening guidelines for postgres now.
I also imagine the first high performance enterprise friendly drop-in db written in something like rust is going to one day be a big deal.
Terr_
Hey now, a large portion of developers are seen as cost-centers too! Not everybody has the skill of flattering managers into approving greenfield projects, and then transferring away before they break horribly. :p
randmeerkat
> Especially as corporate sees devs like they see salesmen…
You’re onto something here. People perceive the world through the lens of their education and environment. Sales, legal, finance, are all easy constructs for a business leader to view the rest of the world through. The secret of the game isn’t to have the best tech or to code the most, it’s to “outsell” your competing business unit.
singpolyma3
Giving untrusted users ssh access... You mean like every shared hosting company or shell provider?
charcircuit
Which is why shared hosting companies get hacked relatively more.
charcircuit
>Beyond Trust did their due diligence by properly calling a sanitization method on the user’s string input using it in a PostgreSQL query.
This is not due diligence. In band messaging of user controlled data has been proven to be bad for security and this is not the first time "escaping" user controlled data for SQL has been done incorrectly.
Spivak
Yep, it's prepared statements or bust. But the long tail of legacy code, examples, documentation, that uses escaping is gonna take a while to get through.
One of the nice things about modern ORMs like SQLAlchemy 2 is that it forces you to use prepared statements even when when calling raw queries.
1oooqooq
without the PR spin: beyond trust did the bare minimum when implementing an open back door to a federal database
kevinsync
Fun fact: MySQL (and I'm sure many other databases?) lets you pass in values directly as hexadecimal strings.
I've avoided SQL injection since ancient times not by escaping strings, but by transforming any given input via "0x" + bin2hex(value) and plopping that into the query.
No quotes needed, no code buried deep in included libraries needed, handles any kind of data possible, and also no funny business sneaking in based on how you may have mangled the input.
vessenes
Cool idea. Genuinely. But it does not in any way guarantee that an injection attack like used here won’t work - unless it’s maintained as hex through the whole pipeline. In this case the (malformed) Unicode was sent to a command line call - if your hex text needed to be parsed to be understood on the command line, then your security plan would have failed.
kevinsync
Totally agree on my tip not being a silver bullet for all situations, just wanted to pass it along in case somebody finds themselves needing to sanitize input for queries rather than constructing prepared statements.
I was a bit perplexed by the final destination of command line for data and/or queries -- seems like an odd choice when they could've just interfaced directly with the database like a civilized human hahaha
mtrovo
So many questions:
- why a no side-effects function on a database can be used to get lateral access to the whole database instance
- why do you need to validate strings on the database itself and not on the client anyway, heck why are there no type safe way of doing it
- why would you want to execute shell commands from the database itself
- Even if there's a real use case for executing commands like that why is it enabled by default on a regular connection to the database without you specifying a THIS_IS_REALLY_DANGEROUS_BUT_I_PINKY_PROMISE_I_KNOW_WHAT_IM_DOING flag to the connection handshake.
It's not always PHP but there are some kirks that are shrugged off on PHP that makes me really concerned about the reliability of projects coded with it.
benmmurphy
They mentioned PAM module so maybe the sql injection just allowed bypassing the authorization of a system that was using the PAM module. Like it’s in the realm of possibility that a PAM module that wanted to validate a user against credentials stored in a pg database might shell out to the psql command to do this. Though, the whole thing is very questionable.
rapind
Yeah we’re missing some info.
What account were they authenticating with when attaching to psql?
If you have the connection string why does psql even matter, couldn’t you use any client? Or is this a case of your input being forwarded to a running, already authenticated, psql instance?
And finally, why do we need unicode support for schema? I assume it’s because the schema is itself data?
c45y
In this case PAM is the name of a type of security product and not the Linux PAM system.
jarebear6expepj
Your questions are programming language agnostic-- where did your PHP angst come in? And are there specific things in PHP that are problematic and avoidable by using a different Turing complete language?
marsovo
PHP has grown up but in its wild youth was notorious for such gems as mysql_escape_string vs mysql_real_escape_string, rather than proper parameterization
It's not so much about Turing as it is libraries and patterns
After all, as I understand it this very issue was caused by escaping SQL rather than parameterizing it
xmprt
I learned about the breach a few days ago but I didn't know that you could "adopt" an emoji: https://aac.unicode.org/sponsors
That's a neat tidbit.
jfengel
Does "adopting an emoji" mean something other than "appearing on that page"?
From the adoption page: Each adoption comes with a digital badge and certificate that you can proudly display.
Implicated
It also appears to be a $5k permanent backlink from unicode.org, custom anchor text even.
nkrisc
It’s a fun way to donate.
thrdbndndn
It's just a way to donate, similar to lots of "adopt a xxx" programs.
null
Validark
Could someone please explain to me why "sanitizing database inputs" was ever considered a good idea? Why not just add a feature in SQL like so?
SELECT * FROM users WHERE username = [<LEN_OF_TEXT>]raw-text-of-len-not-parsed-at-all
SELECT * FROM users WHERE username = [21]flyin' and wavin' guy
^^^^^^^^^^^^^^^^^^^^^
these 21 chars are NOT parsed AT ALL, just taken as data
whatnow37373
How is this not "sanitizing inputs"?
The basic idea behind your proposal exists and is called prepared statements. It's actually, I hope, the normal way to write queries these days.
You write your query like: "SELECT * FROM users WHERE username = ?" and execute your query like "execute(query, username)".
The problem? It's optional.
Validark
To me, "sanitizing inputs" implies a transformation of the data into a string that can be "safely" evaluated as code which hopefully yields the input data. Instead you should be able to just mark a piece of the code as data, that will never be tokenized or parsed or anything, just dropped directly into a buffer.
"Prepared statements" sounds EXACTLY like what I was thinking! I don't understand why people would ever use anything else.
whatnow37373
Ah, I see! It's a cool idea, but .. let's try to be maximally obtuse and pedantic today. I'm a developer and it's HN after all.
[4]tree is also code that yields data. At the end of the day some kind of parser needs to decide what to do with your data and [ ] is just another way of escaping special characters. In this case it escapes entire strings instead of individual characters. It's your special way of sanitizing the input.
Questions: Who is responsible for the number? What is this number: bytes, "characters", runes? What happens if the number is wrong? (If you expose this number to external factors of any kind you get a special, interesting new breed of SQL injection.)
In practice you'd probably do something like:
my_special_superduper_safety_syntax_preprocessor("SELECT * FROM users WHERE username=$$$", "peter")
Which will yield something like:
"SELECT * FROM users WHERE username=[5]peter"
.. so you don't have to calculate the number. If we're doing this, why not just go for:
exec("SELECT * FROM users WHERE username=?", "peter")
.. and be done with it.
> I don't understand why people would ever use anything else.
Yes, I agree. Usually it's some interesting combination of laziness and ignorance.
billpg
Most (all?) SQL client libraries will allow you indicate a parameter placeholder and supply that parameter value separately.
Terr_
> In order to to this, PQescapeStringInternal must call pg_utf_mblen.
For a moment I had a dev-flashback to the problem of utf8mb4 versus (broken) utf8 in mySQL. Easy now, this is Postgres, everything is safe(er)...
nthingtohide
Oh, the fun we had when a single emoji which crashed youtube or whatsapp.
Bluecobra
A while back I stumbled on a way to crash my company’s Jira server simply by sending an email to it containing an emoji. Makes me wonder if that could have been abused by malicious parties if they knew the email address we used to forward new support issues to Jira.
hnlmorg
As someone who’s done a fair amount with parsing of Unicode strings lately, I’m not at all surprised by this bug.
Unicode is a surprisingly elegant system but also an open invitation for all kinds of abuse.
rapind
> Unicode is a surprisingly elegant system…
s/elegant/clever
What could go wrong? I bet unicode is how AGI escapes and enslaves humanity.
hnlmorg
The cleverness is in the simplicity of its implementation while maintaining backwards compatibility. Which satisfies the definition of “elegance”.
Working with Unicode is anything but elegant, but that’s another story.
rapind
> Working with Unicode is anything but elegant, but that’s another story.
Yeah I hear ya. IMO this is really important though and I don’t think there’s much of a story if this isn’t part of it. The design is clever but the resulting usability (and error proneness) leaves much to be desired. Not knowing what’s in there, like a closed envelope, increases user complexity significantly.
giancarlostoro
I havent deep dived unicode enough to know, and probably someone somewhere already made it, but I often wonder if we can do compression more efficiently by abusing unicode somehow, especially regarding plain text.
dalemhurley
I have been playing a lot with emoji smuggling with https://toolnames.com/utilities/emoji-smuggler and https://chat.full.cx inspired by https://emoji.paulbutler.org/
I had not considered it as an injection attack.
verisimi
> Out of compliance, the US Treasury posted this notice to US lawmakers, breaking the news that a “China state-sponsored Advanced Persistent Threat (APT) actor” had breached their systems.
https://www.youtube.com/watch?v=ekr2nIex040
ROSÉ & Bruno Mars - APT. (Official Music Video)
nickdothutton
Today in “why enumerating badness is stupid”.
> Now, all of this might have been fine had Beyond Trust not written a feature which allowed users to directly, programmatically interact with psql (the postgres command line interface).
That's the buried lede.
Yes, there was a vulnerability in psql... but that's so much less a problem than the huge gaping hole of allowing users to directly interact with psql.
No DB can be safe if you are turning untrusted user commands into psql executions. It'd be like giving untrusted users ssh access and then complaining when they find a privilege elevation exploit.