How (not) to sign a JSON object (2019)
40 comments
·February 9, 2025codeflo
WorldMaker
This sort of suggestion is for that worst case that you already have brownfield consumers that don't care about the signature using outer fields and you need to add the signature without breaking those consumers.
The redundancy is absolutely a recipe for disaster, but so is the part where you have brownfield consumers that you can't break and know that they also don't care about message security.
Unfortunately, it's an all too common brownfield to find yourself stepping into, which is why it is such a too common ask for "inline JSON signatures" (or other document languages like XML) that don't change the outer shape of the JSON document to break backwards compatibility with dumber consumers.
Also, unfortunately the most correct answer in cryptographic practice is also often the hardest to sell to those consumers (or to business people prioritizing changes to them): break those consumers and force them to care about security so that a rising tide lifts all boats.
hinkley
Yep. I had a dickens of a time making XML-DSIG secure. I don’t know how they didn’t realize that getElementById returns the first element with an id and doesn’t give a shit if there are multiples. If you chose a different parent element you can get different results. I had to roll my own that threw an error on duplicate IDs and rejected the document.
If you only expect one signature I would recommend you wrap the signed content instead of treating it as a sibling. And even if you have multiple, maybe have signatures be siblings but put them all in the same wrapper. This means the recipient has to know signatures exist but honestly tough shit. If you’re adding sigs you’re going to end up expecting them and that’s a fact not an opinion. You don’t want any tools that ignore the signature and make decisions without validating them first. That’s a Confused Deputy attack waiting to happen.
Also in XML you have to canonicalize the document first, so that any formatting changes don’t invalidate the signature. So a couple other parts of what you said are true but there are solutions, even if annoying ones.
treyd
This is one of the deeply dissatisfying parts of the Matrix spec. They didn't have any constraints forcing them to embed signatures within the json objects, but they elected to invent their own signing scheme and do it anyways, despite being a greenfield. It also includes support for a special "unsigned" portion for extra data that comes along (which is often used for the server to inject the age of an event).
I don't think the protocol still injects the signature into event structures, but this weird "unsigned" field is still there looking at the source json for a message I sent today, but it's possible it's removed after processing and Fluffychat is just removing it.
hinkley
I got so much shit for building an API that would not answer any queries about the signed documents until the signature had been verified. Trying to speed up processing and routing by making decisions before the authenticity of the data has been verified is a fool’s errand and false economy. You can’t make decisions based on what might be lies, and malicious ones at that. I spent a lot of time making the signature checks faster rather than buckling and making the signatures a joke.
lxgr
> Unless you have a good reason why you need an (asymmetric) signature, you want a MAC.
Is "I want the server/validating side to be safe even against server-side attackers with read-only permissions" not a good reason? Because that's one thing that asymmetric signatures provide out of the box compared to MACs.
monocasa
I really wish that XMLDSig wasn't such an awful standard that it turned a good third of the security industry against canonicalization in general.
Saying there's "sure there's lots of ways to serialize, but these specific rules get you the same octet and you sign that" is key to sanity in such situations.
For all of ASN.1's many sins, they got that part absolutely right.
DarkUranium
Not sure if I'm just misunderstanding the article or not, but it feels like an overengineered solution, reminescent of SAML's replacement instructions (just a hardcoded and admittedly way better option --- but still in a similar vein of "text replacement hacks").
I know it's not the most elegant thing ever, but if it needs to be JSON at the post-signing level, why not just something like `["75cj8hgmRg+v8AQq3OvTDaf8pEWEOelNHP2x99yiu3Y","{\"foo\":\"bar\"}"]`, in other words, encode the JSON being signed as a string. This would then ensure that, even if the "outer" JSON is parsed and re-encoded, the string is unmodified. It'll even survive weird parsing and re-encoding, which the regex replacement option might not (unless it's tolerant of whitespace changes).
(or, for the extra paranoid: encode the latter to base64 first and then as a string, yielding something like `["75cj8hgmRg+v8AQq3OvTDaf8pEWEOelNHP2x99yiu3Y","eyJmb28iOiJiYXIifQ"]` --- this way, it doesn't look like JSON anymore, for any parsers that try to be too smart)
If the outer needs to be an object (as opposed to array), this is also trivially adapted, of course: `{"hmac":"75cj8hgmRg+v8AQq3OvTDaf8pEWEOelNHP2x99yiu3Y","json":"{\"foo\":\"bar\"}"}`.
theamk
You can and this will be simple and reliable.. but that's solving the different (and easier) problem that the post. In the post, author wants to have still have parsable JSON _and_ a signature. Think middleware which can check signature, but cannot alter the contents, followed by backend expecting nice JSON. Or a logging middleware which looks at individual fields. Or a load balancer which checks the "user" and "project" fields. Or a WAF checking for right fields. In other words:
> Anyone who cares about validating the signature can, and anyone who cares that the JSON object has a particular structure doesn’t break (because the blob is still JSON and it still has the data it’s supposed to have in all the familiar places).
As author mentions, you can compromise by having "hmac", "json" and "user" (for routing purposes only), but this will increase overall size. This is approach 2 in the blog.
spankalee
Thats no different than the suggestion at the beginning of the article to serialize the JSON and sign the string.
Someone
> in other words, encode the JSON being signed as a string. This would then ensure that, even if the "outer" JSON is parsed and re-encoded, the string is unmodified. It'll even survive weird parsing and re-encoding, which the regex replacement option might not (unless it's tolerant of whitespace changes).
Would it be guaranteed to survive even standard parsing?
It wouldn’t surprise me at all, for example, if there are json parsers out there that, on reading, map “\u0009" and “\t" to the same string, so that they can only round-trip one of those strings. Similarly, there’s the pair of “\uabcd” and “\uABCD”. There probably are others.
LegionMammal978
Presumably when receiving the object, you'd first unescape the string (which should yield a unique output unless you have big parser bugs), check the UTF-8 bytes of the unescaped string against the signature, and only then decode the unescaped string as the inner JSON object. It shouldn't matter how exactly the string is escaped, as long as it can be unescaped successfully.
mbreese
There are many ways to represent the JSON as binary… and all are equally valid. The easiest case to think about is with and without whitespace. Because what HMAC cares about are the byte[] values, not alphanumeric tokens.
Then, if you couple this with sending data through a proxy (maybe invisible to the developers), which may or may not alter that text representation, you end up with a mess. If you base64 encode the JSON, you now lose any benefit you might gain from those intermediate proxies, as they can’t read the payload…
38
Json encoded as a string is cursed, no one should do that and stop suggesting it. Base64 is fine or even ascii85
Groxx
base64 is often even larger than an escaped JSON string. and not human-readable at all.
I'll take stringified json-in-json 90% of the time, thanks. if you're using JSON you're already choosing an inefficient, human-oriented language anyway, a small bit more overhead doesn't hurt.
(obviously neither of these are good options, just defer your parsing so you retain the exact byte sequences while checking, and then parse the substring. you shouldn't be parsing before checking anyway. but when you can't trust people to do that...)
null
benatkin
The comment you replied to was posted in good faith AFAICT. Your “stop suggesting it” is unnecessarily antagonistic.
askvictor
Another problem with signing JSON: you can have two different json objects that mean the same thing, and will do exactly the same thing in your code e.g. {"a": "foo", "b": "bar"} vs {"b": "bar", "a": "foo"}. Also, whitespace. Are there any standards for normalising json, so that two equivalent, but differently written JSON files will have the same signature?
afiori
Honestly it should not really matter, the regex bait-and-switch solution seem like the most practical one, there is some trickery in checking that the magic key does not appear in the string already but they seems far easier
lxgr
That's canonicalization, and the article does mention it (but unfortunately does not offer much insight other than that it's hard).
hansonkd
The difficulty stems from you have to rewrite the encoding/decoding canonicalization library in every language you want to consume the data with as opposed to simply piggy backing off of default implementations and the language's standard crypto libs.
For example most JSON parsers default to interpreting numbers from JSON as floats or ints. but in the canonical format you would have to force all parsers to interpret them as exact decimal values. then determine how to encode them (is one hundred "100" or "1e2") etc.
hinkley
Canonicalization is a pain in the ass. Write yourself a boatload of unit tests.
busymom0
This is why, in one of my projects, I first stringified the JSON using built in JSON.stringify(your_json) function, then signed that string and sent the string, its signature, and public key to server. Server verifies the signature using the string and if passes, then uses JSON.parse(your_string) to get the original json.
askvictor
The problem is the following two lines produce different outputs, despite having content that means the same thing:
console.log(JSON.stringify({ x: 5, y: 6 }));
console.log(JSON.stringify({ y: 6, x: 5 }));heinrich5991
Usually, this is not a problem for signing.
busymom0
I think the relevance of order is allowed to be up to each software's implementation:
https://datatracker.ietf.org/doc/html/rfc8259
Says:
> JSON parsing libraries have been observed to differ as to whether or not they make the ordering of object members visible to calling software. Implementations whose behavior does not depend on member ordering will be interoperable in the sense that they will not be affected by these differences.
So, different signature makes sense. But it should not be an issue as long as both software are calculating/validating the signature on the string and not json.
tgsovlerkhgsel
"it’s OK to sign the exact byte sequence."
Not just "OK". It's the only sane way to do it.
curtisszmania
Signing JSON is tricky, but let's break it down.
* Canonicalization is a potential pitfall.
* Consider including the exact string you're signing.
* The regex bait and switch is an option.
* An alternative format might be the answer.
What approach do you find most practical?
saurik
Supposedly, this is the non-code documentation for AWS Version 3 signing.
https://docs.aws.amazon.com/amazonswf/latest/developerguide/...
hughes
What does JSON object signing provide that TLS doesn't?
Does this imply that the application doesn't trust the transport/presentation layers?
hinkley
Caching or other forms or retransmission of the data.
Not all signed content is meant to be confidential. Or two-party confidential. Think about tokens. You have a refresh token that’s private between you and the destination, but you hand out session tokens to your users so they can talk to the destination directly. Or via another server that doesn’t have a cache coherency with the source.
Zamicol
We addressed these concerns while developing Coze, a cryptographic JSON messaging specification. The specification details how we chose to address these concerns.
er4hn
I saw from the main page that you are aware of COSE (RFC 8152), with it's super similar name, but I didn't see anything in https://github.com/Cyphrme/CozeX/blob/master/coze_vs.md comparing it or CBOR.
Is the improvement COZE has over COSE that the body is default human readable, whereas COSE it's in some machine format that needs a reader util?
Zamicol
We picked the name Coze as a play on words on JOSE ("Cypher JOSE", when Jose is typically pronounced in Spanish, "ho-zay". The English word coze meaning a friendly chat was too perfect for the name of a messaging specification). I somewhat regret not using the three letter "coz".
Cose also picked its name while thinking of JOSE. Cose is binary oriented, and attempts to be as similar to JOSE as possible.
Coze is a first principles reimagining of signing JSON.
As an exercise, we've played around with creating a binary format, (which we're calling Booze, Binary Oriented cOZE) but since Coze is already much more space efficient, it's not as beneficial. It may become more relevant with post quantum, as currently post quantum systems are much larger than ECDSA. The only other advantage would be removing the JSON semantics, but that's at the cost of implementing a binary format. The human readability aspect is paramount for our application, and we feel it's generally better practice.
er4hn
Thanks for explaining the background of that.
I agree that human readability can make things somewhat easier. The part about a binary format for the signatures confuses me a little - the signatures are going to be big no matter what. If you care about minimizing size would it make sense to just find a more efficient way to encode the signatures rather than change the entire message format?
j-krieger
Very cool! It shares its name with "COSE" (RFC 8152), a signing scheme for CBOR objects :)
Muromec
Or just use asn1 like normal people.
I have a feeling that idea 2 is a recipe for disaster:
> Add the tag and the exact string you signed to the object, validate the signature and then validate that the JSON object is the same as the one you got.
In cryptographic practice, redundant information usually spells disaster, because inevitably, someone will use the copy that wasn't verified.
But let's dig into it. If I understand this correctly, the suggestion is to have something like this:
It's mentioned that "the downside is your messages are about twice the size that they need to be". In my opinion, this scheme is pointless. To verify "the JSON object is the same as the one you got", you have to do what?1. Parse the outer object as JSON, extract and remove signatureInfo.
2. Verify the signature.
3. Parse the signedString as JSON.
4. Verify that the object you got in step 1 equal to object you got in step 3 using a some kind of deep equality.
First of all, this is error prone, and as underspecified as JSON is, there are potential exploits if the comparison isn't done carefully. But even worse, if you think about it, the outer JSON is entirely useless, since you need to parse the inner JSON anyway -- so why not just use it directly?
It seems to me that this suggestion is strictly worse than just sending the inner part:
Yes, it's no longer "in-band", but I don't think it was really in-band before, it was just out-of-band with an outer layer of redundant information.