Spotting base64 encoded JSON, certificates, and private keys

It's like how all certificates sound like "miiiiii".

Ey, I'm JSON!

"Is that you again, punk zip?" when seeing the first few bytes of a zip file.

>JWTs all sound like "eyyyyyy" in my head.

"eeey bruh, open the the API it's me"

eyy lmao

That’s really a leap about the writer’s interest.

They could just as easily have felt the underlying reason was so obvious it wasn’t worth mentioning.

I know how base64 encoding works but had never noticed the pattern the author pointed out. As soon as read it, I ubderstood why. It didn’t occur to me that the author should have explained it at a deeper level.

I think the audience already understands why it works, it's more the knowing there's a relatively small set of mnemonics for these things that's interesting. "eyJ" for JSON, "LS0" for dashes (PEM encoding), "MII" for the DER payload inside a PEM, and so on.

I've been doing this a long time but until today the only one I'd noticed was "MII".

I'd be very hesitant to consider this as some runaway symbol of "CS people being incurious now" over the author simply not being this deeply invested in this at the time of writing in the context of their discovery, especially since it almost certainly doesn't actually matter for them beyond the pattern existing, if even that does.

I think CS grads often skip the part of how something actually works and are happy with abstractions.

I would hope that these logs don't include the full details of the credit card (such as number/cvv).. if it does, the company that is logging this info could end up having some issues with Visa/MC

Edit: Now that I looked at it a little deeper, i'm assuming they are talking about these[0] sort of files?

[0]: https://docs.helix.q2.com/docs/card-transaction-file

That's got to be the most niche party trick I've ever heard of.

The other base64 prefix to look out for is `MI`. `MI` is common to every ASN.1 DER encoded object (all public and private keys in standard encodings, all certificates, all CRLs) because overwhelmingly every object is a `SEQUENCE` (0x30 tag byte) followed by a length introducer (top nibble 0x8). `MII` is very very common, because it introduces a `SEQUENCE` with a two byte length.

Thanks for pointing it out! I've added an errata to the blog post

By the way, I would guess that this is almost certainly why LLMs can actually decode/encode base64 somewhat well, even without the help of any MCP-provided tools - it's possible to 'read' it In a similar way to how an LLM might read any other language, and most encoded base64 on the web will come with its decoded version alongside it.

nitpick but ascii would be base128, largest ascii value is 0x7f which in itself is a telltale if you are looking at hex dumps.

For reference, a program to generate the quasi-fixed point from scratch:

  #!/usr/bin/env python3
  import base64

  def len_common_prefix(a, b):
      assert len(a) < len(b)
      for i in range(len(a)):
          if a[i] != b[i]:
              return i
      return len(a)

  def calculate_quasi_fixed_point(start, length):
      while True:
          tmp = base64.b64encode(start)
          l = len_common_prefix(start, tmp)
          if l >= length:
              return tmp[:length]
          print(tmp[:l].decode('ascii'), tmp[l:].decode('ascii'), sep='\v')
          # Slicing beyond end of buffer will safely truncate in Python.
          start = tmp[:l*4//3+4] # TODO is this ideal?

  if __name__ == '__main__':
      final = calculate_quasi_fixed_point(b'\0', 80)
      print(final.decode('ascii'))

  Vm0wd2QyUXlVWGxWV0d4V1YwZDRWMVl3WkRSV01WbDNXa1JTVjAxV2JETlhhMUpUVmpBeFYySkVUbGho

From the other direction, you'd call it a tail-eating unquine?

Note that the suffix will grow in length with the input, making it less and less interesting.

(Because the output is necessarily 8/6 the size of the input, the suffix always adds 33% to the length.)

Years ago I was part of a group of people I knew who could read and edit large parts of sendmail.cf by hand without using m4. Other people who had to deal with mail servers at the time certainly treated it like a superpower.

What’s wrong with this?

The purpose of Base64 is to encode data—especially binary data—into a limited set of ASCII characters to allow transmission over text-based protocols.

It is not a cryptographic library nor an obfuscation tool.

Avoid encoding sensitive data using Base64 or include sensitive data in your JWT payload unless it is encrypted first.

> It's really too bad that there's really nothing quite like json

messagepack/cbor are very similar to json (schemaless, similar primitive types) but can support binary data. bson is another similar alternative. All three have implementations available in many languages, and have been used in big mature projects.

>Everything speaks it and can write it.

asn.1 is super nice -- everything speaks it and tooling is just great (runs away and hides)

We just need to sacrifice n*field_count to a header describing the structure. We also need to define allowed types.

> It'd be nice if something like protobuf was easier to write and read in a schemeless fashion.

If you just want a generic, binary, hierarchical type-length-value encoding, have you considered https://en.wikipedia.org/wiki/Interchange_File_Format ?

It's not that there are widely-supported IFF libraries, per se; but rather that the format is so simple that as long as your language has a byte-array type, you can code a bug-free IFF encoder/decoder in said language about five minutes.

(And this is why there are no generic IFF metaformat libraries, ala JSON or XML libraries; it's "too simple to bother everyone depending on my library with a transitive dependency", so everyone just implements IFF encoding/decoding as part of the parser + generator for their IFF-based concrete file format.)

What's IFF used in? AIFF; RIFF (and therefore WAV, AVI, ANI, and — perhaps surprisingly — WebP); JPEG2000; PNG [with tweaks]...

• There's also a descendant metaformat, the ISO Base Media File Format ("BMFF"), which in turn means that MP4, MOV, and HEIF/HEIC can all be parsed by a generic IFF parser (though you'll miss breaking some per-leaf-chunk metadata fields out from the chunk body if you don't use a BMFF-specific parser.)

• And, as an alternative, there's https://en.wikipedia.org/wiki/Extensible_Binary_Meta_Languag... ("EBML"), which is basically IFF but with varint-encoding of the "type" and "length" parts of TLV (see https://matroska-org.github.io/libebml/specs.html). This is mostly currently used as the metaformat of the Matroska (MKV) format. It's also just complex enough to have a standalone generic codec library (https://github.com/Matroska-Org/libebml).

My personal recommendation, if you have some structured binary data to dump to disk, is to just hand-generate IFF chunks inline in your dump/export/send logic, the same way one would e.g. hand-emit CSV inline in a printf call. Just say "this is an IFF-based format" or put an .iff extension on it or send it as application/x-iff, and an ecosystem should be able to run with that. (And just like with JSON, if you give the IFF chunks descriptive names, people will probably be able to suss out what the chunks "mean" from context, without any kind of schema docs being necessary.)

For anyone here who's never pondered it ("today's lucky 10,000"?), there's a lot of intentional structure in the organization of ASCII that comes through readily in binary or hex.

https://altcodeunicode.com/ascii-american-standard-code-for-...

The first nibble (hex digit) shows your position within the chart, approximately like 2 = punctuation, 3 = digits, 4 = uppercase letters, 6 = lowercase letters. (Yes, there's more structure than that considering it in binary.)

For digits (first nibble 3), the value of the digit is equal to the value of the second nibble.

For punctuation (first nibble 2), the punctuation is the character you'd get on a traditional U.S. keyboard layout pressing shift and the digit of the second nibble.

For uppercase letters (first nibble 4, then overflowing into first nibble 5), the second nibble is the ordinal position of the letter within the alphabet. So 41 = A (letter #1), 42 = B (letter #2), 43 = C (letter #3).

Lowercase letters do the same thing starting at 6, so 61 = a (letter #1), 62 = b (letter #2), 63 = c (letter #3), etc.

The tricky ones are the overflow/wraparound into first nibble 5 (the letters from letter #16, P) and into first nibble 7 (from letter #16, p). There you have to actually add 16 to the letter position before combining it with the second nibble, or think of it as like "letter #0x10, letter #0x11, letter #0x12..." which may be less intuitive for some people).

Again, there's even more structure and pattern than that in ASCII, and it's all fully intentional, largely to facilitate meaningful bit manipulations. E.g. converting uppercase to lowercase is just a matter of adding 32, or logical OR with 0x00100000. Converting lowercase to uppercase is just a matter of subtracting 32, or logical AND with 0x11011111.

For reading hex dumps of ASCII, it's also helpful to know that the very first printable character (0x20) is, ironically, blank -- it's the space character.