I prefer human-readable file formats

I concur with most of these arguments, especially about longevity. But, this only applies to smallish files like configurations because I don't agree with the last paragraph regarding its efficiency.

I have had to work with large 1GB+ JSON files, and it is not fun. Amazing projects such as jsoncons for streaming JSONs, and simdjson, for parsing JSON with SIMD, exist, but as far as I know, the latter still does not support streaming and even has an open issue for files larger than 4 GiB. So you cannot have streaming for memory efficiency and SIMD-parsing for computational efficiency at the same time. You want streaming because holding the whole JSON in memory is wasteful and sometimes not even possible. JSONL tries to change the format to fix that, but now you have another format that you need to support.

I was also contemplating the mentioned formats for another project, but they are hardly usable when you need to store binary data, such as images, compressed data, or simply arbitrary data. Storing binary data as base64 strings seems wasteful. Random access into these files is also an issue, depending on the use case. Sometimes it would be a nice feature to jump over some data, but for JSON, you cannot do that without parsing everything in search of the closing bracket or quotes, accounting for escaped brackets and quotes, and nesting.

I can understand this for "small" data, say less than 10 Mb.

In bioinformatics, basically all of the file formats are human-readable/text based. And file sizes range between 1-2Mb and 1 Tb. I regularly encounter 300-600 Gb files.

In this context, human-readable files are ridiculously inefficient, on every axis you can think of (space, parsing, searching, processing, etc.). It's a GD crime against efficiency.

And at that scale, "readable" has no value, since it would take you longer to read the file than 10 lifetimes.

I journeyed from fancy commercial bookkeeping systems that changed data formats every few years (with no useful migration) to GNU Cash and finally to Plain-Text Accounting. I can finally get the information I need with easy backups (through VCS) and flexibility (through various tools that transform the data). The focus is on content, not tools or presentation or product.

When I write I write text. I can transform text using various tools to provide various presentations consumable through various products. The focus is on content, not presentation, tools, or product.

I prefer human-readable file formats, and that has only been reinforced over more than 4 decades as a computer professional.

Ease of: reading, comprehension, manipulation, short- and long-term retrieval are not the same problems. All file formats are bad at at least one of these.

Given an arbitrary stream of bytes, readability only means the human can inspect the file. We say "text is readable" but that's really only because all our tooling for the last sixty years speaks ASCII and we're very US-centric. Pick up a text file from 1982 and it could be unreadable (EBCDIC, say). Time to break out dd and cross your fingers.

Comprehension breaks down very quickly beyond a few thousand words. No geneticist is loading up a gig of CTAGT... and keeping that in their head as they whiz up and down a genome. Humans have a working set size.

Short term retrieval is excellent for text and a PITA for everything else. Raise your hand if you've gotten a stream of bytes, thrown file(1) at it, then strings(1), and then resorted to od or picking through the bytes.

Long term retrieval sucks for everyone. Even textfiles. After all, a string of bytes has no intrinsic meaning except what the operating system and the application give it. So who knows if people in 2075 will recognise "48 65 6C 6C 6F 20 48 4E 21"?

I see no issue using (or even receiving) an SQLite file, where I can see the tables structure and even export everything to pure text format.

The major problem of both human-readable and binary formats is not the serialized form, but the understanding of the schema (structure) of the data, which more often than not, is completely undocumented. Human-readable formats are worse in this regard, because they justify it by "it's obvious what this is".

/* Technically, most binary formats are legible to a human, given a proper renderer, e.g. journalctl. What TFA speaks about is ASCII/UTF-8 text formats that need no processing besides rendering CR, LF, and TAB characters specially. Assuming a Unix command line, I would call these "cat-readable" formats, or maybe even "less-readable". */

Even "human-readable" formats are only readable if you have proper tools - i.e. editors or viewers.

If a binary file has a well-known format and tools available to view/edit it, I see zero problems with it.

Given that the author mentions CSV and text table formats, the article's list of the "entire Unix toolchain" is significantly impoverished not only by the lack of ex (which is usefully scriptable) but by the lack of mlr.

* https://miller.readthedocs.io/

vis/unvis are fairly important tools for those text tables, too.

Also, FediVerse discussion: https://social.pollux.casa/@adele/statuses/01K1VA9NQSST4KDZP...

We learned the hard way, for some of us it's all too easy to make careless design errors that become baked-in and can't be fixed in a backward-compatible way (either at the DSL or API level). An example in Graphviz is its handling of backslash in string literals: to escape special characters (like quotes \"), to map special characters (like several flavors of newline with optional justification \n \l \r) and to indicate variables (like node names in labels \N) along with magic code that knows that if the -default- node name is the empty string that actually means \N but if a particular node name is the empty string, then it stays.

There was a published study, Wrangling Messy CSV Files by Detecting Row and Type Patterns by Gerrit J. J. van den Burg, Alfredo Nazábal, and Charles Sutton (Data Mining and Knowledge Discovery, 2019) that showed many pitfalls with parsing CSV files found on GitHub. They achieved 97%. It's easy to write code that slings out some text fields separated by commas, with the objective of using a human-readable portable format.

You can learn even more by allowing autofuzz to test your nice simple code to parse human readable files.

I'd love to see us go even further! Having file formats at all is less human readable than going 9p style, less human (and script) readable than a directory tree of simple values.

The OS has a builtin way to create heirarchical structured data. I'd love to see the boldness to try using that!

There was a good example of a json->directory tool submitted yesterday, json2dir. https://github.com/alurm/json2dir https://news.ycombinator.com/item?id=44840307

This does mean you'd need to tar/zip up your file tree to send it around. It's less clear that this directory is a thing, is meant to be a file like one entity. But by taking our data out of complex file formats and turning it into a a filesystem heirarchy, it removes the arbitrariness of choosing any encoding at all, and it directly opens up all data to scripting. You can echo a new hex color into a foreground file to change it. You can watch a user directory for changes to update a rendering of it. You can btrfs snapshot your window layout to save your desktop configuration.

We have so so many specific tools for computing. It's time that we try making some more general systems, that let us work broadly / regardless of specific application! "Just files" could be a key enabler!

Readable files are great… until they’re 1TB and you just want to cry.

Let’s say that hypothetically one were to disagree with this. What would be the best alternative format? One that has ample of tooling for editing and diffing, as though it was text, yet stores things more efficiently.

Most of the arguments presented in TFA are about openness, which can still be achieved with standard binary formats and a schema. Hence the problem left to solve is accessibility.

I’m thinking something like parquet, protobuf or sqllite. Despite their popularities, still aren’t trivial for anyone to edit.