TPSV, an Alternative to TSV (and CSV)

Now sure about "we" but I have used these for years in personal projects for own purposes.

          Dec      Octal  Hex   Binary

          028      034    01C   00011100       FS    (File Separator)
          029      035    01D   00011101       GS    (Group Separator)
          030      036    01E   00011110       RS    (Request to Send)(Record Separator)
          031      037    01F   00011111       US    (Unit Separator)

Not sure if satire, or not. Just in case of not:

https://en.wikipedia.org/wiki/ASCII#Character_groups

https://en.wikipedia.org/wiki/C0_and_C1_control_codes#C0_con...

Yeah :) Though I can think of two reasons why: it's not typable for most people on a keyboard, and most programs are not designed to deal with it, or render it properly in an aligned way, like tab characters.

For multiline cell contents, there is rule 7, the multi line extension. Newlines are not allowed in cells otherwise, because of rule 2, it's a line based format.

I personally use it to write tabular data manually, used to define our datamodel. Because this format is editor agnostic, colleagues can easily read and edit as well. So in my case it's focus on human read/write and machine read.

Well there's JSONL which is used heavily in scientific programs (especially in biology)

But CSV represented as JSON is usually accomplished like so:

  {
    "headers": ["name", "habitat", "food"],
    "data": [
      ["Acorn Woodpecker", "forest", "grain"],
      ["American Goldfinch", "grassland", "grain"],
      ["Anhinga", "wetland", "fish"],
      ["Australian Reed Warbler", "wetland", "grub"],
      ["Black Vulture", "forest", null]
    ]
  }

S-expressions?

The spec says the last cell does not need to end in a tab, so this would be two cells IMO

I think spec tries and fails to translate code implementation into human language. In code cell separator is `\t|`.

> We need binary formats. In this era we are capable for it.

We have them, they're used where appropriate.

> Throw away the text formats.

I would argue that _most_ of the time tsv or csv are used it's because either:

a) the lowest common denominator for interchange. Oh, you don't have _my specific version of $program? How about I give you the data in csv? _everything_ can read that...

b) a human is expected to inspect/view/adjust the data and they'd be using a bin -> text tool anyways. The move to binary based log formats (`journald`) is still somewhat controversial. It would have been a non-starter if the tooling to make the binary "readable" wasn't on-par with (or, in a few cases, better!) than the contemporary text based tooling we've been used to for the prior 30+ years..

Text is universally accessible and widely supported. Binary has it's benefits, but human facing, it has to be text.

The idea that binary formats are the way because "you're going to use a program to interact with the format anyway" ignores the network effects of having things like text editors and unix commands that handle text as a universal intermediate, while having bespoke programs for every format dooms you to developing a full set of tooling for every format (or more likely, writing code that converts the binary format to text formats).

More recently though, consider that LLMs are terrible at emitting binary files, but amazing at emitting text. I can have a GPT spit out a nice diagram in Mermaid, or create calendar entries from a photo of an event program in ical format.

Which formats are helpful can depend on the use. I think DER (which is a binary format) is not so bad (although I added a few additional types (such as key/value list, BCD string, and TRON string), but not all uses are required to use them). I had also made up Multi-DER, which is simply any number of DER concatenated together (there are formats of JSON like that too). (I had also made up TER which is a text format and a program to convert TER to DER. I also wrote a program to convert JSON to DER. It would also be possible to convert CSV, etc.)

It was also my idea of an operating system design, it will have a binary format used for most stuff, similar to DER but different in some ways (including different types are available), which is intended to be interoperable among most of the programs on the system.

Yep, and stuff it into a sqlite db too you have a query interface all built.

Ok but how do you type them? How do you search them? How do you copy-and-paste between documents?

parquet?

Data will always outlive the program that originally produced it.

This is why you should almost always use text formats.

CSV is also a display format for humans and also for computers. Its also a terrible one because its too variable FS is variable, escapes could exist, "" could be used, this slows down parsing.

I second this. I'm using JSONL to bake in the data for my single binary Finnish to English pocket dictionary ( https://github.com/hiAndrewQuinn/tsk ). It just makes things like data transformations so easy, especially with jq.

How so? All you need is a text editor that preserves tabs.