Archival Storage

> This is because most of the non-technical retail customers don't value reliability too much. … Similarly, many users don't have much valuable, unique data on their computers. The most important stuff very often lives completely in the cloud, where it's properly backed up, etc.

There’s a contradiction here which I think is worth fully considering. People definitely do value their data – data recovery services have been getting revenue for decades from people who lost things they care about – but there’s been a market failure where operating system vendors left the users to fend for themselves, everyone agreed that was hard, and the majority of users quite rationally decided that cloud services were better than learning to be unpaid sysadmins.

What we should be asking is why this is all so hard. Local file systems with strong integrity checks and seamless replication onto attached storage should be so basic that people can just assume it’ll work without them having to spend time on it.

And without care. Everyone's precious photo albums will go with them to the grave. We live in a time where the books and data we write are likely to vanish long before any previous period. We don't leave behind books, codexes, or scrolls, or stone tablets anymore.

Scholars will have more clue about life in 4th century via the Oxyrhynchus Papyri collection than 21st century Terra Haute, Indiana.

> Silent data corruption can still be an issue, but, frankly, malware is a much bigger threat for a typical non-technical user.

Complete BS. Cryptolockers and similar are not a concern for regular users while disk failure is something pretty much everyone gets to experience over their lifetime.

Hehe. If you're trying to live with just a laptop and an LTO drive, I can certainly see it! I'd expect most people who get into LTO drives have a massive set of hard drives, some convoluted desktop-and-up case(s) to run them, a couple laptops and random peripherals such as cameras, scanners and whatnot. USB all over the place. - So that for them both the drive and the command line interface are deeper in a technology map.

But that goes back to the minimal market for LTO among amateurs. They will scratch their itch and write software for it but it's not exactly critical mass.

And here I just bought a NIB LTO-5 SAS drive for home file server backup.

Nobody needs to worry about storage density in the sense of how much shelf space you need for cold storage. Tape, NAND flash, and even high-capacity hard drives are all dense enough that the incremental cost of more shelf space is not the most important part of the problem.

And NAND flash is already pretty 3D, with hundreds of layers of memory cells fabricated on the surface of each wafer, and several such dies stacked in each BGA package, and it's not uncommon for U.2 drives to contain two PCBs each with NAND packages on each side.

Tape is a way to get volume. Through easily spooled layers. Not fast access but otherwise not unreasonable. Plus tapes stored outside of the drive.

Another way to look at it: "and yet here we are". Tape is still what has led to the highest density so far (depending how much storage you look at, and cost tradeoff). But also cost tradeoff of separating the drive from the medium. which hasn't really worked: LTO tapes are pro supplies - and so, expensive.

Good thinking - there are many projects around this. Ceramic, DNA, 3d glass, etc. They all run into the same IO problems tape has though.

Tape on a spool is 3D volumetric storage

Tape is generally available, I don't think in of those more esoteric storage medium, like 3D glass cubes are commercially available.

I suppose I can publish on by blog and then keep on writing. Can discover this future together along with anyone who wants to read it. This is my first serious attempt at fiction. I have only just begun.

This is another portion of later in the first chapter.

'...The transport, a ground-based vehicle, levitated silently outside. John glanced at his chronometer as he boarded. Jeg er ked af det, he thought, I'm late. The doors hissed shut, and the vehicle computer announced, “Destination: University. Estimated arrival: 25 minutes.” With a gentle hum, the vehicle glided smoothly along the elevated guideway. The air inside was cool and faintly metallic. Outside, the landscape was a patchwork of green fields, managed forests, and gleaming white research facilities. The transport’s progress was slow; the gentle sway and hum of the engines were a constant reminder of the NAU’s strict energy policies. John sighed, thinking about the upcoming lecture. How could he convey the importance of the Bit Rot Era when so many were focused on the pressing needs of the present?

At the University, John rushed into the classroom, three students already waiting. “Jeg er ked af det, I’m late,” John said, a slight Danish accent coloring his Danglish. The classroom was intimate, a very different design from the great lecture halls of antiquity, designed for perhaps twenty students, with a central platform surrounded by holographic cameras for remote attendees. Historical maps and timelines adorned the interactive displays lining the walls.

John quickly moved to the lectern and carefully removed six artifacts from his bag. “Velkommen to Ancient North America 1,” he began. “Welcome to Ancient North America 1. This is the class where you are going to learn about the past that was and the present that may yet be.” He gestured to a holographic timeline that appeared above the lectern. “In broad strokes, we consider the history of Ancient North America to consist of four periods: the pre-colonial, the rise of the nations – of Canada, the United States of America, and Mexico – the pre-Collapse, and the post-Collapse periods. You can take in-depth courses on most of these. Dr. Jones’ ‘Rise of the Nations’ is well worth your study to learn about the United States, its Constitution, and Canada. This very university is located in a place that was once called Newfoundland, Canada, and in ancient times the climate was quite harsh. Meget forskelligt from the lush green farmland, forests, and even nice beaches we have today. You can also take Dr. Pech’s history of the pre-colonial tribes and empires. That will teach you a great deal about the people who first inhabited the continent and learn about their history, culture, foodstuffs, many of which are still considered extinct. Desværre.

“However,” John continued, his tone shifting, “the class you cannot take is for what we call the Bit Rot Era. A ‘bit’… imagine a light switch. On or off. That’s a bit. One or zero. The basic unit of digital information. The Bit Rot Era, from roughly the 20th to the 25th century, is a complete black box. En sort boks. What we do know comes from fragments of writing on paper. Millions of books were printed, but even those are often lost to time. We have fragments of ancient texts that talk about computers in every home and the ‘digitization’ of information and libraries. Digitization was the process of scanning physical media into computers. We’ve recovered millions of artifacts – disintegrated polycarbonates, silica, and bits of rare metals that were once these computers. But nothing within survived. Then something happened. The books vanished. The period from about the 21st to the 25th centuries was, as it were, expunged from history. Research indicates they went entirely digital, depending on system administrators to maintain the data… until the administrators stopped.”...'

FWIW does any of it remain on archive.org? They have some really great collections on there, and writing on digital archiving, like how to preserve CDs as best as possible (spoiler: the brand of CD is a large factor).

It's a mistake to assume that just because there are 1000 year old text that the optimized-for-cost prints you can buy today will also last that long.

It's interesting. I wouldn't dare to go beyond a 1500 baud signaling rate, but, then, audio tape is amenable to QAM, and that could multiply the transmission to 16 bits per token or more depending on the quality of the tape and recorder.

I would be careful with that, however. If you are archiving your data, it's because you like it and, if you like your data, you want it to be readable a long time from now. I'd suggest vinyl records rather than tapes, as they are very robust, and can be read without physical contact.

Vfx Every commercial, movie or TV worked on would have the assets used to create the content archives.

Similarly, I worked at a place where, before I joined, a system upgrade gone wrong had caused the retrieval of backup tapes stored in a metal safe, where the safe's temperature had been below the dew-point. Neither the tape cases nor the safe were sealable against moisture. This meant they had no backups of data they were required to retain for five years. And of course, the person who attempted the upgrade resigned.

there are a couple tape makers, regardless of how many companies rebadge the product. afaik, there are only 2-3 drive makers too.

but don't forget that tape doesn't make much sense (in its market) without the robotic library. there might be some off-brands that sell small libraries, but the big ones are, afaik, dominated by IBM.

Yes, they all pay royalties, and all the drives pay licensing for read heads.

I've been considering to get in to the Blu-ray backups for a while. Is there a good guide on how to organize your files in order to split it in to multiple backup disks? And how to catalogue all your physical Discs to keep track of them?

I remember about 20 years ago my friend had a huge catalogue of 100s of disks with media(anime) and he used some kind of application to keep track of where each file is located across 100s of discs in his collection. I assume that software must have improved for that sort of a thing?

Adding to this... Spinrite can re-write the bits so their charge doesn't diminish over time. There's a relevant Security Now and GRC article for those curious.

How about a background cron of diff -br copyX copyY , once per week, for each X and Y .. if they are hot/cold-accessible

Although, in my case, the original is evolving, and renaming a folder and few files makes that diff go awry.. needing manual intervention. Or maybe i need a content-based-naming - $ ln -f x123 /all/sha256-of-x123 then compare those /all

I've been reading a lot of eMMC datasheets and I see terms like "static data refresh" advertised quite a bit.

You're quite right that we have no visibility into this process, but that feels like something to bring up with the SFF Committee, who keeps the S.M.A.R.T. standard.

best is to have a filesystem that can do background bit rot scrubbing

Ah good catch. One big downside of reddit replacing forums is the default sorting does not make it easy to find late contributions like this.