Are hard drives getting better?

Over the past couple of years, I've been side hustling a project that requires buying ingredients from multiple vendors. The quantities never work out 1:1, so some ingredients from the first order get used with some from a new order from a different vendor. Each item has its own batch number which when used together for the final product yields a batch number on my end. I logged my batch number with the batch number for each of the ingredients in my product. As a solo person, it is a mountain of work, but nerdy me goes to that effort.

I'd assume that a drive manufacture does similar knowing which batch from which vendor the magnets, grease, or silicon all comes from. You hope you never need to use these records to do any kind of forensic research, but the one time you do need it makes a huge difference. So many people doing similar products that I do look at me with a tilted head while their eyes go wide and glaze over as if I'm speaking an alien language discussing lineage tracking.

I think it's helpful to put on our statistics hats when looking at data like this... We have some observed values and a number of available covariates, which, perhaps, help explain the observed variability. Some legitimate sources of variation (eg, proximity to cooling in the NFS box, whether the hard drive was dropped as a child, stray cosmic rays) will remain obscured to us - we cannot fully explain all the variation. But when we average over more instances, those unexplainable sources of variation are captured as a residual to the explanations we can make, given the avialable covariates. The averaging acts a kind of low-pass filter over the data, which helps reveal meaningful trends.

Meanwhile, if we slice the data up three ways to hell and back, /all/ we see is unexplainable variation - every point is unique.

This is where PCA is helpful - given our set of covariates, what combination of variables best explain the variation, and how much of the residual remains? If there's a lot of residual, we should look for other covariates. If it's a tiny residual, we don't care, and can work on optimizing the known major axes.

Agreed, these type of analyses benefit from grouping by cohort years. Standard practice in analytics.

Right. Does the trouble at year 8 reflect bad manufacturing 8 years ago?

> replacement HDD that I could use for another 5 years

But the warranty lasts only 5 years since the purchase of the drive, doesn't it?

Get yourself a Xeon powered workstation that supports at least 4 drives. One will be your boot system drive and three or more will be a ZFS mirror. You will use ECC RAM (hence Xeon). I bought a Lenovo workstation like this for $35 on eBay.

ZFS with a three way mirror will be incredibly unlikely to fail. You only need one drive for your data to survive.

Then get a second setup exactly like this for your backup server. I use rsnapshot for that.

For your third copy you can use S3 like a block device, which means you can use an encrypted file system. Use FreeBSD for your base OS.

If you don't have too much stuff, you could probably do ok with mirroring across N+1 (distributed) disks, where N is enough that you're comfortable. Monitor for failure/pre-failure indicators and replace promptly.

When building up initially, make a point of trying to stagger purchases and service entry dates. After that, chances are failures will be staggered as well, so you naturally get staggered service entry dates. You can likely hit better than 5 year time in service if you run until failure, and don't accumulate much additional storage.

But I just did a 5 year replacement, so I dunno. Not a whole lot of work to replace disks that work.

One method that seems appealing:

1. Use ZFS with raidz

2. Scrub regularly to catch the bitrot

3. Park a small reasonably low-power computer at a friend's house across town or somewhere a little further out -- it can be single-disk or raidz1. Send ZFS snapshots to it using Tailscale or whatever. (And scrub that regularly, too.)

4. Bring over pizza or something from time to time.

As to brands: This method is independent of brand or distro.

Hard drive failure seems like more of a cost and annoyance problem than a data preservation issue. Even with incredible reliability you still need backups if your house burns down. And if you have a backup system then drive failure matters little.

> 2- burning to M-disk, or

You can't buy those anymore. I've tried.

IIRC, the things currently marketed as MDisc are just regular BD-R discs (perhaps made to a higher standard, and maybe with a slower write speed programmed into them, but still regular BD-Rs).

Would tapes not be an option?

Not great for easy read access but other than that it might be decent storage.

The question is, do we have enough capacity to mine and refine them at a reasonable price? They're there, in the dirt for the taking.