Disk can lie to you when you write to it
18 comments
·December 12, 2025jmpman
I’ve seen disks do off track writes, dropped writes due to write channel failures, and dropped writes due to the media having been literally scrubbed off the platter previously. You need LBA seeded CRC to catch these failures along with a number of other checks. I get excited when people write about this in the industry. They’re extremely interesting failure modes that I’ve been lucky enough to have been exposed to, at volume, for a large fraction of my career.
kami23
I worked with a greybeard that instilled in me that when we were about to do some RAID maintenance that we would always run sync twice. The second to make sure it immediately returns. And I added a third for my own anxiety.
zabzonk
it's not just a good idea for raid
null
jandrewrogers
People consistently underestimate the many ways in which storage can and will fail in the wild.
The most vexing storage failure is phantom writes. A disk read returns a "valid" page, just not the last written/fsync-ed version of that page. Reliably detecting this case is very expensive, particularly on large storage volumes, so it is rarely done for storage where performance is paramount.
formerly_proven
Not that uncommon failure mode for some SSDs, unclean shutdown is like a dice roll for some of them: maybe you get what you wrote five seconds ago, maybe you get a snapshot of a couple hours ago.
jmpman
https://en.wikipedia.org/wiki/Data_Integrity_Field
This, along with RAID-1, is probably sufficient to catch the majority of errors. But realize that these are just probabilities - if the failure can happen on the first drive, it can also happen on the second. A merkle tree is commonly used to also protect against these scenarios.
Notice that using something like RAID-5 can result in data corruption migrating throughout the stripe when using certain write algorithms
jmpman
The paranoid would also follow the write with a read command, setting the SCSI FUA (forced unit access) bit, requiring the disk to read from the physical media, and confirming the data is really written to that rotating rust. Trying to do similar in SATA or with NVMe drives might be more complicated, or maybe impossible. That’s the method to ensure your data is actually written to viable media and can be subsequently read.
compressedgas
I thought an fsync on the containing directories of each of the logs was needed to ensure the that newly created files were durably present in the directories.
jtregunna
Right, you do need to fsync when creating new files to ensure the directory entry is durable. However, WAL files are typically created once and then appended to for their lifetime, so the directory fsync is only needed at file creation time, not during normal operations.
breakingcups
> Conclusion
> A production-grade WAL isn't just code, it's a contract.
I hate that I'm now suspicious of this formulation.
nmilo
You’re not insane. This is definitely AI.
jtregunna
In what sense? The phrasing is just a generalization, production-grade anything needs consideration of the needs and goals of the project.
rogerrogerr
“<x> isn’t just <y>, it’s <z>” is an AI smell.
joecool1029
Flashback to this old thread about SSD vendors lying about FLUSH'd writes: https://news.ycombinator.com/item?id=38371307 (I have a SKHynix drive with this issue)
jeffbee
This FAST '08 paper "Parity Lost and Parity Regained" is still the one I pull out and show people if they seem to be under-imagining all the crimes an HDD can do.
https://www.usenix.org/legacy/event/fast08/tech/full_papers/...
eatonphil
Seconded.
Check out Parity Lost and Parity Regained and Characteristics, Impact, and Tolerance of Partial Disk Failures (which this blog indirectly cites) if you'd like authoritative reading on the topic.
https://www.usenix.org/legacy/event/fast08/tech/full_papers/...
https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&d...