12 years of Backblaze data center storage drives, visualized

sega_sai

It would be nice if it was just a conventional graph...

ziddoap

Data from https://www.backblaze.com/cloud-storage/resources/hard-drive... if you want to make one.

bddicken

[OP]

Backblaze writes about these data sets, and includes some more conventional graphs. For example:

https://www.backblaze.com/blog/backblaze-drive-stats-for-202...

I wanted to do something more fun!

null

[deleted]

dylan604

[flagged]

null

[deleted]

atYevP

Yev from Backblaze here -> Y'all this is one of our favorite things and I've spread it internally. So cool <3

bddicken

OP here - thank you! I'd actually like to more in the future with your data sets. It's so cool that you publish this.

atYevP

Love that! I think I re-shared your tweet about it a few days ago! I've been kinda staring at it on a loop on my monitor, fun to watch the balls bouncing around :D

amelius

Why doesn't it show a pile of failed drives?

toss1

YES

And I'd want to see failed drives somehow organized by TimeInService and maybe origin...

We of course expect their drive usage to grow, but what would be surprising (& provide more info) is how the drives fail or age-out. None of us without huge data centers can get that kind of info

bobbob1921

Exactly, I’m. I was hoping to see all time drive failure data as well.

bddicken

I'd like to do another vis that includes failed drives. Please, keep helping me brainstorm.

FlyingAvatar

Yeah, I think it would be a cooler visualization if the drives were in a line instead of a circle and new drives are added on to the right. The failed drives pile up on the bottom.

Still fun to watch as it is, though.

RedShift1

What exactly is this visualizing? Does each dot represent a chunk of data?

halper

1 small node is a 100 drives. So the small circles represent 100 drives each, I think. Not sure what they … do, though.

RedShift1

Maybe the amount of drives they purchased?

florbo

I'm assuming it's the acquisition and removal of drives over time.

Piraty

> 1 small node -> 100 drives

NKosmatos

Nice one, we can see the (logical) shift to bigger drives. One small comment if I may, after 2020-2021 it gets really crowded with the dots and the number of drives leading to a loss of overall picture ;-)

bddicken

Thats good feedback. It definitely gets crowded, and sluggish due to the number of dom nodes. I didn't spend much time optimizing performance.

joshka

there's a few places where all the dots seem to drop (guessing there's some discontinuities in the data?)

There's also a few places where there's duplicate labels (e.g. Hitachi 3TB)

Would be great to group by manufacturer somehow (e.g. color) and make the size more prominent.

Very cool visualization regardless.

Damogran6

I took that to be massive decomming of drives

null

[deleted]

bddicken

Yeah, when drives fall off the bottom that represents decommissioning.

joshka

No, it's an obvious data error. The same drives reappear in future months.

Pause and look at 2013-08-01 through 2013-11-01, or 2015-09-01 through 2016-01-01 to confirm this.

vladde

The data takes a while to load – a dozen megabytes of data or so. After a while the visualization loaded.

(It loaded about 0.3MB/s for me)

fencepost

Probably wouldn't be that difficult to organize down the y axis based on drive capacity, and the amount of pointless jostling around of small nodes makes it noticeably bog down as the years go by.

Interesting visualization though.

BadJo0Jo0

Fun to look at! Since I also had a mini project, that utilized this data. Sadly, haven't maintained it in awhile. It's a Show HN on my profile if you're curious.

I hope you had a better time with ingesting the data than I did :)

bddicken

I actually built this primarily with chatGPT o1.

One of the things LLMS are really good at is writing scripts for processing and pairing down data. I wanna do a blog post talking about how did some of this, maybe coming up!

pcurve

As a data guy, I'm not sure how useful this chart is.

But it sure is fun to look at. I enjoyed it ;-)

Not sure why it would intermittently redraw the whole scene though.. could be a Chrome thing.

pmontra

It's empty in Firefox Android. Play button, slider and nothing else.

bddicken

[OP] Interesting! I just tested it on Android/FF and it works. Could be a version-specific thing. Could also be just taking awhile to load (it has to download a 32MB json file).

mlry

I had to allow d3.js in NoScript for Fennec. Had the same intermittant redraw of the entire screen mentioned above.

sebmellen

Doesn’t appear to work on mobile safari

jrimbault

Or firefox and chromium ubuntu desktop

alwyn

Works fine for me on Firefox Linux. Interestingly took a lot longer to load in Chromium and Brave but they all work.

patchtopic

or Chrome, Brave on Linux ?

Tijdreiziger

Doesn't appear to work in Firefox or Chrome on Android either.

mvanbaak

It does, it just takes some time to download all the data

codecraze

It works on my iphone with safari

nrh

needs the ball sizes to represent the storage capacity!

pinoy420

That’s not true. Don’t shame me like that

bddicken

Great suggestion.