Data centers contain 90% crap data

> The real question is, how much data do we need to store inefficiently, in a way that uses a lot of power and space?

This is indeed the critical question, and it's far from being trivial.

One issue we all hit is moving the data from the higher tier storage to the cheaper and more efficient one, which requires sync and paying for the transfer most of the time, but also handling two separate access and authorization process, backup and recovery system for data that absolutely needs to be accessible for the few years of legal retention, and can/must completely disappear afterwards.

In most orgs I've seen the cost of going through all that complexity is just not worth it, compared to "just" paying for the higher tier storage for the few years long lifetime of the data.

  probably a similar percentage of people claim on their car insurance. 
  In that 0.01% of cases where the order was fraudulent, placed by mistake, 
  how much data do we need to store inefficiently, in a way that uses a lot of power and space?

The winning ticket ID, e.g. a "0x-636e-7461-4e7b", only makes sense in the context as one among the entire cohort of contestants; I can make one up like I did, but I can't walkout with the payout unless the rest of the lottery didn't exist.

Statistically, philosophically, technically, and all sorts of *-cally speaking, is the 2% data, the winning ticket datum, even data?

I was just pondering this today, in terms of how much data and objects I create in a Google or Microsoft account, for example, and then they create a burden of cost and maintenance for me down the road. Especially deleting old emails and photos. That's arduous and sometimes poignant as I flush away my personal life story.

Cloud services make difficult and sometimes even byzantine processes for deleting stuff, and it's often impossible to operate en masse in order to clean up swaths of stuff quickly and efficiently. It's in their interest to retain everything at all costs, because more used storage can mean more profits. Cloud services also profit from unused storage, because if they're charging $20/year to 100,000 users who use 2% of their storage space, ka-ching!

It irks me to this day that standard or even advanced filesystems don't include "expiration dates" or "purge dates" on file objects. Wouldn't it be logical, if an organization has a "data-rentention policy" that mandates destruction after X date, that the filesystem simply purges it automatically? Why does this always get delegated to handmade userland cron jobs? Moreover, to my knowledge, nobody is really interested in devising a way to comb through backup media in order to [selectively] destroy data that shouldn't exist anymore. Not even the read-write media!

Google is now auto-deleting stuff like OTP SMS messages. I'd love it if auto-delete could be configurable account-wide, for more than just our web histories and Maps Timeline and stuff. Unfortunately, to "delete" cloud data means it still exists on backups anyway. But without deleting data in your cloud account, it becomes a juicier hacker target as it ages and accumulates personal stuff that shouldn't fall into the wrong hands. Likewise for any business, it behooves them to delete and destroy data that shouldn't be stolen. At least move it offline so that only physical access can restore it?

I will say that modern encryption techniques can make it easy to "destroy" data, simply by destroying the encryption keys. You can quickly render entire SSDs unreadable in the firmware itself with such a command. Bonus: sometimes it's even done on purpose!

But even deleting data presents a maintenance cost. So if 90% of an org's data is indeed crap, then 90% or more of your processing resources are going to be wasted on sifting through it at some later date. Imagine when your file formats and storage devices are obsolete, and some grunt needs to retrieve some record that's 30 years old, and 90% of your data was always crap. That grunt is hopefully paid by the hour. We really had this happen at a few of my jobs, where we had old reel-to-reel backup tapes and it was difficult enough to load the data into a modern SunOS machine.

https://m.xkcd.com/1683/

I do cull photos and docs but it takes effort and you will make mistakes. On the other hand, intelligent culling makes it easier to find things.

It’s a trade off especially with digital where there’s not that much cost associated with holding onto things relative to the physical world. My whole house is basically in storage because of a kitchen fire and I’m planning to be pretty selective about what I move back in.

Sometimes I wish there was a feasible system to audit and reduce redundant photos in my iCloud. I have an embarassing ton of pictures that I never look at just like everyone else. I wouldn’t mind a tool that said “here’s 10^x photos that are the most similar to all your other photos would you like to put them in the trash?”

I have about the same in iCloud and then dozens of drives of video footage. Beyond the "keep it in case you need it" aspect, another angle is that it's often cheaper to just keep everything than spend all the time it would take to do a thorough cull. Video is much slower to review, but also chews up far more space, so maybe that balances out.

Crap is in the eye of the beholder...

Offline storage is only easy on the very small and the very large scale. On the very small scale you can get archival-grade CD/DVD/Blurays. On the very large scale you dedicate climate-controlled rooms to an automatic tape library.

In between is a vast gulf where your only good options are disks that you have to occasionally spin up to check integrity, have the disk trigger a data refresh if it's an SSD, and replace any disks that failed and rebuild their data from redundancy (raid or whatever you prefer). HDDs die even when off [1] and flash storage in SSDs is only rated to hold data for three years without power.

Sure, you can roll this yourself, but it can easily go wrong. Easier to either keep the disks spinning or pay someone with a tape archive (e.g. AWS deep glacier)

1 https://www.tomshardware.com/pc-components/storage/twenty-pe...

Doing it in parallel, which of course is only an option if you know about it. And templating that into a general approach and having legal folks sign off on that.

We had a similar request for a sales team once: "If this fails we want an email".... Okay, but we never seen this fail, so you'll never get the email. As expected, they never got an email, because no failures. So instead they wanted an email every time the job ran (once every night), so they'd know that the job had failed, if they DIDN'T get the email. Only time that happened was because the email got trapped in a spamfilter I think.

There must be thousands of copies of that email sitting in inboxes say: Job X ran successfully @ 04:30.

> The requests are endless and I'm convinced there are people who if they could would do their entire job from their inbox and get everything and anything an application can do via email.

That sounds like a reasonable goal for a whole lot of job duties. And yes some entire office jobs. (Excluding some direct human communication but a lot of jobs already have too much of that in situations that could have been an email.)

> "I didn't get the email!" is a constant refrain (yes they did, they always did).

Well having to manually check wouldn't improve that, would it?

I can see how that would work (email centric workflow), not unlike how some people now try to have a chat / Slack centric workflow.

> Probably one reason why free providers don't tend to offer End-to-End encryption.

I would think that email providers tend to not offer E2EE just because it fundamentally isn't practical with email. Providers like Proton try to do it, but it works only if you talk to someone else on Proton (at which point you may as well do it on Signal, which has better encryption).

Nope.

Back in the day Exchange offered SIS but in 2010 they ditched it. It's plainly not effective any more. Even regarding the OP' "the jpg signature logo" - it's a part of multipart in the message, not a separate file.

And one more thing - you can't just turn the dedup and be dandy, now you need to check against the hashes to determine if this chunk is unique or you already have it. And with TBs of data you need TBs of hashes to check. Until you have like 99% dedup efficiency, ie 99% of your incoming data is literally the same data you already have - it doesn't worth it.

https://techcommunity.microsoft.com/blog/exchange/dude-where...

I should have been less specific than “library” as such. Libraries aren’t only institutional. Consider every book, magazine, and newspaper that sits today in everyone’s home, office, and in every institution in the world. The vast majority of them have been read once and then left on a shelf or in a box somewhere, taking up more or less valuable space.

As a librarian, that's only around 35% of new books. The rest are about a hard-working professional woman in the big city that inherits her dead aunt's coffee shop in Cape Cod and has a prickly love-hate relationship with the local handyman. There may be a magic cat involved. And books about vampires having love triangles with a werewolf/zombie/black lagoon monster and a Mary Sue insert.