To understand how we’ve become ‘data hoarders’ in the application economy let me take you back to 1979.

Back then, a 250MB hard drive weighed about 550lbs, roughly equivalent to that of a lion or a green anaconda, and cost tens of thousands of dollars.

Fast-forward to 2013. A 16GB micro SD card holds about 64 times the data as the HD above, weighs less than half a gram and costs $11.

With all of this capacity at our fingertips 24-7 on our smartphones, we’ve become data hoarders. We want everything we own digitally to be accessible anytime, anywhere without deleting anything.

Have you ever deleted a bunch of emails or some pictures as a part of a ‘digital cleanup’ of your belongings? The chances are no or very rarely. For many of us, the thought of not having some information or data at our fingertips is a frightening one.

As digitization and processing power have increased, storage costs have decreased. We can justify the cost of storing more and never cleaning up.

Everyone is collecting data around us including us

Similar to our behavior with our devices, the Internet also has a hording culture.

Google has shown us the power of data hoarding. Google remembers more about us than we do (it knows what I searched for nine years ago).

Amazon also shows us what we might want based on the data collected from us visiting its site.

This hoarding mentality also extends to government regulations such as Sarbanes Oxley, which mandates that certain records be retained for seven years.

The role of Big Data and the Internet of Things

In recent years, Big Data and the Internet of Things (IoT) have done their part in adding to the total data collected and stored.

The reason for doing this is very clear. It’s the benefit you get by analyzing the stored data compared with the cost of storing the additional data.

Data analytics promises businesses the much-needed insight into their customers’ habits and what they will need when. It’s impossible to fight that urge as it impacts the top line.

And it is good for us – the individuals – too. Everyone wins in this Big Data market – though businesses win a tad bit more.

Mind-boggling data

IoT  is a new entrant in the list of data creators of the application economy.

Sensors in your car, street lights and office rooms all generate data. The estimate for the number of devices connected in the IoT is in the trillions. If each device generates 1GB of data in a year, it will amount to 100 Zettabytes

And that is just the data on top of what we have today. Then there is the meta-data about all this data we have been collecting. The result is mind-boggling to say the least.

As Walmart demonstrated, conducting a relatively simple data analysis requires a lot of processing. Imagine trying to digest Yottabytes (1,024 Zettabytes) of data, analyzing it and making sense out of it.

Going forward, it will only get increasingly challenging, time consuming and power consuming.

What are we keeping it for?

The question before us now – is all the data that we are storing relevant? The fact that a particular airplane part had couple of failures in a row nine years back, but since then there have been no mass failures from that manufacturer might not be relevant now.

Just like we have developed ways to digitize, create, store and analyze data, we now need to create ways destroy data. If we don’t, there will be too much data in the future, which will make it difficult for us to get to the potential value of this data.

Then there is the privacy issue of all this data being created and stored forever. If, for example, criminals can get the exact location of your car from Goggle Waze overlaid with Facebook and OnStar, they would know exactly where to hijack your car.

Then there is the aspect of the stored data acting like a ghost from the past that never goes away. All this points to the fact that there’s a need to be able to manage and clean up the data, along with an analysis service that can estimate the value of the data stored.

If it turns out that the value is close to zero, then it’s a prime candidate for purging.  The service can look at the age of the data, the current role it is playing in the analytics, and if it has personal information, and then it can figure out the value of it either for purging it or masking it.

What I’m suggesting is that we need a 1-800-GOT-JUNK for our global data. It’s only a matter of time that we will have services like this being offered by cloud vendors and database vendors.

Until then, I think I’ll hold onto all 3,056 of my pictures on my smartphone.

The post How we’ve become ‘data hoarders’ and why we need a cleanup appeared first on Highlight.