Storage: The Next Big IT Frontier

Finally, a story about storage you’ll want to hear … and share with others!

Just to be clear, I’m not talking about those ergonomically designed stackable boxes or foldable hanging systems that promise to remove clutter from your lives, or your basement, or both. I’m talking about computer storage.

Yes, I know, you’re probably not interested, but you really should be. Why? Because your business thrives on data, and being able to store and interrogate more and more data in increasingly less-expensive ways is key to your success and growth.

Forget “cloud.” The next big IT frontier is computer storage, and it will change the way we use, design and consume products and IT software during the next four to 10 years.

Most of us are (or should be) aware of the ballooning costs associated with recent storage growth — driven during the past few years by things such as greater retention periods, data replication for higher availability, richer media (photos, videos, etc.), and more recently, the latest technology du jour: Big Data. But even the most ardent IT leaders give very little attention to new storage technologies compared to all the other things they have on their plates nowadays:

  • Cloud
  • Mobile
  • Service-oriented architecture (SOA)
  • Social media
  • Automaton
  • Performance
  • Virtual everything
  • Security
  • NoSQL.

But storage? To most IT professionals, computer storage is about as interesting as those ergonomically designed stackable boxes I mentioned above. We understand it has usefulness, but it’s really not going to change our lives … is it?

Actually, yes, it is.

First, we should differentiate between storage and memory:

  • Memory is the area of a computer where data is temporarily stored and executed (for example, DRAM).
  • Storage is permanent storage (Spinning Disk — or more commonly today — Flash Storage used in Solid State Drives).

The biggest difference between the two is that memory is volatile, meaning that as soon as you remove its electrical charge, all of the data it is holding instantly disappears. Storage, on the other hand, doesn’t require an electrical charge to retain its data. Therefore, it is considered non-volatile.

Computer Storage Costs On The Decline

The raw unit cost of storing 1 gigabyte of data in 1980 was US$3 million. That decreased exponentially to US$8,000 in 1990. In 2011, the cost to store the same amount was a small fraction from a couple of decades ago at a mere 3.5 cents, and it is about to drop even more.

The other difference between the two is that memory is fast and expensive while storage is slower and less expensive.

That may seem uninspiring, perhaps boring, right now, but these differences are about to change dramatically, and when they do, the line between memory and storage gets very blurred.

Before I explain how and why, let’s take a look at just how cheap storage has become — certainly compared to how much it used to cost. The raw unit cost of storing 1 gigabyte of data — 1 gigabyte equals 1,000 megabytes:

  • US$3 million
  • US$8,000
  • US$30
  • 8 cents
  • 3.5 cents

And it’s about to get much cheaper.

The main reason the price of storage has come down so significantly is the evolution of new technologies that are able to compact more and more data into a square inch (the standard measure for storage density). The current record (by Seagate) is 1 terabyte (a trillion bits) per square inch — twice that of a year ago.

The maximum available number of bits in a square inch now exceeds the number of stars in the Milky Way (200 billion to 400 billion).

We have seen similar cost benefits with memory as well. This is great for how fast we can read data from memory, but because of the volatility issue I mentioned before, it is limited because if you really need your data to survive a server failure, recycle, power outage, etc., you always have to write it out to permanent storage. Whether you do that a-synchronously or synchronously, it gets expensive from both a performance and real-cost standpoint.

But all that is about to change. Several leading technology companies are working on a type of memory that’s much faster than solid state and almost as fast as RAM. And if you also factor in the massive size expansion, the amount of memory we can affordably load into our computers and onto our computer chips will result in non-volatile memory netting out to be faster than RAM is today.

Faster and much bigger … think multiple terabytes of non-volatile memory. With this, the definitions of memory and storage start to converge. In fact, the industry is now calling this advance “memory/storage convergence.”

What we currently have is:

  • Flash (a la USB) speeds today: Approximately120 microseconds (a microsecond is 1/1,000th of a second)
  • RAM speeds today: Approximately 120 nanoseconds (a nanosecond is 1/1,000th of a microsecond)
    • Therefore RAM is 1,000 times faster than flash
  • New non-volatile storage: Approximately 220 nanoseconds
    • Two times slower than RAM, but because it can be much bigger and cheaper, it will end up being net faster than RAM for most usages because you won’t have to keep paging out to external storage when you’re working with large workloads — and today, that’s just about all we do.
    • 500 times faster than Flash. So does Flash/Solid Sate have a future? Probably not beyond 2020!

Tape backups are just about gone — or should be. Spinning disk will soon follow, replaced temporarily by Solid State/Flash. RAM and Flash will either go away or be supplemented by the next-generation non-volatile memory.

As I mentioned above, there are several technologies competing for this non-volatile memory space. The big players all have their bets on one or more of these, but pretty much all agree that while having their pick win is important to them, at the end of the day, they just need someone to win … and soon.

One of the leading contenders at the moment is a mind-boggling technology called Spintronics, an emerging technology that works at the sub-atomic level. The evolution — so far — has gone something like this:

  • Scientists were able to view electrons within an atom.
  • They were able to see that the electrons rotated.
  • They were able to automatically adjust the axis of a series of electrons in any sequence they wanted.

This meant they were able to use this positioning to represent 1s and 0s — the basic building blocks for binary data storage (all data is currently stored using a binary representation).

More Bits Than Stars In The Milky Way

The primary reason the cost of computer storage has dropped so substantially is the progression of new technologies that are able to compress massive amounts of data into a square inch. Today, the highest existing number of bits in a square inch surpasses the number of stars in the Milky Way, some 200 billion to 400 billion.

Unfortunately, the velocity that the electrons spun was too fast to be read by an electronic reader head. At this point, most of us would have said, “Ah well, there is only so much you can do. You can’t be expected to slow down the rotation of an object that ignores the laws of physics (as sub-atomic particles tend to do), so let’s call it a day.”

But rather than giving up, they persevered and just recently announced that they have been able to actually slow down the rotation of the electrons. Thereby extending the spin cycle to 1.1 nanoseconds … exactly the same speed it takes for a 1Ghz processor to cycle … which makes reading this sub-atomic data a reality!

Game changing? You bet! This isn’t simply a case of who can store data the weirdest way possible (I’ll tell you that winner in a moment). This is about massive leaps in storage density and the magnitude of storage that will be able to be stored in just about everything we own without substantially raising the price of doing so.

Like I said, today we store data compressed to 1 terabit per square inch. With Spintronics, that could go to multiple terabytes per square inch. And costs will fall accordingly.

And if that’s not amazing enough, last August, scientists at Harvard University’s Wyss Institute announced that they had been able to store 700Tb in a single gram of DNA. And before you say, “Nice, but what’s the point?” — when we talk about non-volatile storage, one of the all-time kings of non-volatile storage is DNA. Intact, samples of readable DNA structures over 100 million years old are not uncommon and some have been more than 400 million years old.

How long do you need to retain your data? Additionally, consumer-facing products (cell phones, tablets, etc.) will face a big change. Imagine having a petabyte (1,000 trillion bytes) of storage on your phone. Sites such as Wikipedia, Dictionary.com, Google Maps, etc., could all be served up locally from your phone. No network required apart from occasional updates. Apart from anything else, this means that more attention can be given to enhancing the intelligence and user experience (the final frontier of product differentiation) around these apps. Perhaps they can always be listening and have the words and definitions readily displayed as soon as you say them. Wikipedia could alert you when you incorrectly name the England world cup squad from 1966. How cool (or annoying) would that be?

Travel Web sites could pre-store all of their static photo content on your local device so response times would be blazing fast as they’d just need to return the variable data, such as prices. Or maybe even those could be crunched locally and only availability changes would need to be sent to you.

Relational database will no longer be the default way of storing or representing data. In fairness, this won’t just be thanks to non-volatile memory advances. Big Data technologies such as Map Reduce and NoSQL will team up with non-volatile memory to make that happen. Relational database management systems (RDBMs) will still be the fastest way to retrieve (not add) data, but with advances in Moore’s law; thousand core servers; faster, cheaper, bigger, persistent (non-volatile) memory combined with Big Data tools, the performance difference between the two will — in most cases — become insignificant for some, and then most, data uses. And when this tipping point occurs, people will overwhelmingly choose non-relational databases because of their low cost, simplicity and flexibility.

The Technology Behind Travel

Hear what Sabre Holdings Senior Vice President and Chief Technology Officer Robert Wiseman, along with several other industry experts, has to say about the technology behind travel.

Table of Contents