8 Data Storage Industry Milestones

The data storage industry is moving into an era of intense evolution. It’s worth taking a look back at the most important steps in storage before they are buried by the sands of time!

Let’s look at ancient history first. While many would point to IBM’s Winchester disk architecture as the major trigger for the storage industry’s “Big Bang” of growth, I believe the humble floppy disk (for which I wrote the ECMA industry standard!) had more impact. It triggered the PC industry, which triggered demand for huge volumes of hard drives, creating the economics of the current storage industry. PCs also fostered network-attached storage, which led us to a product line-up lasting three decades that included NAS, SAN, and RAID.

With fierce competition in the '90s, we searched for new, smaller drive form-factors, going from 10-inch drives to 3.25-inch diameter platters in just eight years. Nobel-prize winning technologies, such as giant magnetoresistance (GMR), simultaneously allowed rapid capacity growth. In parallel, interfaces based on SCSI evolved to serial links ranging from Fibre Channel to SAS and SATA. In storage, the '90s could be characterized as bigger capacity, but cheaper.

Then RAID and NAS matured and the industry settled into a sort of middle age. Nothing too exciting happened until 2005, when object storage arrived, followed closely by solid-state drives in 2008. These were game changers, especially when cloud computing joined the picture.

Today, we are looking at Ethernet RDMA interfaces replacing Fibre Channel, universal storage based on object-storage software cores that are abstracted from the underlying storage hardware into virtual machines or containers, and blindingly fast SSDs with larger capacity than any hard drive.

Continue on to look back at the storage technologies that have shaped the industry and helped fuel today's innovation.

(Image: Pobytov/iStockphoto)

SCSI interfaces

The evolution of interfaces played a critical role in storage growth. SCSI dominated the '80s, and its offshoots, Fibre Channel and SAS/SATA, continue to be a major part of storage. SCSI provided a single solution for interfacing drives that was completely interchangeable. It simplified host-system design and opened up the move towards today’s COTS products. What's less well-known is that SCSI also was crucial for increasing drive capacity, since it had a mechanism for masking bad sectors from the server and drives no longer had to be perfect.

(Image: Amazon)

Drive capacity

The breakthrough in capacity begin around the turn of the century, with perpendicular recording and giant magnetoresistive (GMR) recording heads allowing capacities on 3.5” drives to pass 1 TB.  Then came the 2.5” form-factor, which opened up the portable PC market. The last five years has seen capacity growth flatten off, as mechanical technologies hit barriers that may be insurmountable. New approaches in the pipeline offer limited improvement and so doubling capacity every year or so is a thing of the past.

The unintended consequence of rapid capacity growth has been a radical drop in IOPS per gigabyte; accessing data slowed down even as prices fell at over 30% CAGR.

(Image: JasonPinaster/Pixabay)

RAID and SAN

RAID technology helped boost performance, along with protecting data. The idea of striping data over a set of drives with parity blocks added meant that mechanical failures no longer implied lost data. That protection has made much of electronic commerce feasible and to a great extent this accounts for the stellar growth of IT and perhaps for the world economy in general. Expanding the idea to a storage-area network using Fibre Channel to deliver block-IO just like local drives made became a huge market segment.

Arriving in the late-'80s, and famously sold by EMC in SANs, RAID has powered the industry, but even it is faltering in the face of huge capacity drives. Rebuild times have reached the point where a second failure in the array is likely to occur at some point during a rebuild, which would lose data.

(Image: Raysonho/Wikimedia Commons)

Solid-state drives

Solid-state drives arrived in 2008 to solve the IOPS issue. With no moving parts, SSDs access data at much higher speeds than any hard drive can achieve, exceeding the ability of the RAID approach to keep up. The technology took some time to mature, but now matches hard drives in robustness, with write wear out a thing of the past.

Today, SSDs are cheaper than enterprise hard drives and much faster. It is only a matter of a couple of years and SSDs will cost even less than bulk hard drives. Clearly, solid-state is the wave of the future and we have already passed “peak hard drive.”

(Image: YuriyVlasenko/iStockphoto)

NAS

Network attached storage has had a long run, from its early days with the launch of Novell Netware and NCR Modus in 1984. NetApp leads the market in mission-critical NAS storage, but a substantial number of IT shops also use NAS servers based on open-source solutions for departmental computing.

NAS and SAN have battled for market share for decades, but an upstart new approach, object storage, appeared around 2005. Object storage, which allows massive scaling, arrived as the cloud was altering IT forever and creating a need for that scale-out ability.

(Image: NetApp)

NVMe

SCSI and all of its derivatives are about to leave the stage. NVMe is a new interfacing scheme designed for non-volatile memory that handles millions of IOPS while dramatically reducing server overhead. This is the wave of the future in interfaces and it will replace the other choices over time. Even Fibre Channel SANs are not immune since NVMe over Ethernet (using RDMA) is compelling. Drives with native Ethernet interfaces are already in the market.

(Image: Samsung)

Software-defined storage

Bushels of Ethernet drives and appliances need automated management. The industry is heading towards software-defined storage (SDS), which virtualizes almost all the storage code and separates it from hardware. As with servers, this is a very agile approach with implications for scale and flexibility. Coupled with a lot of open-source software, we can expect storage in a few years to evolve to a low-cost market with a la carte software.

SDS is just part of the software-defined infrastructure. Linked with networking elements, the SDI approach all but guarantees that the storage network of the future will be Ethernet based. Fibre Channel will fade away.

(Image: nevarpp/iStockphoto)

The cloud

The cloud’s impact is not so much on actual storage -- data has to have a home, after all. Rather,  it’s on the logistics of delivering storage capacity and access. Already displacing in-house backup and archiving, the cloud offers peak-load handling (cloud-bursting) and a place for non-critical applications.

The cloud approach is driving in-house thinking, too and will likely be the preferred approach within five or 10 years for larger companies that will configure their data centers around a hybrid cloud model. Small businesses will most likely use  public clouds like AWS, Azure or Google, removing the need for internal infrastructure management.

(Image: WolfBlur/Pixabay)