Unstructured data, such as imagery and video, is growing exponentially, resulting in an unprecedented number of files, objects, and exabytes of data that all must move fluidly within and between data centers, the edge, and the cloud. Organizations are struggling to deliver the performance this new type of data requires. As enterprises strain to balance performance, costs, and sustainability in their data centers, NVMe flash is quickly gaining popularity and is replacing legacy disk-based storage for high-performance workloads.
Legacy storage systems, based on hard disk drives (HDDs) and/or solid-state drives (SSDs), run on inefficient SATA and don’t deliver the high-performance today’s modern workloads, like AI/ML, require. In 2023, market forecasts predict that the cost per gigabyte of NVMe flash storage will be cheaper than the cost for higher-end HDDs.
In 2022, for the first time ever, the amount of shipped HDD capacity decreased while the ratio and capacity of flash and tape both increased. The trend is undoubtedly going in a direction where HDD is getting squeezed out of the market by flash from the top and by tape from the bottom.
Today the vast majority of total HDD capacity is already shipped as slower high-capacity HDD drives -- only confirming the trend that organizations are moving away from disk for their higher-performing workloads. Price, coupled with performance and sustainability, makes NVMe flash storage an attractive option for the data center. With the need for more performance, ESG considerations becoming a priority (Environment, Science, and Governance), and costs decreasing, it isn’t surprising that organizations are shifting more of their active workloads to flash.
Most SSDs leveraged in data centers today still use SATA, a protocol once designed to work with mechanical HDDs. However, as SSDs have become faster and faster, SATA has become increasingly inadequate. The NVMe protocol is an advancement over traditional SAS and SATA protocols because it was specifically designed for solid-state memory. This design allows host hardware and software to calculate processes simultaneously, which reduces I/O overhead, improves performance, and reduces latency.
With NVMe, it’s possible to process not only thousands, but potentially millions of concurrent operations, a massive improvement over legacy architecture. NVMe and the increased performance it delivers is already being utilized in use cases to increase read-/write-speeds for large files or to lower rendering times for high-resolution content like 4K and 8K video. Paired with RDMA, NVMe can also reduce the cost for networking infrastructure and complexity because it runs on standard Ethernet.
In addition to performance, ESG has also become an important part of the equation for many enterprises. Compared to SSDs, NVMe flash can reduce a system’s footprint due to the reduced amount of hardware required while using much less energy than HDD-based storage. This results in lower energy costs as well as carbon emissions, allowing for organizations to have high performance without high environmental impacts.
This is especially critical when considering dwindling global resources. Currently, it’s estimated that a large data center requires around 100 megawatts of power, which is the equivalent of about 80,000 households’ worth of electricity. NVMe makes that large data center less so by reducing that overall hardware footprint, putting organizations further on the track towards sustainability.
Without question, NVMe flash offers many advantages over legacy storage solutions. Current solutions on the market often require custom hardware and will not run natively in the cloud, making them difficult to scale and manage, and at the same time, customers are not able to benefit from decreasing flash costs as off-the-shelf hardware gets cheaper.
What today's data requires is a modern cloud-native distributed file and object storage platform built for the latest flash and memory technologies like NVMe and RDMA. These protocols unlock the true performance potential of flash by enabling more parallel reads and write, as well as dramatically reducing network latencies for flash storage nodes communicating via an NVMe fabric. Truly distributed storage architectures also eliminate bottlenecks related to metadata management and overhead, providing consistent low latency performance for both high bandwidth and high IOPs applications.
To keep up with the explosion of unstructured data growth, Organizations will need more performance out of their storage systems, and NVMe-flash and RDMA are poised to replace disk-based storage for high-performance workloads.
Organizations with digital transformation initiatives require faster performance, more scale, and more flexibility to operate in hybrid and multi-cloud environments, and legacy storage systems using HDDs and SSDs will struggle to meet these requirements. An investment in a modern NVMe flash storage platform will provide the performance, scale, and simplicity to power the unstructured data applications of the next decade. Organizations have the opportunity to move into the future with less complexity, high performance, sustainable practices -- and lower costs.
Tim Sherbak is the Enterprise Product and Solutions Marketing Manager at Quantum.
Related articles:
Bob Friday, Chief AI Officer for Juniper Networks, explains how the advanced technology is transforming operations.
Contact center leaders from 8x8, Awaken Intelligence, and 360insight discuss the importance of agent experience.
AI may force enterprises to rewire parts of their data centers so they are fully optimized to run such workloads. The question is do you use Ethernet or InfiniBand?
