Fibre Channel Really Is Dead

Last fall's release of a routable version of Remote Direct Memory Access over Converged Ethernet (RoCE) has led to a sea change in network convergence and data center architecture. This new routable RoCE specification overcomes the three problems that have prevented widespread adoption of converged compute and storage networks: 

1. Versatility: True network convergence requires a versatile network that offers unified support for block, file, and object based storage, as well as compute and network traffic.  Furthermore, in order to share a single physical network, this network must efficiently provide isolation and quality of service.
2. Routability: The Fibre Channel over Ethernet (FCoE) protocol promised convergence, but failed to deliver, in part because its support was limited to layer 2 environments, which constricted scalability, preventing true convergence and unification of the compute and storage networks.
3. Performance: While iSCSI is routable on layer 3 networks, poor performance and high CPU utilization has prevented customers from moving away from Fibre Channel.

The new routable RoCE technology overcomes all of these problems and the release of the specification enabling multi-vendor, interoperable solutions signals the demise of Fibre Channel.

Pundits have long predicted the demise of Fibre Channel, but this time it's real. In fact, the decline has already begun and now that RoCE is routable, this decline will accelerate. The recent Dell’ Oro SAN report highlighted a 3% year over year decline in Fibre Channel revenues, citing the need for “distributed scale-out and object-based storage” and a preference for "Ethernet speeds and converged infrastructure.”

While the decline of Fibre Channel in the enterprise market has just begun, Fibre Channel is already dead in other markets, including the fast-growing cloud and Web 2.0 segments. Major providers have already adopted converged infrastructure and scale-out storage to cope with massive scale, virtualization, mobility, and the explosion of unstructured data. And this scale-out storage has converged on either InfiniBand or Ethernet. But Fibre Channel is nowhere to be seen.

Server SANs

Wikibon defines this new cloud architecture, which combines compute and pooled storage running over a converged network, as “server SAN.” Wikibon  predicts that over the next 15 years traditional SAN storage will decline at a -17.8% CAGR and be rapidly displaced by server SAN storage, which will grow at a 44.2% CAGR.

This server SAN converged network has been deployed in all major public clouds, including Azure. The cloud has been at the vanguard of server SAN because public cloud providers benefit most from the improved cost and efficiency of a converged compute, networking and storage infrastructure. Wikibon predicts that this converged server SAN will grow dramatically over the next five years to dominate the enterprise market too.

RoCE

RoCE will accelerate the adoption of server SAN as it overcomes the three limitations discussed above that have slowed adoption of a converged network. First, RoCE supports block-, file-, and object-based storage protocols and is being deployed at massive scale. RoCE-based distributed file systems include Microsoft’s SMB-Direct, Hadoop/HDFS, GPFS, and Gluster. In addition, RoCE supports hybrid systems such as Ceph and RDMA-enabled iSCSI block storage.

Second, RoCE is routable. FCoE arrived with great fanfare in 2007 but never lived up to its expectations. The layer 2 restriction of FCoE is fundamental because it is implicit in the underlying Fibre Channel technology that FCoE simply encapsulates. RoCE, launched in 2010, was also initially to layer 2 networks, but in contrast, the underlying technology included network addressing, enabling RoCE to quickly become routable across layer 3 networks. In fact, the largest installations have deployed RoCE at scale in a fully routable environment with hundreds of thousands of nodes shipping each quarter.

Finally, Ethernet link speeds are 2.5x that of legacy Fibre Channel technology. And now, with 100 Gigabit Ethernet available, this advantage will only increase, and the Remote Direct Memory Access  (RDMA) technology at the heart of RoCE delivers unmatched block-, object-, and file-based storage performance.

Figure 1:

For example, Microsoft SMB Direct file system supports both block and file based applications in both virtualized and bare metal environments. Microsoft has demonstrated 10x faster Live Migration and 4x SQL Server performance improvements using SMB Direct over RoCE storage.

In addition, big data workloads such as Hadoop can achieve up to 2x performance improvements using RoCE-based, RDMA-enabled file systems like HDFS, GPFS, and Ceph.

RoCE also supports the industry standard iSCSI extensions for RDMA (iSER) which delivers a “turbo-iSCSI” block storage solution. This protocol leverages RoCE to deliver transparent performance advantages to all block-based applications and accelerate traditional enterprise workloads, including database, app/web server, CRM, analytics, and email.

RoCE has now joined InfiniBand to enable a unified, converged network. This convergence can already be found in the cloud with large scale deployments over both RoCE and InfiniBand. These hyper-scale deployments act as bellwethers and pave the way for the enterprise transition to a converged infrastructure. Fibre Channel is dead, and RoCE has put the last nail in the coffin.