I was having a chat the other day with a senior exec at one of the cloud storage vendors. He said they were closing a big deal with a service provider to OEM their services as the service provider's other option, which involved a major vendor's private cloud hardware. I started wondering how it was that a huge service provider, with lots of smart engineers and the economies of scale of building a multi-petabyte system couldn't get their costs down to competitive with a third party, who is of course marking up their own costs.
Then I realized that having worked for start-ups, SMEs, and underfunded state colleges for 20 years, I knew a lot more about building reliable systems at minimal cost than the folks at big corporations. Over at Engulf and Devour they value consistency and reliability over cost effective and "good enough." I remember one other consultant telling me I had to stop designing systems like it was my own money.
The first place I see corporations spending extra money is making every element of their system as reliable as possible even though they've designed the system as a whole to be fault tolerant. So they use DL380s with extra cost RAID controllers and redundant power supplies for the nodes in their RAIN architecture private cloud when the underlying software, say ParaScale, is designed to handle node failures without data loss or service disruption.
Let's be clear here. I wouldn't go as far Backblaze and build my own storage system with rubber bands holding the SATA drives in place. I would get SuperMicro servers with their 24-bay 4U chassis a combo that prices out at Newegg for about $1.50/GB. Add in ParaScale's software and object replication space and total cost is under $8/GB for three copies across the cluster. I can also frequently squeeze some money out of the budget by substituting spare parts for service contracts. I routinely see clients paying 15 percent or more of the cost of their hardware for service contracts on devices they own many of. If you have 100 Ethernet switches or servers, and five percent fail each year, it's way cheaper to buy spares and throw them out when they die than to pay for service.
When an edge switch or server fails, you don't wait for the service guy to come in four hours before replacing it anyway, do you? You swap it out and then the fellow comes to fix it. Since most of the cost is in sending the guy to your site, I'd like to see vendors offer a depot service contract that keeps giving me software updates, and I send the dead switch or server to them to repair. That should cost 5 percent, not 15-20 percent, of the cost of the device.
