It's been a long time since anyone at Network Computing, or anyone among our erstwhile competitors, has talked seriously about KVM switches, power distribution, cable management or any of the other little things that keep a data center running smoothly. Since we just moved DeepStorage labs into a new data center, I figured I'd share what I learned in the process and describe what I'd do the next time I get to build a small-to-medium size data center from scratch. I'll start with power conditioning.
Over the years, my consulting clients have used just about every conceivable method for conditioning and distributing power in their data centers. While most large scale data centers have followed a similar model with large freestanding UPSes and separate battery rooms, those of us who live in data centers with up to 50-plus racks of gear have gone in various directions. I've worked in data centers with power systems from Smart-UPSes in each rack to Liebert enterprise UPSes, including modular systems, flywheel UPSes and even a UPS-fuel cell lash-up that MGE sold for a few weeks in 2005.
I can always tell a data center built by computer guys that are building something bigger than they've ever worked on before, as that's where I find a pair of 3KVA Smart-UPSes in the bottom of each rack. They have the electrician run a 30 amp circuit to an L5-30 recepticle for each UPS, and then connect one power supply for each server to each of the UPSes. This means the load on each UPS has to be kept under 50 percent of its capacity, since it may have to handle the full load if its partner goes off line.
The worst case of UPS sprawl was at a site where nobody realized that even though the running load was half the rated output of the UPSes, they would draw significantly more power when charging their batteries. When we added a backup generator, we carefully checked that all the UPSes had fully charged batteries and cut the power for five minutes to connect in the transfer switch. When we restored power, the UPSes all started charging their batteries and blew the main breaker for the room.
Clearly one big UPS is a better and more efficient solution than a lot of little UPSes. The question, like with storage, is do you go modular or monolithic? Since I'm frequently working with fast growing or otherwise dynamic environments I'll go modular every time. A modular UPS, like APC's Symmetra PX or Eaton's BladeUPS, allows me to buy a 20 or 24KVA UPS and grow it over 100 percent in the same footprint. They also include internal redundancy so inverter or rectifier modules can fail, and be hot swapped, just like server power supplies. Add in that they come in a standard size rack so my rows still line up. Now I just have to figure out how to get the power from the UPS to my servers and storage gear.
