|
We hosted the latest gear from HP, Rackable Systems and Sun in our new Green Bay Labs. While HP’s well-rounded c-Class BladeSystem took our Editor’s Choice, Sun’s massive bandwidth and Rackable’s dense configuration have wide appeal.
For our latest round of testing, we asked seven leading blade server vendors to send a chassis and four x86-based server blades with Ethernet connectivity to our new Green Bay, Wisconsin, US, Real-World Labs. Hewlett-Packard, Rackable Systems and Sun Microsystems agreed. Fujitsu and Dell declined; IBM said it had no available hardware; and Egenera begged off, saying it “doesn’t connect to iSCSI storage.”
HP sent its ProLiant c-Class BladeSystem, which for our testing comprised a 10U ProLiant c7000 chassis, two 3.2 GHz Xeon ProLiant BL460c blades, two dual-3.2 GHz Xeon ProLiant BL480c blades, and four GbE (Gigabit Ethernet) 2c Ethernet blade switches for the c-Class BladeSystem.
Rackable Systems provided five of its Scale Out Blade Servers—four dual 2.2 GHz Low-Power Opteron 275 HE Scale Out Blade Servers and a dual 2.6 GHz Opteron 285-based Scale Out Blade Server.
Sun delivered its recently-released Sun Blade 8000 Modular System, which included a 19U Sun Blade 8000 Chassis, four full-height quad-2.6 GHz Opteron 885 Sun Blade X8400 server modules, two Sun Blade 8000 GbE 20-port Network Express modules, four PCIe Dual-GbE ExpressModules, two PCIe Dual Infiniband ExpressModules, and two PCIe QLogic dual 4-Gb Fibre Channel ExpressModules.
When we first reviewed blade servers, in June 2003 (see ‘Pitching Blades,’ at nwc.com/1411/1411f2.html), we were looking to answer the question: is managing blades easier than dealing with a stack of 1U or 2U servers? The answer, then as now, is an unqualified yes. But other aspects of blade servers, including design, energy efficiency, throughput and fabrics supported have improved significantly.
(For an overview of what to look for when shopping for a blade, see ‘Data Center Diet Plan.’)
FORM UP
We found significant architectural variations among the three blade systems tested. Rackable’s Scale Out system is targeted at high-density environments looking to provide hundreds, even thousands of individual system nodes. Its hot-plug design lets IT load one rack with as many as 88 server modules in a side-by-side and back-to-back configuration.
With an optional 19-inch mount kit, conventional racks can hold five vertical chassis in a 7U space. Rackable’s affordable Scale Out blades come in both single- and dual-processor versions and can be populated with Intel Xeon or AMD Opteron processors. On-blade storage is provided by one or a pair of internal, 3.5-inch SATA or SCSI drives, and server I/O options are limited to dual GbE ports.
But what makes Rackable unique in this market is its multiple power alternatives—letting Scale Out blades be used in AC or DC environments—and the fact that its blades are based on an open-architecture server module designed to accommodate a wide range of standard motherboards and hard disks. If you need the flexibility to tailor blade configurations based on specific hardware requirements, Rackable has you covered.
Meanwhile, the Sun Blade 8000 Modular System is built for speed, with the performance to handle demanding applications such as Oracle or SAP. Its 19U chassis can hold 10 four-processor blade modules, each equivalent to a conventional 4U racked server. Storage is provided by one or a pair of 2.5-inch SAS drives per blade.
Sun has chosen to align with AMD, so it offers only Opteron processors. This may not sit well with some IT managers, but we can only hope that the days of namebrand prejudice will come to an end eventually.
What sets Sun’s servers apart are its exclusive use of four-processor blades and its gutsy decision to move network components off the blades and on to the chassis backplane. This breaks the traditional blade-design mold, where each blade must be configured with the desired combination of NICs or HBAs at the time of purchase.
The Sun Blade 8000 server modules have no NICs or HBAs, and all communication from the blade is kept strictly PCIe. Each blade has four x8 PCIe and two x4 PCIe links that run through the passive midplane of the chassis. These provide substantial bandwidth to two individual x8 PCIe Express modules per blade, as well as four Network Express modules that provide shared connectivity across all ten blades.
This unique methodology allows hot-pluggable I/O customization of individual blades using optional Ethernet, FC or InfiniBand PCIe Express modules, while simultaneously supporting as many as eight additional GbE connections per blade.
Finally, the new HP ProLiant c-Class BladeSystem offers the most balanced approach of the three systems we reviewed. Targeted at enterprise data centers and server consolidation, the 10U c7000 enclosure supports as many as eight full-height dual-processor blades, 16 half-height dual-processor blades, or any combination thereof. HP offers blades with either Intel Xeon or AMD Opteron processors, and its halfheight blades can support two hot-swappable 2.5-inch SAS drives, while the full-height blades can handle as many as four drives each.
Although not quite as beefy as the Sun Blade when it comes to chassis-level I/O bandwidth, the c-Class surpasses its predecessors by offering the flexibility of multiple, integrated GbE NICs on each blade, as well as the option to add two additional NIC or HBA modules—known as mezzanine cards—on the halfheight blades and three more on the full height.
The internal GbE NICs and mezzanine cards are mapped to eight bays on the chassis backplane that can be populated with a wide choice of switches and pass-through modules, using up to four different fabrics.
MANAGE ME
We weighted system management at 20 percent of our grade. Rackable’s optional Roamer management system provides Ethernet or serial connectivity and an LCD status panel. The text-based management interface gave us remote access to basic power cycling, BIOS configuration, system reset and temperature monitoring, but little else in terms of system status information, event logging or automated problem reporting.
The Sun Blade 8000 System offers on each Server Module an Integrated Lights Out Manager (ILOM) on each Server Module that combines with the Chassis Monitoring Module to provide secure, browser-based and onsite access to a broad spectrum of status monitoring and system configuration tools. The hardcore Sun worshipper will be pleased to find a DMTF-compliant CLI as well.
Sun also offers its optional Sun N1 System Manager software, which supports an even broader set of features, including bare-metal discovery, OS provisioning, and the ability to manage hundreds of systems across multiple chassis and racks from a single console. We believe N1 System Manager would be a worthwhile addition for large environments, but the software is available only for Solaris.
HP’s Integrated Lights-Out 2 (ILO 2) management interface, now in its fourth generation, teams with HP’s Onboard Administrator module to provide single-session management of all blades and I/O devices in the chassis. We found many similarities in the ILO management capabilities of the HP and Sun systems, but HP goes the extra mile, providing additional features with its new HP Onboard Administrator.
This handy app integrates with existing ILO 2 features on all components, plus provides both local and remote control over the enhanced management tools added to the BladeSystem c-series; these include such unique features as real-time control over power and cooling, visual indicators of fault conditions and configuration errors, and simplified diagnostics and graphical instructions for troubleshooting.
We were also impressed with HP’s elegant and flexible local interface. A multifunction LCD display, located at the base of each chassis, gave us access to all Web interface management features, including role-based security, context-sensitive help, and graphical displays of problem components. A chat mode allows technicians at a remote location to communicate. Handy stuff.
Building on these integrated tools are a wealth of high-level software products from HP that offer expanded control of the server environment; these include the Insight Control Data Center Edition management suite for rapid provisioning, software installation and license management, and the ILO Select Pack for additional security and reporting features. And at the top of the food chain, there’s the HP Open View line of enterprise-level management tools.
FEEL THE HEAT
Rackable’s Scale Out blades support three power options. The least efficient is an all-AC configuration, using blades with conventional AC power supplies. The second setup combines DC-powered blades with AC/DC rectifiers for power conversion at rack level. These rectifiers can be configured for redundancy and mounted either on-rack or outside the data center for better heat management.
In addition, Rackable’s Scale Out blades are mounted in a back-to-back configuration at rack-level, so exhaust heat is ducted to the top of the rack rather than to the back of the system and into the aisle or rack behind it.
Perhaps the most energy-efficient option would be for data centers wired for redundant -48 DC. These shops can use DC-powered blades in combination with Rackable’s DC power-distribution system, which allows the rack to connect directly to a central DC power supply.
Rackable’s DC-powered blades incorporate a DC power supply with no moving parts that generates much less heat—while being several times more reliable—than conventional power supplies. One liability to the Scale Out blade design, however, is its incorporation of two high-speed fans in each blade module. This setup works well to manage heat generated by the blades themselves, but it offers no redundancy or hot-swap capabilities; not to mention a full rack of 88 Scale Out blades would have 176 fans running at all times.
Most of the power efficiency provided by the chassis-based blade systems from HP and Sun stems from their ability to consolidate power and cooling at rack level. While supplying redundant power to conventional servers requires that dual, active power supplies be mounted in each server, the blade chassis designs from HP and Sun use a maximum of six power supplies to handle the entire rack. This provides both AC redundancy and N+N power-supply redundancy. Cooling fans are handled in the same manner, with both chassis offering redundant, high-efficiency, multi-speed, hot-swappable fan modules that automatically respond to variations in the cooling needs of all blades in the chassis.
A feature unique to HP is Dynamic Power Saver, which lets an administrator choose between AC redundancy, power-supply redundancy and a power-saving mode that allows some power supplies to shut down during off-peak periods.
One of the main problems with AC power supplies is that they’re inefficient when running under reduced demand, so the Dynamic Power Saver automatically puts unnecessary power supplies on standby, letting the system run fewer power supplies at higher and more efficient utilization levels.
Administrators can actually set a hard limit on the amount of AC power a chassis or rack can consume. This affects only new loads and not cooling, so there’s no risk to systems already online. This is not a hard limit on the power, but rather a threshold setting.
The number of available power supplies and their redundancy level also determine the power available to the chassis; the system will automatically limit the number of server blades that can come online based on those guidelines and the power draw of the system. It’s a nifty high-level feature.
I/O BANDWIDTH
We evaluated all three vendors’ designs to see whether they’d be able to keep up with high-speed fabrics such as 4 Gb FC, 4x InfiniBand and 10 GbE, while still providing GbE links for management and network use.
The dual GbE ports in Rackable’s Scale Out system provide 2-Gbps throughput; this lower bandwidth will limit use in many high-performance applications—however, Rackable’s target audience seems to run more toward those needing high server density rather than fat pipes.
In contrast, the Sun Blade 8000 wowed us with its capacity. Its midplane can handle up to 9.6 Tbps in combined throughput, and Sun specs the systems at 160 Gbps of usable bandwidth per blade.
HP’s BladeSystem c-series is no slouch either, with 5 Tbps midplane bandwidth. We liked that HP provides additional high-speed cross connects between adjacent blade sockets—a boon for multi-blade clustered and storage-specific applications.
System bandwidth aside, the major difference between the Sun and HP designs lies in the location of their I/O devices. Sun has chosen to extend multiple PCIe interfaces from the blade, through a passive midplane, and to the chassis backplane, where it locates all I/O devices for FC, InfiniBand and Ethernet connectivity. On the other hand, HP’s approach uses a passive serial midplane to connect specific fabrics that originate on the blade, with the backplane reserved for switch or pass-through modules.
ANY PORT IN A STORM?
We also evaluated port diversity and flexibility. Ideally, integrated switch modules enable blade systems to support multiple fabrics across multiple servers. If this is important to you, HP’s BladeSystem c-series offers the greatest port diversity in terms of available backplane switches and pass-through modules.
On the HP c-series, GbE switch modules are available from both HP and Cisco, and FC adapters from either Emulex or QLogic. You also can use pass-through adapters that can provide direct connections between blade-level adapters and third-party switches. Aside from the dual, integrated GbE ports on the half-height and the four ports on the full-height blades, HP offers multifunction GbE mezzanine cards that support TCP/IP off-loading as well as iSCSI acceleration and RDMA (remote direct memory access) capabilities.
When it comes to features, port diversity and overall system flexibility, we have to give this round to HP. But it was a close call thanks to the remarkable bandwidth offered by the Sun Blade’s PCIe midplane architecture.
Sun offers connectivity for most fabrics through the use of individual Ethernet, InfiniBand and FC PCIe Express modules connected by two individual x8 backplane slots per blade. But at this point, the only option available for Sun’s four aggregated Network Express slots on the chassis backplane is a 20-port GbE pass-through module. Even though this offers the potential for eight individual GbE ports per blade—with room to spare for two x8 PCIe Express Modules—the absence of shared switching modules limits the Sun Blade 8000’s overall flexibility. This makes Sun pretty much all dressed up with nowhere to go at this point, but we’ll be keeping an eye on developments in this area.
On the Scale Out blades from Rackable Systems, dual onboard GbE ports are wired along with serial/Ethernet management communications through a blind-mating connector at the back of the blade; they terminate within the rack as standard RJ-45 network cables. Because there is no dedicated chassis, there are no integrated switch modules; instead, space is left available in a full rack to allow for installation of third-party switching equipment. An open PCI slot located on the front panel of the blade is available for conventional FC or Infini-Band adapters, but this would necessitate routing individual cables to the front of each blade and finding room in the rack to accommodate additional switching.
PRICE DILEMMA
The disparate architectures of these blade systems made comparing price difficult. We created a formula that pitted two dual-processor, Opteron 285-based server blades from HP and Rackable Systems against a single four-processor Opteron 885-based blade from Sun. We priced each system on a blade level with similar memory, SATA storage and dual-port GbE connectivity, without consideration for the differences in chassis/rack capabilities.
The Scale Outs from Rackable Systems came in at $9,780 for two dual-processor blades, making them the least-expensive option on a processor-by-processor basis. The HP BladeSystem landed in the middle of the pack, at $11,768 for a pair of ProLiant BL465c half-height blades. An equivalent Sun Blade would run $24,885 when you include the cost of the two PCIe Express GbE modules required to match the port counts of the other two systems. We expected this—the fact that its design requires 800-series rather than 200-series Opterons would make Sun about $6,000 more expensive than rivals from the get-go. We think the premium is worthwhile, though, for companies that can make use of the Sun Blade’s power.
Our as-tested pricing for each system: HP, $36,546, Sun, $149,680 (both including chassis and modules). Rackable, $9,780 (server blades only, no chassis or power supplies).
From a warranty aspect, both HP and Sun offer a three-year warranty on the chassis and blades with next-business-day response times for onsite service, while the Rackable Systems Scale Out servers have a one-year warranty combined with an RMA repair program.
All three vendors have both Web-based and 24/7/365 telephone accessibility and offer a number of optional, multi-year support plans that can provide up to 24/7/365 onsite service. Overall, HP offers the most complete matrix of service options for both hardware and software support, but Sun has upped the ante by launching a new pricing scheme that reduces maintenance costs by covering all devices within the entire chassis.
AND THE WINNER IS...
Our Editor’s Choice is HP’s ProLiant c-Class BladeSystem. This well-rounded offering provides a winning combination of system performance, module flexibility and price, backed by a strong service and support structure. We found the HP management system the most user-friendly and energy-conscious, and it’s obvious that HP has put a great deal of thought into solving many of the problems associated with earlier generations of blade servers. It earned consistently high scores across all grading areas.
The beefy Sun Blade 8000 Modular System earned a perfect score for I/O bandwidth and could be a solid alternative to conventional servers for core-level applications such as large Oracle or SAP environments, as well as high-performance I/O and computationally intensive apps. Its combination of four-way blades and massive bandwidth at both blade and midplane level sets a new standard for x86-based blade performance and throughput. Still, as much as we liked its performance capabilities, the Sun Blade 8000 is somewhat pricey and currently offers only one I/O module option for its four Network Express slots.
In contrast, the cost-effective Scale Out blades from Rackable Systems would be well suited to edge applications that value extremely high server density over I/O performance.
HEWLETT-PACKARD: C-CLASS BLADESYSTEM
HP designed the new c-Class BladeSystem, released in June 2006, to replace the p-Class, with an eye toward providing enough system performance and flexibility to carry the platform well into the next decade. One thing that came through loud and clear in our testing is HP’s focus on simplifying management. This, combined with outstanding energy efficiency and a substantial list of connectivity options, earned HP top marks for enterprise-class blade servers.
In an interesting departure from its earlier systems, the new c-Class offers a choice of full- or half-height server blade modules. This flexible design lets the eight-slot chassis support as many as eight full-height or 16 half-height blades, or any combination thereof. At this point, both full and half blades support a maximum of two processor sockets per blade, but the full-blade design appears to have plenty of room for four-socket full-height blades at some point in the future.
The system we tested came equipped with two full-height blades sporting a pair of 3.2 GHz dual-Core Xeon processors, and two half-height blades, each with a single 3.2 GHz dual-Core Xeon.
There is a wide variety of half-height blade options available for the c-Class BladeSystem; the Intel-based BL460c modules sport two CPU sockets and a choice of nine different dual-core Xeon processors ranging from 1.6 GHz to 3.2 GHz, and also support as many as eight sockets of PC2-5300 DDR2 FB-DIMM memory.
The AMD-based BL465c blades offer a choice of seven different Opteron 2000-series processors and provide eight sockets for PC2-5300 DDR2-registered DIMM memory. Each half blade has room for dual 2.5-inch hot-swappable SAS or SATA drives and comes with an integrated HP SmartArray E220i RAID controller that supports 0, 1 and 0+1 arrays.
Common to both half-blade designs are a pair of integrated, multifunction GbE NICs that support TCP/IP off-load under Windows, as well as accelerated iSCSI and RDMA capabilities. For other connectivity options, there are two onboard mezzanine sockets that can be used for additional standard or multifunction GbE modules and 4x DDR InfiniBand, as well as 4 GB FC modules from Emulex or QLogic. All blades for the c-Class provide connectivity for HP’s Integrated Lights Out 2 (ILO 2) system management interface without sacrificing a dedicated GbE port, and a high-density connector on the front of each blade uses a custom fan-out cable for direct KVM/USB connections.
The full-height blade we tested for the c-Class was the BL480c; like the BL460c, the 480 is available with 11 dual-core Xeon processor options. Although sharing the same processing capacity as its half-height brethren, the 480’s full-height design leaves room for 12 total slots of PC2-5300 DDR2 FB DIMMs, two additional embedded GbE ports, three total mezzanine cards, and as many as four hot-swap SAS or SATA drives. Also included is an integrated HP Smart Array P400i SAS controller with 256 MB of cache and support for hardware RAID 0, 1 and 5. HP has also recently released the new BL685c blade, which is based on four Opteron 8000-series processors and offers as many as 16 DIMM sockets supporting DDR2 memory.
The c7000 chassis offers a substantial number of features that improve the system and energy management capabilities of the c-Class compared with HP’s previous p-Class design.
At the heart of the 10U chassis is an Onboard Administrator Management Module that provides secure single-sign-on access to device configuration, power usage, system monitoring and temperature control over all components in single or multiple enclosures. It also serves as a single point of access to the ILO 2 systems on each blade, and supports virtual KVM services over Ethernet for ILO 2-enabled systems across multiple enclosures assigned to a common management domain.
HP put a lot of effort into simplifying its management interface, and it shows. Along with detailed online management capabilities, a front-mounted LCD Insight Display at the base of each chassis provides for text-based trouble-shooting, graphical error indication and a live-chat mode that allows a technician in a data center to communicate with an administrator working from a remote management terminal.
The c7000 chassis is available in single- and three-phase versions for US or international power sources. Slots for as many as six front-accessible power supplies provide multiple levels of power redundancy.
HP offers one of the most detailed power management systems in blades today, and unique to HP’s c7000 is new Dynamic Power Saver technology that allows the system to reduce the number of active power supplies during off-peak usage. Also unique is the ability of an administrator to set a hard power budget for single or multiple chassis, enabling the system to dynamically choose the amount of resources that will be allowed to come online.
System cooling is another integral part of HP’s energy strategy, and its new Thermal Logic technology combines detailed inflow and outflow temperature analysis with a highly manageable, load-balancing cooling system.
Based on the PARSEC (Parallel Redundant Scalable Airflow) architecture, the c7000 can use as many as 10 of the new high-flow, low-noise HP Active Cooling fans to provide independent zone-based cooling at blade level, with full redundancy across the chassis. This new fan module design is the quietest we’ve ever heard, which is especially, well, cool given its capacity—HP told us that a single Active Cooling module has the power to cool as many as five DL360 G4 1U servers.
The c7000 is rated for 5 Tbps in raw SerDes (serialization-deserialization) bandwidth. Because HP continues to keep storage and I/O modules on the blade, the chassis must be able to ensure that fabrics are connected properly through the midplane. To do this, the Administrator Management Module offers port-mapping capabilities that let us ensure that adapters mounted in mezzanine ports on the blades were properly connected to the interconnect bays in the backplane. A dedicated link between adjacent blade slots is available to support server clustering and connect to storage-specific blades in the future.
The last step in the I/O chain is connectivity at the backplane, and the c-Class offers the largest number of backplane options of the systems we tested. At the rear of the c7000 chassis are two bays for dual, redundant Administrator Management Modules and eight additional interconnect bays, each with 16 connections to the blade modules. Two of the interconnect bays are dedicated to redundant GbE switches; the remaining six can be populated with connectivity modules for Ethernet, FC, InfiniBand or SAS fabrics.
This design can offer fully redundant backplane support for as many as four fabrics simultaneously on a system filled with full-height blades. There are also pass-through adapters that can provide direct connections between blade-level adapters and third-party Ethernet or FC switches for companies that prefer to handle switching outside the chassis.
SUN MICROSYSTEMS: SUN BLADE 8000 MODULAR SYSTEM
When Sun discontinued its Sparc/Athlon/Xeon-based Sun Fire B1600 Blade Platform in mid-2005, the company vowed to be back in a year with a whole new blade system. It wasn’t kidding. The new Sun Blade 8000 Modular System, which bears little resemblance to its 3U predecessor, incorporates one of the most interesting technology decisions we’ve seen to date: by taking I/O modules off the blades and extending the PCI Express bus through the midplane, Sun can locate fabric connectivity modules at the backplane and provide for hot-swapping of PCIe Express modules without the need to take a blade offline.
From the start, it was obvious to us that the major focus of the Sun Blade 8000 is performance. This is perhaps best illustrated by the fact that only four-socketed blades are being built for the system. For the new Sun Blade X8400 Server Modules, Sun is partnering with AMD. At present, its blades are based on quad-940 sockets, which leave three available Opteron processor options: the 870, 875 and 885. Of course, this is only a temporary limitation because blades based on the new 1207-pin Rev.F socket and 8000-series processors will support both current dual-core and future quad-coreOpterons without a major blade redesign.
At 19.5 inches by 18.5 inches, the X8400 server modules are the largest that we tested. But Sun takes full advantage of that real estate and loads them up with four dual-core processors and sixteen sockets of system memory. Each blade has two hot-swappable 2.5-inch SAS or SATA drives and front-mounted sockets for conventional KVM/USB hookups. There’s also an on-board ILOM service processor that offers remote KVM and storage capabilities and supports Sun’s management interface as well as third-party IMPI 2.0-compliant management solutions.
Conspicuously absent on these modules are the blade-mounted Ethernet and storage adapters that we’ve come to know and love. All the I/O capabilities in the Sun Blade design are routed directly through the passive midplane using four x8 and two x4 PCI Express links per blade. This unique methodology supports 160 Gbps of total usable bandwidth from each individual blade and eliminates the need to pull a blade to replace a failed I/O module.
Like the X8400 blades, the Sun Blade 8000 chassis was the largest we tested. At a height of 19U, it weighed in at over 500 pounds and supports 10 server modules when fully configured. But, considering that a fully-loaded Sun Blade 8000 Modular System offers the processing equivalent of 10 conventionally racked, four-socket 4U servers, the 8000 is an impressive, brushed aluminum tower of power. Like the HP c-Class, the Sun Blade 8000 chassis supports dual redundant CMMs (Chassis Management Modules), which provide a unified management interface for all components and servers.
The CMM offered us detailed monitoring capabilities for module status and temperature parameters, and is designed to integrate with management tools for Sun’s high-end X64-based systems. The 8000 chassis has six front-mounted power supplies that offer N+N redundancy for input and output power; it uses 3-6 single-phase 20A 220V circuits, depending on the level of redundancy desired. System cooling is provided with multi-speed, hot-swappable fans. Three fans are dedicated to cooling the top PCI Express modules, six are for the power supplies, and nine are mounted at the back of the chassis to cool the server modules.
Perhaps the most impressive aspect of the Sun Blade 8000 Modular System stems from Sun’s decision to use a passive midplane concept that passes the PCI Express system bus through the system, rather than using blade-based GbE, FC or InfiniBand I/O fabrics. This design offers 40 serial links per server—or a total of 400 serial links—and provides a massive 9.7 Tbps of overall SerDes chassis bandwidth that can be designated to any combination of fabrics and I/O modules that can be mounted at the backplane.
Though this may seem like overkill at present, the adoption of high-speed interconnects like 10 GbE, X4 DDR InfiniBand and 8- or 10-Gbps FC will all occur during the anticipated lifecycle of these blade systems, a factor that should make total system bandwidth a point of consideration to ensure long-term investment protection.
The bandwidth and flexibility offered by the PCI Express-based design has provided Sun with some interesting and unique I/O options.
Of the four 8x PCIe channels available per server module, two slots per blade are dedicated to individual PCIe Express modules based on the PCI SIG form factor and mounted at the top rear of the chassis. To fill the 20 available slots per chassis, Sun offers three possible PCIe Express modules: an Intel-based dual-port GbE NIC, a QLogic-based dual-port 4-Gbps FC HBA, and a Mellanox-based dual-port 4x InfiniBand host channel adapter. These hot-pluggable modules can be mixed and matched on a server-by-server basis, allowing for a great deal of granularity.
The remaining two 8x and two 4xPCIe channels are connected to four PCI Express Network Express Module (NEM) slots located directly below the individual PCIe modules. These four slots are designed to aggregate the links from all 10 server modules, and could be used for four different, or two redundant, aggregated I/O fabrics.
At this point, the only NEM module available is a 20-port GbE device offering dual GbE NICs per server module; Sun says it will address this limitation in the near future. In spite of the current lack of NEM options, we respect the leap of faith Sun took to adopt this I/O strategy. Given its adoption of an industry-standard PCIe-compliant architecture, it’s hard to imagine that I/O device vendors will have difficulty fitting Sun’s design specifications for the smaller, individual PCIe Express modules.
The bigger challenge may be to find switch vendors with designs that will convert multi-server PCIe input to an aggregated, switched fabric output. A technology that seems ideally suited here is the anticipated development of I/O devices with hardware-enabled virtualization capabilities. An I/OV-enabled device would be capable of presenting multiple virtual interfaces to the combined PCIe connections provided by the NEM slots; for example, four I/OV-capable 10 GbE NICs could potentially be shared among all 10 server modules.
RACKABLE SYSTEMS: SCALE OUT SERVER SERIES
Rackable Systems introduced its Scale Out servers in 2004 as an alternative to conventional blade servers. Rather than adopting the typical chassis/module concept, Rackable’s design is based on providing the highest possible server density combined with a unique, fullrack-oriented design. By mounting Scale Out Blades in a side-by-side and back-to-back configuration, Rackable squeezes 88 servers in a single 88 in-by-28 in-by-44 in cabinet, offering as many as 176 CPUs—or 352 cores. Now that’s density.
|
Rackable has also been a long-term advocate of the adoption of DC power to improve overall energy efficiency and reduce the cooling burden in the modern data center. Rackable prides itself on delivering what it calls ‘open blades’—which in this case means blade designs based on industry-standard components such as standard-formfactor motherboards, 3.5-inch disk drives, and a wide variety of processor selections from either Intel or AMD.
Rackable could base a Scale Out Blade on almost any server-class ATX motherboard that can fit in its case, offering a great deal of flexibility for individual customers with specific needs.
For this review we requested an AMD system based on a pair of dual-core Opteron processors, but Rackable Systems would have been able to provide practically any combination of processor we could have requested. When we opened up the Scale Out blade we found a conventional ATX dual-socket 940 motherboard with 8 DIMM sockets mounted with the I/O risers facing the front of the blade chassis. This allowed access to the
KVM and USB ports through a panel on the front. The dual onboard GbE NIC ports were jumpered using Category 6 cabling to a blind-mating connector at the back of the chassis.
By mounting the motherboard backward—relatively speaking—Rackable leaves room above the PCI slots to internally mount dual, 160-GB, 3.5-inch SATA drives; we could have incorporated SAS storage or as many as four 2.5-inch drives in either flavor. The power supply is mounted right behind the drives at the back of the chassis, and dual high-speed fans provide cooling for the entire module. Like conventional blade designs, the Scale Out modules are designed to be hot-pluggable without the need for tools.
Chassis? Who needs a chassis? Perhaps the biggest departure from the conventional blade model is Rackable’s chassis-less approach. The closest thing we could find to a chassis is the custom rack-mounting system that the Scale Out blades require. Cooling air is drawn from both the front and back of the rack, and exhaust air is directed straight up through the top of the rack. Each blade has dual independent fans, which equates to 176 total fans in a full rack.
Each server module uses a relatively low-density blind-mating connector that provides power to the blade as well as I/O connectivity for the Ethernet and optional serial connections. In the middle of the rack these connectors tie into an integrated cabling system that terminates in RJ-45 connectors near the switch mounting bays. For companies requiring support for additional connectivity, there’s access to an open PCI slot in the front of the blade, but this would necessitate routing cables from the face of any affected servers to the required switch.
Absent the chassis, there’s no need for chassis-level management. But this also means that Rackable Systems offers no integrated, unified server management interface. The optional Roamer management interface available on the Scale Out blades provides individual control of basic power-cycling and BIOS settings while monitoring ambient case temperatures, but absent a centralized management interface we saw limited out-of-the-box options for large-scale system management without resorting to third-party software. In this regard, the Scale Out Blades are identical to conventionally racked servers.
Aside from extremely high processor density, what makes the Scale Outs stand out are the three powering options offered by Rackable Systems. For those choosing an AC-only power option at blade-level, the Scale Out Blades offer pretty much the same energy efficiency as conventional servers, and depend on a 1:1 ratio between AC power supply and blade.
The real power efficiency lies in systems based on DC-powered blades that use system-wide DC or DC power that’s converted at rack level.
First, the internal DC power supplies for the Scale Out servers are more efficient, generate less heat, and are far less prone to failure than conventional AC power supplies. And by supporting DC at blade-level, you can use multiple DC rectifiers at rack level to provide the same N+N power redundancy found in chassis-based designs. For the ultimate in power efficiency, Rackable Systems advocates the adoption of data-center-level DC power, which could be converted outside the data center to decrease thermal load inside.
The biggest downsides of Rackable Systems’ design are limited I/O bandwidth and options for the Scale Out blades we tested. These systems can obviously provide substantial processing capabilities, but would be bottlenecked for a number of applications by the limitations imposed by their dual-GbE NICs.
Even though there is room for one additional, front-facing PCI card, this clearly wouldn’t be an optimal solution in many data centers. We hope this is an issue Rackable addresses in the next generations of Scale Out systems. Given the company’s ‘open blade’ strategy, it’s likely that rapid advancements in server-class ATX motherboard design will offer new alternatives in the future, but the relatively low density of its existing blind-mating connector will still lead to bandwidth limitations.
In all fairness, there are a number of applications that function perfectly well in a GbE-only environment. We’ve also found that many shops using Ethernet-based iSCSI for storage rarely stress their dual GbE connections, and most 1U and 2U racked servers come standard with only two GbE ports. With its affordable Scale Out blades, Rackable Systems is clearly targeting server-intensive environments such as telcos and service providers, where massive scalability equals efficiency, and affordability can be a key factor.
|