Case Study

Grid Computing for a Telecommunications

Data Center
InfiniBand as the Key Enabler for Enterprise HPC Clustering

Executive Summary Overview

Key Challenges One of the world’s largest telecommunications companies, evaluated next-generation solutions in
order to accommodate their growing business demands, to reduce costs and to establish a more
• Reduce hardware acquisition and database
transaction costs compact infrastructure footprint. The company launched an all InfiniBand grid computing proof of

• Support future growth and “green” initiatives concept (PoC) to deploy “Database-as-a-Service” for mission-critical Oracle deployments in high-end
transactional situations. Test results revealed that performance of grid architecture—leveraging
• Provide next-generation agility to
meet the demanding needs of mission- blade servers as well as InfiniBand networking and storage—is capable of achieving results equivalent
critical applications to traditional, high-end SMP systems at significantly lower cost and power consumption levels.

Proposed Solution
The business need
• H
 igh performance, high availability blade Organizations rely heavily on their data centers to run mission-critical applications and deliver
server grid computing infrastructure leverag-
ing IBM® BladeCenter® blade servers sustainable services to both internal and external users. With constant demands for better
performance and high reliability at lower prices, innovative solutions that break the current model
• High bandwidth, low latency
InfiniBand network for grid and storage of data center operations are imperative to gain competitive advantages.
systems interconnect
• LSI 7900 HPC enterprise-class native The company realized that its traditional SMP infrastructure and enterprise-class Fibre Channel
InfiniBand storage system storage area networks (SAN)—in operation for nearly a decade—were insufficient to accommodate

Results future demands. Operating close to 30,000 servers worldwide, with virtually half of the deployments
on SMP systems, the company faced increased operating costs and diminishing benefits due to
• D
 emonstrated that Oracle® database perfor-
mance on a grid computing infrastructure scalability and performance limitations. Projections indicated that doubling the investment in their
can achieve equivalent results to traditional current SMP systems would only yield 70 percent additional application performance, thereby
SMP systems at approximately 90 percent
proving to be an inefficient and unacceptable solution. Additionally, the company wanted to
lower acquisition cost
improve energy efficiency to offset rising energy costs and support environmental (“green IT”)
• Proved that a grid computing infrastructure—
initiatives, requiring more efficient systems be explored.
based on blade servers and InfiniBand
networks—can provide performance
and scalability beyond traditional SMP The proposed solution
systems at approximately 70 percent lower
power consumption The company initiated a proof of concept (PoC) study to identify appropriate solutions that
• Validated the ability of native InfiniBand could reduce data center cost and deliver next-generation agility and efficiency. After careful
networked storage to support 1.6 million consideration, the company decided to leverage grid computing for its open architecture,
Oracle database transactions per second,
while reducing IT complexity and cost by scalability and ability to exploit industry standard servers.
eliminating the need for a separate dedicated
storage network
 The PoC called for a high performance computing (HPC) grid solution—based on blade server
technology—to provide improved performance, availability and flexibility, with reduced power
“Using native InfiniBand storage
usage. To overcome the known performance issues inherent with many grid computing solutions,
in the data center can bring
multi-core processing and low latency InfiniBand networking and storage were utilized to
tremendous price/performance maximize performance and scalability.
improvements to solutions.”
The HPC grid study was intended to demonstrate new technologies and platforms to support
the company’s most critical applications; thereby making it imperative to deliver enterprise-
Asaf Somekh class availability at every level. As such, industry leading providers —including IBM, Oracle and
Vice President, Strategice Alliances LSI— were engaged to supply the required products, technical guidance and broad expertise
Voltaire necessary to build and benchmark the system. The global team constructed a leading, all native
InfiniBand HPC grid solution with high-density blade servers, native InfiniBand storage and
clustered database applications, as indicated by the following selected devices:

• I BM BladeCenter blade servers provide industry leading performance, availability and fault-
tolerant features—including redundant and hot-swappable components, predictive failure
analysis and advanced server management capabilities

• L SI 7900 HPC storage platform delivers a powerful combination of raw performance, native
InfiniBand connectivity and class-leading reliability and availability features—including redundant
controllers and host connections, persistent cache backup and hardware accelerated RAID
data protection.

• I nfiniBand network technology maximizes the performance of the overall solution due to its
very high bandwidth, low latency features and ability to support the quality of service and
high availability requirements associated with the company’s critical business applications.

InfiniBand is the key enabler to the high performance, scalable grid computing solutions,
providing high bandwidth (up to 20 GB/sec) over a single connection and ultra-low latency
(>2 micro seconds (µs)). Additionally, with Remote Direct Memory Access (RDMA) to dramatically
improve node-to-node communication, InfiniBand adapters can read and write data directly to
and from the memory of a server or storage device, without requiring CPU cycles. Because RDMA is
built into the lowest levels of the network adapter, high overhead protocol drivers—such as TCP/IP
or Fibre channel—are not required. Instead, data is moved directly into or out of application memory
space by the InfiniBand network adapter, virtually eliminating CPU communications overhead.

Utilizing native InfiniBand storage greatly improves grid computing performance by reducing
l/O latency and CPU overhead through RDMA. I/O intensive applications—such as the company’s
transactional databases—benefit greatly from reduced I/O latency and the data transfer efficiency
provided by native InfiniBand storage. Using InfiniBand as the shared grid and storage interconnect
eliminates the complexity and cost associated with implementing a separate dedicated storage
network, such as Fibre Channel.

From an application perspective, a reduction of network latency noticeably impacts clustered

database performance. Database instances, within a cluster, share global resources that are
continually updated and must remain synchronized, generating significant intra-node traffic.
InfiniBand networking technology significantly reduces intra-node communication latency—
enabling near-linear database cluster performance scalability.

Customer Success Story: Penspen   2

Logical Diagram of Grid Computing Architecture

High density IBM

BladeServers

20Gb/sec
InfiniBand Network Oracle RAC Cluster Apps Servers

Oracle RAC interconnect + IO (SRP)

20GB/sec InfiniBand HCAs

InfiniBand Network
Switches LSI 7900 HPC
Native InfiniBand storage

All node-to-node communications and node-to-storage

I/O traffic were transferred across the switched Infini-
Band network, eliminating the performance limitations
and latency issues associated with traditional Ethernet
and Fibre Channel networks.

As stated recently by Voltaire’s vice president of strategic alliances, Asaf Somekh, “Using native
InfiniBand storage in the data center can bring tremendous price/performance improvements
to solutions, such as clustered databases…”

Next-Generation Infrastructure: InfiniBand HPC Grid Computing

The PoC yielded compelling reasons to consider the adoption of InfiniBand HPC grid computing and
native InfiniBand storage for the company’s next-generation infrastructure platform.

Database performance testing demonstrated that the grid architecture and InfiniBand infrastructure
would provide the scalability and performance required to meet the company’s most demanding
database application loads:

• Using a worst case application test scenario with 100 percent random I/O pattern and a 50/50 read/
write ratio, the infrastructure achieved more than 760,000 transactions per minute (TPM) with 4
database nodes and scaled beyond 1.6 million TPM with an 8 node configuration.

• At peak test loads, more than 46,000 I/Os per second were being serviced by the storage system
with an average application level response time of 33 milliseconds.

In addition, the results specifically indicated the solution was able to provide the necessary avail-
ability, scalability and performance requirements at a much lower cost than traditional data center
approaches. The near-linear scalability of the InfiniBand grid infrastructure required fewer compute
resources to meet application performance requirements, thereby reducing power requirements

Customer Success Story: Penspen   3

 and helping the company support an important green initiative. In a side-by-side comparison
with similar application loads, the grid architecture consumed approximately one-third the power
“InfiniBand coupled with
of traditional SMP systems. In a financial comparison, it was determined that a cost savings of
modular, scalable storage can
approximately 90 percent could be achieved if the existing SMP systems were replaced with a grid
build a truly adaptable storage solution based on native InfiniBand infrastructure.
platform that can match even
The study proved to be successful in identifying a next-generation, cost-effective HPC solution. The
unforeseen future demands.”
company will be able to deploy mission-critical business intelligence and database applications on
Arun Taneja industry standard blade servers and storage systems with an ultra-low latency interconnect fabric.
Additionally, the InfiniBand HPC grid computing solution successfully created an agile, scalable,
Analyst, The Taneja Group
high-density infrastructure solution that the company can effectively leverage for their customer
base at a much lower cost.

“Native InfiniBand storage is gaining momentum in the data center. The factors driving this
growth are demand for high capacity storage systems with high bandwidths and ever-evolving
compute-intensive business needs,” said Arun Taneja, analyst, The Taneja Group. “InfiniBand
coupled with modular, scalable storage can build a truly adaptable storage platform that can
match even unforeseen future demands.”

For more information, visit:

IBM:
www-03.ibm.com/systems/bladecenter/
Oracle:
www.oracle.com/index.html
LSI:
www.lsi.com/storage_home/index.html

For more information and sales office locations, please visit the LSI web sites at:  lsi.com and lsi.com/contacts

North American Headquarters LSI Europe Ltd. LSI KK Headquarters

Milpitas, CA European Headquarters Tokyo, Japan
T: +1.866.574.5741 (within U.S.) United Kingdom Tel: [+81] 3.5463.7165
T: +1.408.954.3108 (outside U.S.) T: [+44] 1344.413200

LSI Corporation and the LSI logo design are trademarks or registered trademarks of LSI Corporation. All other brand and product names may
be trademarks of their respective companies.
LSI Corporation reserves the right to make changes to any products and services herein at any time without notice. LSI does not assume any
responsibility or liability arising out of the application or use of any product or service described herein, except as expressly agreed to in
writing by LSI; nor does the purchase, lease, or use of a product or service from LSI convey a license under any patent rights, copyrights,
trademark rights, or any other of the intellectual property rights of LSI or of third parties.
Copyright ©2007 LSI Corporation. All rights reserved.  WQ 0507