SPEC 95

FAIRFAX, VA, August 21, 1995 -- The Standard Performance Evaluation Corp. (SPEC) announces the

release of the SPEC95 benchmark suites, the latest version of the worldwide standard for measuring

and comparing computer performance across different hardware platforms. SPEC95 was developed by

SPEC's Open Systems Group (OSG), which includes more than 30 leading computer vendors, systems

integrators, publishers and consultants throughout the world.

SPEC95 is a software benchmark product produced by the Standard Performance Evaluation Corp.

(SPEC), a nonprofit group of computer vendors, systems integrators, universities, research

organizations, publishers and consultants throughout the world. It was designed to provide measures of

performance for comparing compute intensive workloads on different computer systems. SPEC95

contains two suites of benchmarks: CINT95 for measuring and comparing compute- intensive integer

performance, and CFP95 for measuring and comparing computeintensive floating point performance.

"Computer systems technology evolves so rapidly that we must provide new benchmark suites every

two to three years to ensure a level playing field," says Kaivalya M. Dixit, SPEC president. "SPEC92

was a great success, but it is time to make the transition to standardized benchmarks that reflect the

advances in chip technologies, compilers and applications that have taken place over the last three

years; those benchmarks constitute SPEC95."

SPEC95 comprises two sets (or suites) of benchmarks: CINT95 for compute-intensive integer

performance and CFP95 for compute-intensive floating point performance. The two suites provide

component-level benchmarks that measure the performance of the computer's processor, memory

architecture and compiler. SPEC benchmarks are selected from existing application and benchmark

source code running across multiple platforms. Each benchmark is tested on different platforms to

obtain fair performance results across competing hardware and software systems.

SPEC95 is the third major version of the SPEC benchmark suites, which in 1989 became the first

widely accepted standard for comparing computeintensive performance across various architectures.

The new release replaces SPEC92, which will be gradually phased out between now and June 1996,

when SPEC will stop publishing SPEC92 results and stop selling the benchmark suite. Performance

results from SPEC95 cannot be compared to those from SPEC92, since new benchmarks have been

added and existing ones changed.

 "Compiler writers have learned to optimize performance for individual SPEC92 benchmarks," says Jeff

Reilly, SPEC95 release manager. "The best way to avoid these benchmark specific optimizations is to

develop new benchmark suites." SPEC95 builds on the lessons learned from the SPEC89 and

SPEC92 suites, according to Reilly. The new benchmarks were analyzed to ensure that they are as

resistant as possible to compiler optimizations that might not translate into realworld performance

gains. Improvements to the suites include longer run times and larger problems for benchmarks, more

application diversity, greater ease of use, and standard development platforms that will allow SPEC to

produce additional releases for other operating systems.

A Sun SPARCstation 10/40 with 128 MB of memory was selected as the SPEC95 reference machine

and Sun SC3.0.1 compilers were used to obtain reference timings on the new benchmarks. By

definition, the SPECint95 and SPECfp95 numbers for the Sun SPARCstation 10/40 are both "1."

The Metrics for Measurement

SPEC95 rules permit both baseline and optimized results for CINT95 and CFP95 suites. The baseline

rules restrict the number of optimizations that can be used for performance testing. In general, SPEC95

rules are more restrictive in regard to optimizations than the SPEC92 rules. Baseline metrics are

mandatory for all reported results; reporting of optimized results is optional.

SPEC95 also allows performance to be measured for both speed and throughput (rate). Speed metrics

such as SPECint95 measure how fast a computer completes a single task. Rate metrics such as

SPECint_rate95 measure how many tasks a computer can accomplish in a certain amount of time.

SPEC95 measures rate performance for single processors, symmetric multi processor systems and

cluster systems.

The CINT95 suite, written in C language, contains eight CPUintensive integer benchmarks. It is used to

measure and calculate the following metrics:

SPECint95

The geometric mean of eight normalized ratios (one for each integer benchmark) when

compiled with aggressive optimization for each benchmark.

SPECint_base95

The geometric mean of eight normalized ratios when compiled with conservative optimization

for each benchmark.

SPECint_rate95
 The geometric mean of eight normalized throughput ratios when compiled with aggressive

optimization for each benchmark.

SPECint_rate_base95

The geometric mean of eight normalized throughput ratios when compiled with conservative

optimization for each benchmark.

The CFP95 suite, written in FORTRAN language, contains 10 CPU-intensive floating point

benchmarks. It is used to measure and calculate the following metrics:

SPECfp95

The geometric mean of 10 normalized ratios (one for each floating point benchmark) when

compiled with aggressive optimization for each benchmark.

SPECfp_base95

The geometric mean of 10 normalized ratios when compiled with conservative optimization for

each benchmark.

SPECfp_rate95

The geometric mean of 10 normalized throughput ratios when compiled with aggressive

optimization for each benchmark.

SPECfp_rate_base95

The geometric mean of 10 normalized throughput ratios when compiled with conservative

optimization for each benchmark.

Vendor Reporting

Initial results for systems from six vendor companies are included with this release. Additional results

will be reported in the next issue of the SPEC Newsletter, scheduled for publication at the end of

September. SPEC members are being encouraged to report SPEC95 results on older platforms to

provide an historical perspective for the new results.

Availability

SPEC95 (CINT95 and CFP95) is available on CD-ROM from SPEC's administrator, the National

Computer Graphics Association (NCGA). The cost is $600 for new customers, $300 for new university

customers, $300 for current SPEC92 licensees and $150 for current university licensees.
SPEC is a non-profit corporation formed to establish, maintain and endorse a standardized set of

relevant benchmarks that can be applied to the newest generation of high-performance computers.

Included in its membership are the Open Systems Group (OSG); OSG Associates, consisting of

leading universities and research facilities; the HighPerformance Group (HPG); and HPG Associates.

Benchmarks

Q3: What does the "C" in CINT95 and CFP95 stand for?
A3: In its product line, SPEC uses "C" to denote a "component-level" benchmark and "S" to denote a
"system level" benchmark. CINT95 and CFP95 are component-level benchmarks.

Q4: What components do CINT95 and CFP95 measure?

A4: Being compute-intensive benchmarks, they emphasize the performance of the computer's
processor, the memory architecture and the compiler. It is important to remember the contribution of
the latter two components; performance is more than just the processor.

Q5: What component performance is not measured by CINT95 and CFP95?

A5: The CINT95 and CFP95 benchmarks do not stress other computer components such as I/O (disk
drives), networking or graphics. It might be possible to configure a system in such a way that one or
more of these components impact the performance of CINT95 and CFP95, but that is not the intent of
the suites.

Q6: What is included in the SPEC95 package?

A6: SPEC provides the following in its SPEC95 package: SPEC95 tools for compiling, running and
validating the benchmarks, compiled for a variety of operating systems; source code for the SPEC95
tools, to allow the tools to be built for systems not covered by the precompiled tools; source code for
the benchmarks; tools for generating performance reports; run and reporting rules defining how the
benchmarks should be used to produce standard results; and SPEC95 documentation.
The initial offering of SPEC95 will have tools for most UNIX operating systems. Additional products for
other operating systems (Windows NT, OpenVMS, etc.) will be released as later products if SPEC
detects enough demand. All of this will be shipped on a single CD-ROM disk.

Q7: What does the SPEC95 user have to provide?

A7: The user must have a computer system running a UNIX environment with a compiler installed and
a CD-ROM drive. Approximately 150 MB will be needed on a hard drive to install SPEC95. It is also
assumed that the system has at least 64 MB of RAM to ensure that the benchmarks remain compute-
intensive (SPEC is assuming this will be the standard amount of desktop memory during the life of this
suite).

Q8: What are the basic steps in running the benchmarks?

A8: Installation and use are covered in detail in the SPEC95 User Documentation. The basic steps are
as follows: Install SPEC95 from media. Run the installation scripts specifying your operating system.
Compile the tools, if executables are not provided in SPEC95. Determine what metric you wish to run.
Create a configuration file for that metric. In this file, you specify compiler flags and other system
dependent information. Run the SPEC tools to build (compile), run and validate the benchmarks. If the
above steps are successful, generate a report based on the run times and metric equations.

Q9: What source code is provided? What exactly makes up these suites?
A9: CINT95 and CFP95 are based on compute- intensive applications provided as source code.
CINT95 contains eight applications written in C that are used as benchmarks:
 099.go -- Artificial intelligence; plays the game of "Go"
124.m88ksim -- Moto 88K chip simulator; runs test program
126.gcc -- New version of GCC; builds SPARC code
129.compress -- Compresses and decompresses file in memory
130.li -- LISP interpreter
132.ijpeg -- Graphic compression and decompression
134.perl -- Manipulates strings (anagrams) and prime numbers in Perl
147.vortex -- A database program

CFP95 contains 10 applications written in FORTRAN that are used as benchmarks:

101.tomcatv -- A mesh-generation program

102.swim -- Shallow water model with 1024 x 1024 grid
103.su2cor -- Quantum physics; Monte Carlo simulation
104.hydro2d -- Astrophysics; Hydrodynamical Navier Stokes equations
107.mgrid -- Multi-grid solver in 3D potential field
110.applu -- Parabolic/elliptic partial differential equations
125.turb3d -- Simulates isotropic,homogeneous turbulence in a cube
141.apsi -- Solves problems regarding temperature, wind, velocity and distribution of pollutants
145.fpppp -- Quantum chemistry
146.wave5 -- Plasma physics; electromagnetic particle simulation

Benchmarks exist to act as a reference point for comparison. It's the same reason that EPA gas mileage
exists, although probably no driver in America gets exactly the EPA gas mileage. If you understand what
benchmarks measure, they're useful. It's important to know that CINT95 and CFP95 are CPU-focused and
not system-focused benchmarks. These CPU benchmarks focus on only one portion of those factors that
contribute to applications performance. A graphics or network performance bottleneck within an application,
for example, will not be reflected in these benchmarks. Understanding your own needs helps determine the
relevance of the benchmarks.

Q14: Which SPEC95 metric should be used to determine performance?

A14: It depends on your needs. SPEC provides the benchmarks and results as tools for you to use. You
need to determine how you use a computer or what your performance requirements are and then choose
the appropriate SPEC benchmark or metrics.

A single user running a compute-intensive integer program, for example, might only be interested in
SPECint95 or SPECint_base95. On the other hand, a person who maintains a machine used by multiple
scientists running floating point simulations might be more concerned with SPECfp_rate95 or
SPECfp_rate_base95.
 Types & Size of Operands
How is the type of an operand designated? Normally, encoding in the opcode
designates the type of an operand—this is the method used most often. Alternatively,
the data can be annotated with tags that are interpreted by the hardware.
These tags specify the type of the operand, and the operation is chosen accordingly.
Computers with tagged data, however, can only be found in computer
museums.
Let's start with desktop and server architectures. Usually the type of an operand—
integer, single-precision floating point, character, and so on—effectively
gives its size. Common operand types include character (8 bits), half word (16
bits), word (32 bits), single-precision floating point (also 1 word), and doubleprecision
floating point (2 words). Integers are almost universally represented as
B-14 Appendix B Instruction Set Principles and Examples
two's complement binary numbers. Characters are usually in ASCII, but the 16-
bit Unicode (used in Java) is gaining popularity with the internationalization of
computers. Until the early 1980s, most computer manufacturers chose their own
floating-point representation. Almost all computers since that time follow the
same standard for floating point, the IEEE standard 754. The IEEE floating-point
standard is discussed in detail in Appendix I.
Some architectures provide operations on character strings, although such
operations are usually quite limited and treat each byte in the string as a single
character. Typical operations supported on character strings are comparisons
and moves.
For business applications, some architectures support a decimal format,
usually called packed decimal or binary-coded decimal—4 bits are used to
encode the values 0-9, and 2 decimal digits are packed into each byte. Numeric
character strings are sometimes called unpacked decimal, and operations—
called packing and unpacking—are usually provided for converting back and
forth between them.
One reason to use decimal operands is to get results that exactly match decimal
numbers, as some decimal fractions do not have an exact representation in
binary. For example, 0.1010 is a simple fraction in decimal, but in binary it
requires an infinite set of repeating digits: 0.0001100110011. . . 2. Thus, calculations
that are exact in decimal can be close but inexact in binary, which can be a
problem for financial transactions. (See Appendix I to learn more about precise
arithmetic.)
Our SPEC benchmarks use byte or character, half-word (short integer), word
(integer), double-word (long integer), and floating-point data types. Figure B.ll
shows the dynamic distribution of the sizes of objects referenced from memory
for these programs. The frequency of access to different data types helps in
deciding what types are most important to support efficiently. Should the computer
have a 64-bit access path, or would taking two cycles to access a double
word be satisfactory? As we saw earlier, byte accesses require an alignment network:
How important is it to support bytes as primitives? Figure B.ll uses memory
references to examine the types of data being accessed.
In some architectures, objects in registers may be accessed as bytes or half
words. However, such access is very infrequent—on the VAX, it accounts for no
more than 12% of register references, or roughly 6% of all operand accesses in
these programs.