This Unit

•  Metrics
•  Latency and throughput

CIS 501 •  Reporting performance

Computer Architecture •  Benchmarking and averaging

•  CPU performance equation & performance trends

Unit 2: Performance

Slides originally developed by Amir Roth with contributions by Milo Martin

at University of Pennsylvania with sources that included University of
Wisconsin slides by Mark Hill, Guri Sohi, Jim Smith, and David Wood.

CIS 501 (Martin/Roth): Performance 1 CIS 501 (Martin/Roth): Performance 2

Readings Performance: Latency vs. Throughput

•  H+P •  Latency (execution time): time to finish a fixed task
•  Chapter 1: Section 1.8 •  Throughput (bandwidth): number of tasks in fixed time
•  Different: exploit parallelism for throughput, not latency (e.g., bread)
•  Often contradictory (latency vs. throughput)
•  Will see many examples of this
•  Choose definition of performance that matches your goals
•  Scientific program: latency; web server: throughput?

•  Example: move people 10 miles

•  Car: capacity = 5, speed = 60 miles/hour
•  Bus: capacity = 60, speed = 20 miles/hour
•  Latency: car = 10 min, bus = 30 min
•  Throughput: car = 15 PPH (count return trip), bus = 60 PPH

CIS 501 (Martin/Roth): Performance 3 CIS 501 (Martin/Roth): Performance 4

Comparing Performance Processor Performance and Workloads
•  A is X times faster than B if •  Q: what does latency(ChipA) or throughput(ChipA) mean?
•  Latency(A) = Latency(B) / X •  A: nothing, there must be some associated workload
•  Throughput(A) = Throughput(B) * X •  Workload: set of tasks someone (you) cares about
•  A is X% faster than B if
•  Latency(A) = Latency(B) / (1+X/100) •  Benchmarks: standard workloads
•  Throughput(A) = Throughput(B) * (1+X/100) •  Used to compare performance across machines
•  Either are or highly representative of actual programs people run
•  Car/bus example
•  Latency? Car is 3 times (and 200%) faster than bus •  Micro-benchmarks: non-standard non-workloads
•  Throughput? Bus is 4 times (and 300%) faster than car •  Tiny programs used to isolate certain aspects of performance
•  Not representative of complex behaviors of real applications
•  Examples: towers-of-hanoi, 8-queens, etc.

CIS 501 (Martin/Roth): Performance 5 CIS 501 (Martin/Roth): Performance 6

SPEC Benchmarks Other Benchmarks

•  SPEC (Standard Performance Evaluation Corporation) •  Parallel benchmarks
•  http://www.spec.org/ •  SPLASH2: Stanford Parallel Applications for Shared Memory
•  Consortium that collects, standardizes, and distributes benchmarks •  NAS: another parallel benchmark suite
•  Post SPECmark results for different processors •  SPECopenMP: parallelized versions of SPECfp 2000)
•  1 number that represents performance for entire suite •  SPECjbb: Java multithreaded database-like workload
•  Benchmark suites for CPU, Java, I/O, Web, Mail, etc.
•  Updated every few years: so companies don’t target benchmarks •  Transaction Processing Council (TPC)
•  TPC-C: On-line transaction processing (OLTP)
•  SPEC CPU 2006 •  TPC-H/R: Decision support systems (DSS)
•  12 “integer”: bzip2, gcc, perl, hmmer (genomics), h264, etc. •  TPC-W: E-commerce database backend workload
•  17 “floating point”: wrf (weather), povray, sphynx3 (speech), etc. •  Have parallelism (intra-query and inter-query)
•  Written in C/C++ and Fortran •  Heavy I/O and memory components

CIS 501 (Martin/Roth): Performance 7 CIS 501 (Martin/Roth): Performance 8

SPECmark 2006 Mean (Average) Performance Numbers
•  Reference machine: Sun UltraSPARC II (@ 296 MHz) •  Arithmetic: (1/N) * !P=1..N Latency(P)
•  Latency SPECmark •  For units that are proportional to time (e.g., latency)
•  For each benchmark
•  Take odd number of samples •  You can add latencies, but not throughputs
•  Choose median •  Latency(P1+P2,A) = Latency(P1,A) + Latency(P2,A)
•  Throughput(P1+P2,A) != Throughput(P1,A) + Throughput(P2,A)
•  Take latency ratio (reference machine / your machine)
•  1 mile @ 30 miles/hour + 1 mile @ 90 miles/hour
•  Take “average” (Geometric mean) of ratios over all benchmarks
•  Average is not 60 miles/hour
•  Throughput SPECmark
•  Run multiple benchmarks in parallel on multiple-processor system
•  Harmonic: N / !P=1..N 1/Throughput(P)
•  Recent (latency) leaders •  For units that are inversely proportional to time (e.g., throughput)
•  SPECint: Intel 3.3 GHz Xeon W5590 (34.2)
•  SPECfp: Intel 3.2 GHz Xeon W3570 (39.3)
•  Geometric: N"#P=1..N Speedup(P)
•  (First time I’ve look at this where same chip was top of both) •  For unitless quantities (e.g., speedup ratios)
CIS 501 (Martin/Roth): Performance 9 CIS 501 (Martin/Roth): Performance 10

Processor Performance Equation

•  Multiple aspects to performance: helps to isolate them
•  Program runtime = “seconds per program” =
(instructions/program) * (cycles/instruction) * (seconds/cycle)
•  Instructions per program: “dynamic instruction count”
•  Runtime count of instructions executed by the program
•  Determined by program, compiler, instruction set architecture (ISA)
•  Cycles per instruction: “CPI” (typical range: 2 to 0.5)
•  On average, how many cycles does an instruction take to execute?
•  Determined by program, compiler, ISA, micro-architecture
•  Seconds per cycle: clock period, length of each cycle
•  Inverse metric: cycles per second (Hertz) or cycles per ns (Ghz)
•  Determined by micro-architecture, technology parameters
•  For low latency (better performance) minimize all three
•  Difficult: often pull against one another
CIS 501 (Martin/Roth): Performance 11 CIS 501 (Martin/Roth): Technology 12
Cycles per Instruction (CPI) Another CPI Example
•  CPI: Cycle/instruction for on average •  Assume a processor with instruction frequencies and costs
•  IPC = 1/CPI •  Integer ALU: 50%, 1 cycle
•  Used more frequently than CPI •  Load: 20%, 5 cycle
•  Favored because “bigger is better”, but harder to compute with •  Store: 10%, 1 cycle
•  Different instructions have different cycle costs •  Branch: 20%, 2 cycle
•  E.g., “add” typically takes 1 cycle, “divide” takes >10 cycles
•  Depends on relative instruction frequencies
•  Which change would improve performance more?
•  A. “Branch prediction” to reduce branch cost to 1 cycle?
•  CPI example •  B. “cache” to reduce load cost to 3 cycles?
•  A program executes equal: integer, floating point (FP), memory ops •  Compute CPI
•  Cycles per instruction type: integer = 1, memory = 2, FP = 3 •  Base = 0.5*1 + 0.2*5 + 0.1*1 + 0.2*2 = 2
•  What is the CPI? (33% * 1) + (33% * 2) + (33% * 3) = 2 •  A = 0.5*1 + 0.2*5 + 0.1*1 + 0.2*1 = 1.8
•  Caveat: this sort of calculation ignores many effects •  B = 0.5*1 + 0.2*3 + 0.1*1 + 0.2*2 = 1.6 (winner)
•  Back-of-the-envelope arguments only

CIS 501 (Martin/Roth): Performance 13 CIS 501 (Martin/Roth): Performance 14

MIPS (performance metric, not the ISA) Mhz (MegaHertz) and Ghz (GigaHertz)
•  (Micro) architects often ignore dynamic instruction count •  1 Hertz = 1 cycle per second
•  Typically work in one ISA/one compiler ! treat it as fixed 1 Ghz is 1 cycle per nanosecond, 1 Ghz = 1000 Mhz
•  (Micro-)architects often ignore dynamic instruction count…
•  CPU performance equation becomes •  … but general public (mostly) also ignores CPI
•  Latency: seconds / insn = (cycles / insn) * (seconds / cycle) •  Equates clock frequency with performance!
•  Throughput: insn / second = (insn / cycle) * (cycles / second) •  Which processor would you buy?
•  Processor A: CPI = 2, clock = 5 GHz
•  MIPS (millions of instructions per second) •  Processor B: CPI = 1, clock = 3 GHz
•  Cycles / second: clock frequency (in MHz) •  Probably A, but B is faster (assuming same ISA/compiler)
•  Example: CPI = 2, clock = 500 MHz ! 0.5 * 500 MHz = 250 MIPS •  Classic example
•  800 MHz PentiumIII faster than 1 GHz Pentium4!
•  Pitfall: may vary inversely with actual performance •  More recent example: Core i7 faster clock-per-clock than Core 2
–  Compiler removes insns, program gets faster, MIPS goes down •  Same ISA and compiler!
–  Work per instruction varies (e.g., multiply vs. add, FP vs. integer) •  Meta-point: danger of partial performance metrics!
CIS 501 (Martin/Roth): Performance 15 CIS 501 (Martin/Roth): Performance 16
Latency vs. Throughput Revisited Inter-Insn Parallelism: Pipelining
•  Latency and throughput: two views of performance …
•  … at the program level +
4
•  ... not at the instructions level
Register
File Data
•  Single instruction latency PC Insn s1 s2 d Mem
Mem
–  Doesn’t matter: programs comprised of [billions]+ of instructions
–  Difficult to reduce anyway
Tinsn-mem Tregfile TALU Tdata-mem Tregfile

•  As number of dynamic instructions is large… •  Pipelining: cut datapath into N stages (here 5) Tsinglecycle
•  Instruction throughput ! program latency or throughput •  Separate each stage of logic by latches
+  Can reduce using parallelism •  Clock period: maximum logic + wire delay of any stage =
•  Multiple cores (more units executing instructions)… more later max(Tinsn-mem, Tregfile, TALU, Tdata-mem)
•  Inter-instruction parallelism example: pipelining •  Base CPI = 1, but actual CPI > 1: pipeline must often stall
•  Individual insn latency increases (pipeline overhead), not the point
CIS 501 (Martin/Roth): Performance 17 CIS 501 (Martin/Roth): Performance 18

Pipelining: Clock Frequency vs. IPC CPI and Clock Frequency

•  Increase number of pipeline stages (“pipeline depth”) •  Clock frequency implies CPU clock
•  Keep cutting datapath into finer pieces •  Other system components have their own clocks (or not)
+  Increases clock frequency (decreases clock period) •  E.g., increasing processor clock doesn’t accelerate memory latency
•  Latch overhead & unbalanced stages cause sub-linear scaling •  Example: a 1 Ghz processor with
•  Double the number of stages won’t quite double the frequency •  80% non-memory instructions @ 1 cycle
–  Decreases IPC (increase CPI) •  20% memory instructions @ 6 nanoseconds (6 cycles)
•  Base: CPI is 2, frequency is 1Ghz ! MIPS is 500
•  More pipeline “hazards”, higher branch penalty
•  Memory latency relatively higher (same absolute lat., more cycles) •  Impact of double the core clock frequency?
•  Without speeding up the memory
–  Result: after some point, deeper pipelining can decrease performance
•  Non-memory instructions retain 1-cycle latency
•  “Optimal” pipeline depth is program and technology specific
•  Memory instructions now have 12-cycle latency
•  Classic example •  CPI = (80% * 1) + (20% * 12) = 3.2 CPI @ 2Ghz ! MIPS is 625
•  Pentium III: 12 stage pipeline, 800 MHz •  Speedup = 625/500 = 1.25, which is << 2
•  Pentium 4: 22 stage pipeline, 1 GHz (Actually slower due to IPC) •  What about an infinite clock frequency? (non-memory free)
•  Only a factor of 1.66 speedup (example of Amdahl’s Law)
CIS 501 (Martin/Roth): Performance 19 CIS 501 (Martin/Roth): Performance 20
Measuring CPI Performance Trends
•  How are CPI and execution-time actually measured? 386 486 Pentium PentiumII Pentium4 Core2
•  Execution time? stopwatch timer (Unix “time” command) Year 1985 1989 1993 1998 2001 2006
•  CPI = CPU time / (clock frequency * dynamic insn count) Technode (nm) 1500 800 350 180 130 65
•  How is dynamic instruction count measured? Transistors (M) 0.3 1.2 3.1 5.5 42 291
Clock (MHz) 16 25 66 200 1500 3000
Pipe stages “1” 5 5 10 22 to 31 ~15
•  More useful is CPI breakdown (CPICPU, CPIMEM, etc.)
(Peak) IPC 0.4 1 2 3 3 “8”
•  So we know what performance problems are and what to fix
(Peak) MIPS 6 25 132 600 4500 24000
•  Hardware event counters
•  Available in most processors today •  Historically, clock provides 75%+ of performance gains…
•  One way to measure dynamic instruction count
•  Achieved via both faster transistors and deeper pipelines
•  Calculate CPI using counter frequencies / known event costs
•  … that’s changing: 1GHz: ‘99, 2GHz: ‘01, 3GHz: ‘02, 4Ghz?
•  Cycle-level micro-architecture simulation (e.g., SimpleScalar)
•  Deep pipelining can be power inefficient
+ Measure exactly what you want … and impact of potential fixes!
•  Physical scaling limits? (Intel’s 65nm process wasn’t great, 45nm is)
•  Method of choice for many micro-architects (and you)
CIS 501 (Martin/Roth): Performance 21 CIS 501 (Martin/Roth): Performance 22

Improving CPI: Caching and Parallelism Performance Rules of Thumb

•  CIS501 is more about improving CPI than clock frequency •  Amdahl’s Law
•  Techniques we will look at •  Literally: total speedup limited by non-accelerated piece
•  Caching, speculation, multiple issue, out-of-order issue •  Example: can optimize 50% of program A
•  Vectors, multiprocessing, more… •  Even “magic” optimization that makes this 50% disappear…
•  Moore’s Law can help CPI – “more transistors” •  …only yields a 2X speedup
•  Best examples are caches (to improve memory component of CPI) •  Corollary: build a balanced system
•  Parallelism:
•  Don’t optimize 1% to the detriment of other 99%
•  IPC > 1 implies instructions in parallel •  Don’t over-engineer capabilities that cannot be utilized
•  And now multi-processors (multi-cores)
•  Design for actual performance, not peak performance
•  But also speculation, wide issue, out-of-order issue, vectors…
•  Peak performance: “Performance you are guaranteed not to exceed”
•  All roads lead to multi-core •  Greater than “actual” or “average” or “sustained” performance
•  Why multi-core over still bigger caches, yet wider issue? •  Why? Caches misses, branch mispredictions, limited ILP, etc.
•  Diminishing returns, limits to instruction-level parallelism (ILP) •  For actual performance X, machine capability must be > X
•  Multi-core can provide linear performance with transistor count
CIS 501 (Martin/Roth): Performance 23 CIS 501 (Martin/Roth): Performance 24
Summary

•  Latency = seconds / program =

•  (instructions / program) * (cycles / instruction) * (seconds / cycle)
•  Instructions / program: dynamic instruction count
•  Function of program, compiler, instruction set architecture (ISA)
•  Cycles / instruction: CPI
•  Function of program, compiler, ISA, micro-architecture
•  Seconds / cycle: clock period
•  Function of micro-architecture, technology parameters

•  Optimize each component

•  CIS501 focuses mostly on CPI (caches, parallelism)
•  …but some on dynamic instruction count (compiler, ISA)
•  …and some on clock frequency (pipelining, technology)
CIS 501 (Martin/Roth): Performance 25