Benchmarking M ATLAB and R

on a

ThinkPad X200 and a Precision 690

Derek OConnor February 19, 2012*

Matrix Benchmark Tests

This is a brief report on the results of a set of very simple benchmarks performed on a variety of numerical systems. These tests are not meant to be denitive, exhaustive, etc. They do nothing more than test the standard matrix functions on a single 1000 1000 matrix A generated by the equivalent of the M ATLAB statement A = rand(1000,1000). The M ATLAB and R code is in the Appendix 3.2. Each of these numerical software systems is based on the matrix data type or has the matrix data type as a prominent feature. If such a system does not perform well on the standard matrix functions then something is wrong.

Benchmark Hardware
The machines and operating systems used in these tests is a one-week-old Lenovo ThinkPad X220, Intel Core i7-2640M CPU @ 2.80GHz, 8GB ram 1 Processor, 2 Cores and 4 Threads Windows 7 Professional, 64-bit Although the clock rate of the CPU is 2.8GHz, Intels Turbo Boost can increase this to 3.5GHz when it deems it necessary. It is not clear (yet) how Turbo Boost affects these benchmarks because there appears to be no way (yet) to turn it on or off within the numerical systems tested. Most systems such as M ATLAB have, over time, decreased the amount of control the user has over hardware and low-level program settings. This is a good idea in general but frustrates the amateur benchmarker.
* Started:

30 Dec 2011.

web:

http://www.derekroconnor.net

email : [email protected]

Derek OConnor The second machine is is a 4 year-old

Benchmarking R and M ATLAB

Dell Precision 690, 2 x Intel Xeon E53405 CPU @ 2.33GHz, 16GB ram 2 Processors, 8 Cores and 8 Threads Windows 7 Professional, 64-bit

Benchmark Software
Three numerical software systems were used, all under Windows 7 64-bit: 1. M ATLAB 7.13 (R2011b) 2. R version 2.14.1 (2011-12-22), Platform: x86 64-pc-mingw32/x64 (64-bit) 3. R EVOLUTION R E NTERPRISE form: x86 64-pc-mingw32
FOR

W INDOWS, R version 2.13.2 (2011-09-30), Plat-

The version of Intel Math Kernel used by M ATLAB and Revolution R is not known but is presumed to be less than a year out of date. The M ATLAB and R code is in the Appendix 3.2.

Benchmark Results

I have tested Matrix Decompositions only. These are the foundations on which all other matrix algorithms are built, and if these are shaky then all other algorithms will be shaky too. These changes will, I hope, make the results more useful.
Table 1. B ENCHMARK T IMES FOR ALL S YSTEMS ( SECS )
Machine ThinkPad X220 System (64-bit) R2.14.1 Revol R 2.13.2 M ATLAB R2011b R2.14.1 Precision 690 Revol R 2.13.2 M ATLAB R2011b A*A 0.773 0.121 0.061 0.904 0.363 0.054 chol(A) 0.162 0.029 0.014 0.157 0.112 0.017 LU 0.208 0.062 0.033 0.260 0.206 0.070 QR 0.566 0.411 0.056 0.568 1.066 0.050 SVD 3.68 1.62 0.34 4.26 5.45 0.34 Eig 6.09 3.27 0.96 6.97 15.38 2.02 Total 11.47 5.51 1.47 13.11 22.57 2.54 N. Tot 8 4 1 5 9 1

Averaged over 20 runs with n = 1000; A = rand(n,n);

D EREK OC ONNOR , F EBRUARY 19, 2012

Derek OConnor

Benchmarking R and M ATLAB

Table 2. N ORMALIZED B ENCHMARK T IMES FOR ALL S YSTEMS ( SECS )

Machine ThinkPad X220 System (64-bit) R2.14.1 Revol R 2.13.2 M ATLAB R2011b R2.14.1 Precision 690 Revol R 2.13.2 M ATLAB R2011b A*A 1.000 1.000 1.000 1.000 1.000 1.000 chol(A) 0.210 0.237 0.260 0.174 0.308 0.318 LU 0.269 0.515 0.610 0.288 0.567 1.295 QR 0.732 3.41 0.967 0.628 2.933 0.925 SVD 4.76 13.5 5.77 4.71 14.99 6.22 Eig 7.88 27.1 16.8 7.71 42.30 37.4 Total 14.6 45.7 25.4 14.5 62.1 47.1

Averaged over 20 runs with n = 1000; A = rand(n,n);

Conclusions
M ATLAB is 4 to 8 times faster than the Rs. If you need to do a lot of matrix calculations then make sure that your software system uses the latest version of the Intel Math Kernel. Also, it is well to remember that this kernel includes highly optimized Fast Fourier Transform code which is crucial in DSP (Digital Signal Processing). The most surprising conclusion, for me at least, is economic: Price: A 2012 top-of-the-range super-portable laptop costing 1,500 can beat a 2007 top-of-the-range desktop costing 9,000. If you dont like small screens and keyboards, then the 7,500 difference in price leaves you plenty for a docking station with all the best peripherals. Running Costs: In these days of high-priced sustainable electricity (0.20 per KWh average over my bills last year), running a powerful desktop PC can be expensive. At 0.20/KWh, running a ThinkPad (65W charger) will cost about 20 30/year; running the Precision 690 (1KW power supply) will cost 500 700/year, about my yearly electricity bill. As the Americans say, Go gure!

3.1

Total Cost of Ownership

Like many simple but important ideas, Total Cost of Ownership (TCO), gets re-invented every generation or two.1 I never paid much attention to it, dismissing it with Yes, yes, I know its important for corporations who buy 5,000 PCs at-a-time, but its not
Cost is another simple idea but is much more subtle. Most people cannot grasp it, and even if they do, they never apply it to themselves. Likewise, the foolishness of basing decisions on Sunk Costs, escapes the most sophisticated, especially government ministers who have spent 40 million on planning a sports stadium that will never be built.
D EREK OC ONNOR , F EBRUARY 19, 2012

1 Opportunity

Derek OConnor

Benchmarking R and M ATLAB

important to the lone PC owner, although I do remember pointing out years ago that the running cost of my expensive (at the time) laser printer was 2/sheet printed, while a cheap inkjet cost about 12/sheet. Also, I remember pointing out in 1993, a year after I got my rst laptop, that batteries cost from 100 to 150 per year. Some people were aghast re-chargeable batteries dont wear out, do they? The battery in my Zenith SuperSport SL lasted less than 6 months replacement cost was 120+VAT, in 1993! I never replaced it and the laptop became, effectively, a mobile hard disk. If you turn on you PC every week or so to check your email and you print off 4 or 5 sheets every month (a nice recipe for Irish Stew, or a great shot of a 1965 Mustang hatchback), then none of this is important. Figures 1 and 2 illustrate TCO clearly.

Figure 1. TOTAL C OST OF O WNERSHIP FOR A TOYOTA AVENSIS

Figure 2. TOTAL C OST OF O WNERSHIP FOR AN IP HONE

D EREK OC ONNOR , F EBRUARY 19, 2012

Derek OConnor

Benchmarking R and M ATLAB

3.2

Total Cost of Ownership of the Prec 690 and X220

Twenty years ago the average personal computer buyer was fairly knowledgeable, was probably a programmer, or at least a spread-sheet user, and had some experience using mainframes, minis, and micro computers. They got this experience as students, or in work, or in computer clubs. Today, computers have become a commodity2 that cost less than a TV or a good refrigerator. Along with this, the number of people buying and using computers has increased dramatically. Grandmothers buy desktops for their 12-year-old grandsons; grandfathers are given laptops by their grand-daughters, and so on. Most of these buyers have no knowledge of operating systems, RAM, hard-disk capacity, etc. And why should they? Most mobile phone users are not excited by the fact that it uses the Fast Fourier Transform.3 As a result, most buyers base their choice of computer on the price, and whatever cheap peripherals that are bundled with it. Unfortunately this leads to people buying computers that dont have enough RAM to handle the operating system smoothly, a processor whose clock-rate cannot cope with videos, or a hard-disk that is full after 2 months of downloading photos of your grand-children in Australia. In fact, a typical user may spend more than $600 on ink over the life of their printer. TO BE COMPLETED

2A

3 They

commodity is something you buy by the bagful or cartload, such as sugar, beer, sand, etc will become very irate, however, if it does not use the FFT.

D EREK OC ONNOR , F EBRUARY 19, 2012

Derek OConnor Appendix M ATLAB and R Code

function times = MatBench(n,ns); for s = 1:ns A = rand(n,n); S = A*A; k = 1; tic; A*B; t(k) = t(k) + toc; k = k+1; tic; chol(S); t(k) = t(k) + toc; k = k+1; tic; lu(S); t(k) = t(k) + toc; k = k+1; tic; qr(S); t(k) = t(k) + toc; k = k+1; tic; svd(S); t(k) = t(k) + toc; k = k+1; tic; eig(S); t(k) = t(k) + toc; end % for s avgt = t/ns; avgt(1,10) = sum(avgt); navgt = avgt/avgt(1); times = [avgt;navgt];

Benchmarking R and M ATLAB

MatBench = function(n,ns){ for (s A = S = k = in 1:ns) { matrix(runif(n*n),n,n); t(A)%*%A 1

t[k] = t[k] + system.time(A%*%A)[1]; k = k+1 t[k] = t[k] + system.time(chol(S))[1]; k = k+1 t[k] = t[k] + system.time(chol(S))[1]; k = k+1 t[k] = t[k] + system.time(qr(A))[1]; k = k+1 t[k] = t[k] + system.time(svd(A))[1]; k = k+1 t[k] = t[k] + system.time(eigen(A))[1];

} # endfor s
avgt = t/ns; avgt = c(avgt,sum(avgt)) navgt = avgt/avgt[1]; return(rbind(avgt,navgt))

D EREK OC ONNOR , F EBRUARY 19, 2012