Wikipedia

Pairwise summation: Difference between revisions

Browse history interactively
← Previous edit
Content deleted Content added
VisualWikitext
No edit summary
 
(22 intermediate revisions by 14 users not shown)
Line 2:
first1=Nicholas J. | last1=Higham | journal=[[SIAM Journal on Scientific Computing]] |
volume=14 | issue=4 | pages=783–799 | doi=10.1137/0914050 | year=1993
| citeseerx=10.1.1.43.3535 }}</ref> Although there are other techniques such as [[Kahan summation]] that typically have even smaller round-off errors, pairwise summation is nearly as good (differing only by a logarithmic factor) while having much lower computational cost&mdash;it can be implemented so as to have nearly the same cost (and exactly the same number of arithmetic operations) as naive summation.
 
In particular, pairwise summation of a sequence of ''n'' numbers ''x<sub>n</sub>'' works by [[recursion (computer science)|recursively]] breaking the sequence into two halves, summing each half, and adding the two sums: a [[divide and conquer algorithm]]. Its worst-case roundoff errors grow [[Big O notation|asymptotically]] as at most ''O''(ε&nbsp;log&nbsp;''n''), where ε is the [[machine precision]] (assuming a fixed [[condition number]], as discussed below).<ref name=Higham93/> In comparison, the naive technique of accumulating the sum in sequence (adding each ''x<sub>i</sub>'' one at a time for ''i''&nbsp;=&nbsp;1,&nbsp;...,&nbsp;''n'') has roundoff errors that grow at worst as ''O''(ε''n'').<ref name=Higham93/> [[Kahan summation]] has a [[error bound|worst-case error]] of roughly ''O''(ε), independent of ''n'', but requires several times more arithmetic operations.<ref name=Higham93/> If the roundoff errors are random, and in particular have random signs, then they form a [[random walk]] and the error growth is reduced to an average of <math>O(\varepsilon \sqrt{\log n})</math> for pairwise summation.<ref name=Tasche>Manfred Tasche and Hansmartin Zeuner ''Handbook of Analytic-Computational Methods in Applied Mathematics'' Boca Raton, FL: CRC Press, 2000).</ref>
 
A very similar recursive structure of summation is found in many [[fast Fourier transform]] (FFT) algorithms, and is responsible for the same slow roundoff accumulation of those FFTs.<ref name="Tasche"/><ref name=JohnsonFrigo08>S. G. Johnson and M. Frigo, "[http://cnx.org/content/m16336/latest/ Implementing FFTs in practice], in ''[http://cnx.org/content/col10550/ Fast Fourier Transforms]'', edited by [[C. Sidney Burrus]] (2008).</ref>
 
Pairwise summation is the default summation algorithm in [[NumPy]]<ref>[https://github.com/numpy/numpy/pull/3685 ENH: implement pairwise summation], github.com/numpy/numpy pull request #3685 (September 2013).</ref> and the [[Julia (programming language)|Julia technical-computing language]],<ref>[https://github.com/JuliaLang/julia/pull/4039 RFC: use pairwise summation for sum, cumsum, and cumprod], github.com/JuliaLang/julia pull request #4039 (August 2013).</ref> where in both cases it was found to have comparable speed to naive summation (thanks to the use of a large base case).▼
 
==The algorithm==
 
In [[pseudocode]], the pairwise summation algorithm for an [[Array data type|array]] ''{{var|x''}} of length ''{{var|n''}} ≥ 0 can be written:
 
''s'' = '''pairwise'''(''x''[1&hellip;...''n''])
'''if''' ''n'' &le; ''N'' ''base case: naive summation for a sufficiently small array''
''s'' = ''x''[1]0
'''for''' ''i'' = 21 to ''n''
''s'' = ''s'' + ''x''[''i'']
'''else ''' ''divide and conquer: recursively sum two halves of the array''
''m'' = [[Floor and ceiling functions|floor]](''n'' / 2)
''s'' = '''pairwise'''(''x''[1&hellip;...''m'']) + '''pairwise'''(''x''[''m''+1&hellip;...''n''])
endif'''end if'''
 
For some sufficiently small ''{{var|N''}}, this algorithm switches to a naive loop-based summation as a [[Recursion#base case|base case]], whose error bound is ''O''(ε''N''Nε).<ref>{{cite book|first=Nicholas Therefore,| thelast=Higham |title=Accuracy and Stability of Numerical Algorithms (2 ed)| publisher=SIAM|year=2002 | pages=81–82}}</ref> The entire sum has a worst-case error that grows asymptotically as ''O''(ε''N''&nbsp;log&nbsp;''n'') for large ''n'', for a given condition number (see below), and the smallest error bound is attained for&nbsp;''N''&nbsp;=&nbsp;1.
 
In an algorithm of this sort (as for [[Divide and conquer algorithm#Choosing the base cases|divide and conquer algorithm]]s in general<ref>Radu Rugina and Martin Rinard, "[http://people.csail.mit.edu/rinard/paper/lcpc00.pdf Recursion unrolling for divide and conquer programs]," in ''Languages and Compilers for Parallel Computing'', chapter 3, pp. 34–48. ''Lecture Notes in Computer Science'' vol. 2017 (Berlin: Springer, 2001).</ref>), it is desirable to use a larger base case in order to [[Amortized analysis|amortize]] the overhead of the recursion. If ''N''&nbsp;=&nbsp;1, then there is roughly one recursive subroutine call for every input, but more generally there is one recursive call for (roughly) every ''N''/2 inputs if the recursion stops at exactly&nbsp;''n''&nbsp;=&nbsp;''N''. By making ''N'' sufficiently large, the overhead of recursion can be made negligible (precisely this technique of a large base case for recursive summation is employed by high-performance FFT implementations<ref name=JohnsonFrigo08/>).
Line 50 ⟶ 48:
Note that the <math>1 - \varepsilon \log_2 n</math> denominator is effectively 1 in practice, since <math>\varepsilon \log_2 n</math> is much smaller than 1 until ''n'' becomes of order 2<sup>1/ε</sup>, which is roughly 10<sup>10<sup>15</sup></sup> in double precision.
 
In comparison, the relative error bound for naive summation (simply adding the numbers in sequence, rounding at each step) grows as <math>O(\varepsilon n)</math> multiplied by the condition number.<ref name=Higham93/> In practice, it is much more likely that the rounding errors have a random sign, with zero mean, so that they form a random walk; in this case, naive summation has a [[root mean square]] relative error that grows as <math>O(\varepsilon \sqrt{n})</math> and pairwise summation ashas an error that grows as <math>O(\varepsilon \sqrt{\log n})</math> on average.<ref name="Tasche"/>
 
==Software implementations==
 
▲Pairwise summation is the default summation algorithm in [[NumPy]]<ref>[https://github.com/numpy/numpy/pull/3685 ENH: implement pairwise summation], github.com/numpy/numpy pull request #3685 (September 2013).</ref> and the [[Julia (programming language)|Julia technical-computing language]],<ref>[https://github.com/JuliaLang/julia/pull/4039 RFC: use pairwise summation for sum, cumsum, and cumprod], github.com/JuliaLang/julia pull request #4039 (August 2013).</ref> where in both cases it was found to have comparable speed to naive summation (thanks to the use of a large base case).
 
Other software implementations include the HPCsharp library<ref>https://github.com/DragonSpit/HPCsharp HPCsharp nuget package of high performance C# algorithms</ref> for the [[C Sharp (programming language)|C#]] language and the standard library summation<ref>{{Cite web|title=std.algorithm.iteration - D Programming Language|url=https://dlang.org/phobos/std_algorithm_iteration.html#sum|access-date=2021-04-23|website=dlang.org}}</ref> in [[D (programming language)|D]].
 
==References==
{{reflist}}
<references/>
 
[[Category:Computer arithmetic]]
Retrieved from "https://en.wikipedia.org/wiki/Pairwise_summation"