Content deleted Content added
rephrasing |
m MOS:ELLIPSES (via WP:JWB) |
||
(48 intermediate revisions by 22 users not shown) | |||
Line 2:
first1=Nicholas J. | last1=Higham | journal=[[SIAM Journal on Scientific Computing]] |
volume=14 | issue=4 | pages=783–799 | doi=10.1137/0914050 | year=1993
| citeseerx=10.1.1.43.3535 }}</ref> Although there are other techniques such as [[Kahan summation]] that typically have even smaller round-off errors, pairwise summation is nearly as good (differing only by a logarithmic factor) while having much lower computational cost—it can be implemented so as to have nearly the same cost (and exactly the same number of arithmetic operations) as naive summation.
In particular, pairwise summation of a sequence of ''n'' numbers ''x<sub>n</sub>'' works by [[recursion (computer science)|recursively]] breaking the sequence into two halves, summing each half, and adding the two sums: a [[divide and conquer algorithm]]. Its worst-case roundoff errors grow [[Big O notation|asymptotically]] as at most ''O''(ε log ''n''), where ε is the [[machine precision]] (assuming a fixed [[condition number]], as discussed below).<ref name=Higham93/> In comparison, the naive technique of accumulating the sum in sequence (adding each ''x<sub>i</sub>'' one at a time for ''i'' = 1, ..., ''n'') has roundoff errors that grow at worst as ''O''(ε''n'').<ref name=Higham93/> [[Kahan summation]] has a [[error bound|worst-case error]] of roughly ''O''(ε), independent of ''n'', but requires several times more arithmetic operations.<ref name=Higham93/> If the roundoff errors are random, and in particular have random signs, then they form a [[random walk]] and the error growth is reduced to an average of <math>O(\varepsilon \sqrt{\log n})</math> for pairwise summation.<ref name=Tasche>Manfred Tasche and Hansmartin Zeuner ''Handbook of Analytic-Computational Methods in Applied Mathematics'' Boca Raton, FL: CRC Press, 2000).</ref>
==The algorithm==
In [[pseudocode]], the pairwise summation algorithm for an [[Array data type|array]]
''s'' = '''pairwise'''(''x''[1
'''if''' ''n'' ≤ ''N''
''s'' =
'''for''' ''i'' =
''s'' = ''s'' + ''x''[''i'']
'''else
''m'' = [[Floor and ceiling functions|floor]](''n'' / 2)
''s'' = '''pairwise'''(''x''[1
For some sufficiently small
Regardless of ''N'', exactly ''
A variation on this idea is to break the sum into ''b'' blocks at each recursive stage, summing each block recursively, and then summing the results, which was dubbed a "superblock" algorithm by its proposers.<ref name=Castaldo08>Anthony M. Castaldo, R. Clint Whaley, and Anthony T. Chronopoulos, "Reducing floating-point error in dot product using the superblock family of algorithms," ''SIAM J. Sci. Comput.'', vol. 32, pp. 1156–1174 (2008).</ref> The above pairwise algorithm corresponds to ''b'' = 2 for every stage except for the last stage which is ''b'' = ''N''.
==Accuracy==
Suppose that one is summing ''n'' values ''x''<sub>''i''</sub>, for ''i'' = 1, ..., ''n''. The exact sum is:
:<math>S_n = \sum_{i=1}^n x_i</math>
(computed with infinite precision). With pairwise summation for a base case ''N'' = 1, one instead obtains <math>S_n + E_n</math>, where the error <math>E_n</math> is bounded above by:<ref name=Higham93/>
:<math>|E_n| \leq \frac{\varepsilon \log_2 n}{1 - \varepsilon \log_2 n} \sum_{i=1}^n |x_i| </math>
where ε is the [[machine precision]] of the arithmetic being employed (e.g. ε≈10<sup>−16</sup> for standard [[double precision]] floating point). Usually, the quantity of interest is the [[relative error]] <math>|E_n|/|S_n|</math>, which is therefore bounded above by:▼
:<math>\frac{|E_n|}{|S_n|} \leq \frac{\varepsilon \log_2 n}{1 - \varepsilon \log_2 n} \left[\frac{\sum_{i=1}^n |x_i|}{\left| \sum_{i=1}^n x_i \right|}\right]. </math>▼
▲where ε is the [[machine precision]] of the arithmetic being employed (e.g. ε ≈ 10<sup>−16</sup> for standard [[double precision]] floating point). Usually, the quantity of interest is the [[relative error]] <math>|E_n|/|S_n|</math>, which is therefore bounded above by:
In the expression for the relative error bound, the fraction [Σ|''x<sub>i</sub>''|/|Σ''x<sub>i</sub>''|] is the [[condition number]] of the summation problem. Essentially, the condition number represents the ''intrinsic'' sensitivity of the summation problem to errors, regardless of how it is computed.<ref>L. N. Trefethen and D. Bau, ''Numerical Linear Algebra'' (SIAM: Philadelphia, 1997).</ref> The relative error bound of ''every'' ([[backwards stable]]) summation method by a fixed algorithm in fixed precision (i.e. not those that use [[arbitrary precision]] arithmetic, nor algorithms whose memory and time requirements change based on the data), is proportional to this condition number.<ref name=Higham93/> An ''ill-conditioned'' summation problem is one in which this ratio is large, and in this case even pairwise summation can have a large relative error. For example, if the summands ''x<sub>i</sub>'' are uncorrelated random numbers with zero mean, the sum is a [[random walk]] and the condition number will grow proportional to <math>\sqrt{n}</math>. On the other hand, for random inputs with nonzero mean the condition number asymptotes to a finite constant as <math>n\to\infty</math>. If the inputs are all [[non-negative]], then the condition number is 1.▼
▲:<math>\frac{|E_n|}{|S_n|} \leq \frac{\varepsilon \log_2 n}{1 - \varepsilon \log_2 n} \left
▲In the expression for the relative error bound, the fraction
Note that the <math>1 - \varepsilon \log_2 n</math> denominator is effectively 1 in practice, since <math>\varepsilon \log_2 n</math> is much smaller than 1 until ''n'' becomes of order 2<sup>1/ε</sup>, which is roughly 10<sup>10<sup>15</sup></sup> in double precision.
In comparison, the relative error bound for naive summation (simply adding the numbers in sequence, rounding at each step) grows as <math>O(\varepsilon n)</math> multiplied by the condition number.<ref name=Higham93/> In practice, it is much more likely that the rounding errors have a random sign, with zero mean, so that they form a random walk; in this case, naive summation has a [[root mean square]] relative error that grows as <math>O(\varepsilon \sqrt{n})</math> and pairwise summation
==Software implementations==
Pairwise summation is the default summation algorithm in [[NumPy]]<ref>[https://github.com/numpy/numpy/pull/3685 ENH: implement pairwise summation], github.com/numpy/numpy pull request #3685 (September 2013).</ref> and the [[Julia (programming language)|Julia technical-computing language]],<ref>[https://github.com/JuliaLang/julia/pull/4039 RFC: use pairwise summation for sum, cumsum, and cumprod], github.com/JuliaLang/julia pull request #4039 (August 2013).</ref> where in both cases it was found to have comparable speed to naive summation (thanks to the use of a large base case).
Other software implementations include the HPCsharp library<ref>https://github.com/DragonSpit/HPCsharp HPCsharp nuget package of high performance C# algorithms</ref> for the [[C Sharp (programming language)|C#]] language and the standard library summation<ref>{{Cite web|title=std.algorithm.iteration - D Programming Language|url=https://dlang.org/phobos/std_algorithm_iteration.html#sum|access-date=2021-04-23|website=dlang.org}}</ref> in [[D (programming language)|D]].
==References==
{{reflist}}
[[Category:Computer arithmetic]]
[[Category:Numerical analysis]]
[[Category:Articles with example pseudocode]]
|