What Is An Algorithm
What Is An Algorithm
Yuri Gurevich
Microsoft Research
Technical Report
MSR-TR-2011-116, July 2011
Microsoft Research
Microsoft Corporation
One Microsoft Way
Redmond, WA 98052
Preamble
We present a two-part exposition on the notion of algorithm and foundational analyses of computation. The first part is below, and the second is here: http://research.microsoft.com/en-us/um/people/gurevich/
Opera/210.pdf. This preamble was added in February 2012.
In the first part, after a short introduction, we clarify some common
misconceptions related to Turings analysis of computation, consider whether
the notion of algorithm can be rigorously defined in full generality, discuss
what kind of entities algorithms are, and examine two approaches to the
title problem in the literature. The first part appeared in SOFSEM 2012:
Theory and Practice of Computer Science, Springer LNCS 7147, 3142.
The second part is devoted entirely to fundamental analyses of computations in the literature, by Turing and by others including this author. It will
appear in the proceedings of Turing Centenary Conference, CiE 2012: How
the World Computes, to be held in Cambridge, England, in June 2012.
What is an algorithm?
Yuri Gurevich
Microsoft Research
We must, incidentally, make it clear from the beginning
that if a thing is not a science, it is not necessarily bad.
For example, love is not a science. So, if something is said
not to be a science, it does not mean that there is something
wrong with it; it just means that it is not a science.
Richard Feynman
Introduction
Two articles in a recent book [10] present two approaches to the title problem
and offer different answers. Article [19] presents an approach developed by
Yiannis Moschovakis. A characteristic feature of this approach is the adoption of a very abstract notion of algorithm that takes recursion as a primitive
operation and is so wide as to admit non-implementable algorithms [19,
p.87]. The article starts thus.
In the sequence of articles . . . , Moschovakis has proposed a mathematical modeling of the notion of algorithm a set-theoretic
definition of algorithms, much like the definition of real numbers as Dedekind cuts on the rationals or that of random variables
as measurable functions on a probability space.
We discuss this definition of algorithms in 6.
Article [22] presents an approach originally developed by Robin Gandy, a
student of Alan Turing, in a 1980 article [9]. Gandy intended to complement
Turings analysis of human computers with an analysis of computation by
mechanical devices. He came up with an axiomatically defined class of computation devices, later named Gandy machines. The approach was adopted
2
by Wilfried Sieg. In article [22], Sieg uses Gandy machines to dispense with
[Churchs and Turings] theses. The article starts thus.
Churchs and Turings theses dogmatically assert that an informal notion of effective calculability is adequately captured by a
particular mathematical concept of computability. I present an
analysis of calculability that . . . dispenses with theses. . . . The
analysis leads to axioms for discrete dynamical systems (representing human and machine computations) and allows the reduction of models of these axioms to Turing machines.
We discuss this axiomatization of discrete dynamical systems and dispensing
with the theses in 4.
In 2, we discuss whether it is possible at all to define algorithms. (There
is also a question why bother to define algorithms. Well, understanding
what algorithms are should and does have practical applications, to
software specification and model-based testing in particular, as well as theoretical application, like semantics of software or algorithmic completeness of
computation models. But that is a different issue to be addressed elsewhere.)
In 3, we discuss and clarify a couple of misconceptions related to Turings
analysis of computations. In 4 we discuss Gandy machines. In 5, we discuss
what kind of entities algorithms are; this discussion is closely related to 6
where we discuss Moschovakiss recursor theory.
This article can be seen as a companion to our older article [5].
Acknowledgments
Many thanks to Andreas Blass for numerous illuminating discussions, and to
Serge Grigorieff and Oron Shagrir for useful suggestion.
Two articles [19] and [22], mentioned in 1, give different answers to the
title question. The two answers are not at all equivalent. A question arises
whether the notion of algorithm can be defined at all. The answer is yes and
no. Let us explain.
(and Godel and other experts) had in mind only sequential algorithms, which
we believe they did. The axiomatic definition was extended to synchronous
parallel algorithms in [3] and to interactive sequential algorithms in [6, 7].
length on the Turing tape. More generally, the functions computed by rulerand-compass algorithms are not Turing computable. And let us emphasize
that ruler and compass were practical tools in ancient Greece and that a
number of ruler-and-compass algorithms were practical algorithms.
It is common in mathematics to consider algorithms that work on abstract
objects. The functions computed by these algorithms may not be Turing
computable. One example is Gaussian elimination. Here is another example:
a bisection algorithm that, given a real > 0 and a continuous function f on
a real segment [a, b] with f (a) < 0 < f (b), computes a point c [a, b] with
|f (c)| < .
while |f ((a + b)/2)| do
if f ((a + b)/2) < 0 then a := (a + b)/2 else b := (a + b)/2
c := (a + b)/2
One can argue that these functions are not truly computable, that in
practice we can only approximate them. But then there are analog computers
in practical use that work in real time and compute functions that are not
Turing computable.
Of course Turing would not be surprised by our examples. He explicitly
restricted his analysis to symbolic (symbol-pushing, digital) algorithms.
He implicitly restricted his analysis to sequential algorithms, essentially the
only algorithms in his time. It is interesting that it turned out easier to
axiomatize all sequential algorithms [12], whether symbolic or not, including
the ruler-and-compass algorithms, Gaussian elimination and the bisection
algorithm (but excluding analog algorithms which are not sequential in our
sense).
What about quantum algorithms? Do they compute functions that are
not Turing computable? Erich Gradel and Antje Nowack checked that the
quantum computing models in the literature can be faithfully simulated by
parallel abstract state machines [11]. And, as we mentioned above, functions
computed by parallel ASMs are Turing computable.
There is also a rather common misunderstanding that Turing defined the
notion of algorithm, albeit restricted to symbolic sequential algorithms. Let
us restrict attention to such algorithms for a moment. Suppose that your
computation model (e.g. a programming language) is Turing complete. Does
it mean that the model allows you to express all algorithms? Not necessarily.
Turing machines simulate faithfully only the input-output behavior of algorithms. But there may be much more to algorithms than their input-output
6
Robin Gandy argues in [9] that Turings analysis of computation by a human being does not apply directly to mechanical devices. The reason is
that humans compute sequentially but machines can perform parallel computations. In this connection, Gandy analyzed computations by mechanical
devices and introduced (what we call now) Gandy machines.
A set-theoretic form of description for discrete deterministic machines is elaborated and four principles (or constraints) are enunciated, which, it is argued, any such machine must satisfy. . . . It
is proved that if a device satisfies the principles then its successive
states form a [Turing] computable sequence. [9, p. 123]
Note successive states. Gandy machines work in sequential time. This type
of parallelism is called synchronous. In the rest of this section, parallelism
will by default be synchronous.
Gandy pioneered the use of axioms in the analysis of computation. He
came up with four principles (or constraints, or axioms) satisfied, he claimed,
by all discrete deterministic machines. Contrast this with Turings analysis.
While Turings analysis was convincing, it is hard to isolate first principles
that, in Turings opinion, are satisfied by all symbolic sequential computations.
Wilfried Sieg adopted Gandys approach and reworked Gandys axioms;
see [23] and references there.
Critical remarks
In a 2002 article [20], Oron Shagrir suggests that there is an ambiguity regarding the types of machines that Gandy was postulating. He offers three
interpretations: Gandy machines as physical machines, Gandy machines
as finite-physical machines, and Gandy machines as a mathematical notion. Shagrir concludes that none of the three interpretations provides the
basis for claiming that Gandy characterized finite machine computation.
This agrees with our own analysis. By the way, for our purposes, there is no
difference between Gandys original axioms and Siegs versions of the axioms.
So we will just speak about Gandys axioms.
What real-world devices satisfy Gandys axioms? Probably very
few do. One problem is the form of the states of Gandy machines: a collection
of hereditary finite sets. Another problem is the requirement that state
transitions are synchronous. You, the reader, may say that we have not
proved our point. Well, the burden of proof is on the proponents of the
approach. And there are precious few examples in the papers of Gandy and
Sieg, and none of the examples is a real-world device. The most prominent
example in Gandys paper is the cellular automaton known as Conways game
of life. Note that a cellular automaton can grow without any bound. In the
real-world, such a cellular automaton would not stay synchronous.
It seems obvious that Gandy abstracts from material and views discrete
deterministic machines as algorithms, abstract algorithms. So Gandys claim
can be restated thus: parallel algorithms satisfy the axioms.
What algorithms satisfy Gandys axioms? Typical parallel or even
sequential algorithms do not satisfy the axioms. Consider for example a factorial algorithm. The state of the algorithm is naturally infinite and consists
of natural numbers. There is of course a Gandy machine that simulates the
factorial algorithm. Note that, in addition to simulating the factorial algorithm, the simulating machine may be forced to construct set representations
of additional numbers.
In our view, Gandys axioms are really used just to define another parallel
computation model. (By the way, it is our ambition in [3] that parallel
algorithms, on their natural abstraction levels, satisfy our axioms.)
transits from one state to the next until, if ever, it halts or breaks. The very
first postulate in our axiomatizations of sequential and synchronous parallel
algorithms [12, 3] is that the algorithms in question are sequential time.
The question arises what kind of entities states are. In our view, rather
common in computer science, algorithms are not humans or devices; they are
abstract entities. According to the second postulate in the axiomatizations
of sequential and synchronous parallel algorithms, the states are (first-order)
structures, up to isomorphism. This admittedly involves a degree of mathematical modeling and even arbitrariness. A particular form of structures is
used; why this particular form? But this is a minor detail. Structures are
faithful representations of states, and that is all that matters for our purposes.
It is convenient to declare that states are structures, up to isomorphism; but
there is no need to do so.
The point of view that sequential-time algorithms are state transition systems extends naturally to other classes of algorithms. In particular, a sequential-time interactive algorithm (until now we considered noninteractive algorithms) is a state transition system where a state transition
may be accompanied by sending and receiving messages. A distributed algorithm is an ensemble of communicating sequential-time interactive algorithms.
if x = 0
if x > 0
again, obtaining additionally exp(x, 1) = x for all x, and so on. After steps
(where is the first infinite ordinal), you reach a fixed point; now exp(x, y)
is defined for all x, y except x = y = 0. Often the evolution toward the fixed
point involves not only the function that you are computing but also some
auxiliary functions.
In 1934, Godel formulated a recursion-based calculus of (in general partial) numerical functions. Godels calculus can be seen as a specification
language where a specification of a function f is a system of recursive equations that, taking into account some global conventions, suggests a particular
(possibly inefficient) way to compute f . Churchs thesis (extended to partial
functions by Kleene) asserts that every effectively calculable, that is computable by an algorithm, function on natural numbers is programmable in
Godels calculus.
Recursive specification of functions has much appeal. It is declarative and
abstracts from computation details. It is often concise. There has been much
progress since the 1930s. Logicians developed recursion theory. McCarthy
created a functional (that is recursion-based) programming language LISP,
and many other functional languages followed.
The key ideas of Moschovakiss approach appear already in the 1984 article [16] that seems to be the very first publication on the subject.
If, by Churchs Thesis the precise, mathematical notion of recursive function captures the intuitive notion of computable function,
then the precise, mathematical notion of recursion . . . should
model adequately the mathematical properties of the intuitive
notion of algorithm. [16, p. 291]
Moschovakis discusses Euclids algorithm for the greatest common divisor of
two natural numbers. Then he says:
Following the drift of the discussion, we might be expected at
this point to simply identify the Euclidean algorithm with the
functional gcd. We will not go quite that far, because the timehonored intuitive concept of algorithm carries many linguistic and
intensional connotations (some of them tied up with implementations) with which we have not concerned ourselves. Instead we
will make the weaker (and almost trivial) claim that the functional gcd embodies all the essential mathematical properties of
the Euclidean algorithm. [16, p. 295]
11
Critical remarks
Recursors vs. algorithms We think that Moschovakis was right the
first time around, in [16, p. 295] when he refrained from identifying (what
he later called) recursors with algorithm because the time-honored intuitive
concept of algorithm carries many linguistic and intensional connotations
which are contrary to such identification.
Consider the system of two recursive equations (and thus a recursor) for
the exponentiation in the beginning of this section. Is it an algorithm or not?
The recursor certainly looks like an algorithm, and in many functional programming languages, this recursor would be a legitimate program (modulo
12
13
Recursion is but one aspect of an algorithm The theory of algorithms does not reduce to recursion. For one thing, there are clever data
structures. For many linear-time algorithms, for example, it is crucially important that an algorithm does not manipulate large objects directly; instead
it manipulates only pointers to those objects. Such aspects of complexity
analysis seem below the abstraction level of recursors.
Distributed algorithms The recursor approach does not seem to extend
to distributed algorithms, and the number of useful distributed algorithms
is large and growing.
Monotonicity limitation Here is something that the recursor theory
should be able to cover but doesnt. The current recursor theory is limited
to recursors with semantics given by the least fixed point of a monotone
operator. That is a serious limitation.
For a simple example consider Datalog with negation [1]. The operator defined by a Datalog-with-negation program is not monotone but it is
inflationary, and semantics is given by the inflationary fixed point [13].
For illustration, here is a Datalog-with-negation program computing the
complement C of the transitive closure T of a nonempty binary relation R
on a finite domain [1, Example 3.3].
T (x, y) R(x, y)
T (x, y) R(x, z), T (z, y)
U (x, y) T (x, y)
V (x, y) T (x, y), R(x0 , z 0 ), T (z 0 , y 0 ), T (x0 , y 0 )
C(x, y) T (x, y), U (x0 , y 0 ), V (x0 , y 0 )
Explanation. At every step all rules are fired. By the first two rules, the
computation of T proceeds in the usual way. Since the domain is finite, the
computation of T completes after some number k of steps. The pairs of T
are stored in U with a delay of one step, so the computation of U completes
after k + 1 steps. The computation of V is identical to that of U , except that
at the step k + 1, when U is completed, the last batch of pairs from T is not
stored in V . The final rule is idle during the first k steps but on step k + 1
it stores the complement of T into C.
14
References
[1] Serge Abiteboul and Victor Vianu, Datalog extensions for database
queries and updates, J. of Computer and System Sciences 43 (1991)
62124.
[2] Therese Biedl, Jonathan F. Buss, Erik D. Demaine, Martin L. Demaine,
Mohammadtaghi Hajiaghayi, Tomas Vinar, Palindrome recognition using a multidemensional tape, Theoretical Computer Science 302 (2003)
475480.
[3] Andreas Blass and Yuri Gurevich, Abstract state machines capture parallel algorithms, ACM Transactions on Computational Logic 4:4 (2003)
578651. Correction and extension, same journal, 9:3 (2008) article 19.
[4] Andreas Blass and Yuri Gurevich, Algorithms vs. machines, Bull. European Association for Theoretical Computer Science 77 (2002), 96118.
[5] Andreas Blass and Yuri Gurevich, Algorithms: A quest for absolute
definitions; in Current Trends in Theoretical Computer Science, World
Scientific, 2004, 195225; also in Churchs Thesis after 70 Years (eds. A.
Olszewski et al.), Ontos Verlag, 2006, 2457.
[6] Andreas Blass and Yuri Gurevich, Ordinary interactive small-step algorithms, ACM Trans. Computational Logic 7:2 (2006) 363419 (Part I),
plus 8:3 (2007), articles 15 and 16 (Parts II, III).
[7] Andreas Blass, Yuri Gurevich, Dean Rosenzweig, and Benjamin Rossman, Interactive small-step algorithms, Logical Methods in Computer
Science 3:4 (2007), papers 3 and 4 (Part I and Part II).
[8] Nachum Dershowitz and Yuri Gurevich, A natural axiomatization of
computability and proof of Churchs thesis, Bull. of Symbolic Logic
14:3 (2008) 299350.
15
[9] Robin Gandy, Churchs thesis and principles for mechanisms, In The
Kleene Symposium (eds. J. Barwise et al.), North-Holland, 1980, 123
148.
[10] S. Barry Cooper, Benedict Lowe and Andrea Sorbi, editors, New Computational Paradigms: Changing Conceptions of what is Computable,
Springer, 2008.
[11] Erich Gradel and Antje Nowack, Quantum computing and abstract
state machines, Springer Lecture Notes in Computer Science 2589
(2003), 309323.
[12] Yuri Gurevich, Sequential Abstract State Machines Capture Sequential
Algorithms, ACM Transactions on Computational Logic 1:1 (2000) 77
111.
[13] Yuri Gurevich and Saharon Shelah, Fixed-point extensions of firstorder logic, Annals of Pure and Applied Logic 32 (1986) 265280.
[14] Andrei N. Kolmogorov, On the concept of algorithm, Uspekhi Mat.
Nauk 8:4 (1953) 175176, Russian. English translation in [25, p. 1819].
[15] John McCarthy, A basis for a mathematical theory of computation,
in Computer Programming and Formal Systems (eds. P. Brafford and
D. Herschberg), North-Holland, 1963, 3370.
[16] Yiannis N. Moschovakis, Abstract recursion as a foundation of the theory of algorithms, in Computation and Proof theory, Springer Lecture
Notes in Mathematics 1104 (1984) 289364.
[17] Yiannis N. Moschovakis, What is an algorithm?, in Mathematics Unlimited 2001 and beyond (eds. B. Engquist and W. Schmid), Springer,
2001, 919936.
[18] Yiannis N. Moschovakis, Algorithms and implementations,
Tarski Lecture 1, March 3, 2008,
http://www.math.ucla.edu/~ynm/lectures/tlect1.pdf.
[19] Yiannis N. Moschovakis and Vasilis Paschalis, Elementary algorithms
and their implementations, in [10], 87118.
16
17