Information Processing Letters 79 (2001) 223–227

Analyzing variants of Shellsort

Bronislava Brejová 1
Department of Computer Science, University of Waterloo, Waterloo, ON N2L 3G1, Canada

Received 30 January 2000; received in revised form 24 November 2000

Communicated by P.M.B. Vitányi

Abstract
We consider two variants of Shellsort — Dobosiewicz sort and Shaker sort. Not much is known about the running time of
these algorithms. We prove that the worst-case time of Shaker sort is O(n3/2 log3 n) for certain sequences of increments and
the average-case time for both variants is (n2 /cp ) where p is the number of increments and c = 2 for Dobosiewicz sort and
c = 4 for Shaker sort. In our proofs we adapt techniques and results from analysis of Shellsort.  2001 Elsevier Science B.V.
All rights reserved.
Keywords: Analysis of algorithms; Sorting; Kolmogorov complexity

1. Introduction symmetrical and probably has better properties (we

will call it Shaker sort).
Shellsort is an algorithm that sorts a sequence in Shellsort is guaranteed to sort each sequence as
several stages. Each stage uses some given distance far as the last increment is 1. The variants, on the
h. The sequence is divided into subsequences, each contrary, may leave a sequence unsorted even if the
containing the elements that were in the sequence h last increment is 1. Therefore a special “mop-up”
elements apart. These subsequences are sorted using phase is required to finish the sort. We will assume that
Insertion sort. Time complexity of Shellsort depends Insertion sort is used as the last phase. Other solutions
on a choice of distances used in individual phases were also considered in the literature.
(these distances are often called increments). Unlike Shellsort, which has been widely studied,
Several other variants of Shellsort were proposed in there is hardly any work concerning the two variants.
literature. Dobosiewicz [1] proposed to use only one Early papers [1,2,9] as well as survey [8] bring
pass of Bubblesort for each subsequence instead of mainly empirical results. These results suggest that
sorting the subsequences in each stage. 2 Incerpi and the variants have good running time (comparable with
Sedgewick [2] suggest using two passes of Bubble-
Shellsort) on random sequences of a moderate length
sort in each stage, one going from left to right and
and that the worst-case time might be quadratic for
the other from right to left. This variant is more
increment sequences that perform well in tests with
random sequences.
√
Poonen in [7] proves lower bound
E-mail address: [email protected] (B. Brejová). 1+c/ p
1 Supported in part by NSERC Research Grant OGP0046506. (n ) for a sequence of p increments. This
2 This idea was presented already by Knuth [5, exercise 5.2.1.40, bound holds for a class of “Shellsort-type” algorithms
p. 105]. including Shellsort, Shaker sort and Dobosiewicz
0020-0190/01/$ – see front matter  2001 Elsevier Science B.V. All rights reserved.
PII: S 0 0 2 0 - 0 1 9 0 ( 0 0 ) 0 0 2 2 3 - 4
224 B. Brejová / Information Processing Letters 79 (2001) 223–227

sort. Moreover for nearly geometric sequences of When we exchange an inversion pair in a sequence
increments (increments of the form (α i )) Poonen of 0’s and 1’s having x elements between them, we
proves the lower bound (n2 ) which holds for Shaker remove exactly x + 1 inversions. Thus the ith pass
sort and Dobosiewicz sort. removes at least 2(k − i) + 1 inversions, and together
In this article we show that Shaker sort runs in in all k passes we remove at least
O(n3/2 log3 n) time for certain sequence of incre-
(2k − 1) + (2k − 3) + (2k − 5) + · · · + 1 = k 2
ments. This is the first known sub-quadratic upper
bound for this algorithm. In the construction we use Therefore if the original sequence has m
inversions. √
known results for Shellsort. inversions, m passes suffice to sort it. ✷
We also prove an (n2 /4p ) lower bound for the
average case of Shaker sort and (n2 /2p ) for Do- Note, that the previous lemma does not hold for Do-
bosiewicz sort where p is the number of increments. bosiewicz sort. Sequence containing n−1 occurrences
The proof uses incompressibility method similarly as of 1 followed by one zero has n − 1 inversions and re-
it is done in [4,3]. quires n − 1 passes of Dobosiewicz sort with incre-
Throughout the paper we use log n to denote the ment 1 to become sorted.
logarithm with base 2 (i.e., log2 n).
Theorem 2. Let H be a sequence of p increments
such that Shellsort for this sequence uses at most m
2. Upper bound for Shaker sort exchanges to sort any input sequence. Then there is an

√ sequence H for which Shaker sort works in
increment
Here we will show how to use results for Shell- O(pn m ) time.
sort to construct an increment sequence for Shaker sort
that has a sub-quadratic worst-case running time. Let Proof. We will√ construct H  very simply: each h in H
1-shake denote one stage of Shaker sort using incre- is replaced by  √m  occurrences of h. Such sequence
ment h = 1 (i.e., sequence of compare-exchange op- H  has length  m p and√thus all stages of Shaker
erations for pairs (x1 , x2 ), (x2 , x3 ), . . . , (xn−2 , xn−1 ), sort together require O(np m ) time. We will prove
(xn−1 , xn ), (xn−2 , xn−1 ), . . . , (x2 , x3 ), (x1 , x2 )). Inver- that after using this sequence we always obtain sorted
sion in a sequence x1 , x2 , . . . , xn is any pair (i, j ) such sequence and the final Insertion sort is not needed (and
that i < j and xi > xj . if it is used√it works in O(n) time). Therefore we have
time O(np m ).
Lemma 1. Let X be a sequence √ of 0’s and 1’s In order to prove that sequence H  sorts every input
containing m inversions. Then m passes of 1-shake we will use 0–1 principle. 0–1 principle states that if
are sufficient to sort it. a comparator network sorts every input sequence of
zeroes and ones, then it sorts every input sequence
Proof. Let us assume that sequence X requires ex- (for the proof see, for example, in [5, p. 224]). We can
actly k passes of 1-shake to become sorted. In each easily express Shaker sort (without the final Insertion
of these passes the leftmost 1 is exchanged with the sort) as a comparator network and therefore 0–1
rightmost 0, the rest of sequence remaining unchanged principle applies to Shaker-sort as well (this statement
(see [9]). Consider the situation in the ith pass. There was proved in [9]).
are some 0’s at the beginning followed by the leftmost Therefore we need to prove that Shaker sort with
1. Then there is some arbitrary subsequence of length increments H  sorts every 0–1 sequence. Assume that
x followed by the right-most 0 and some sequence of such sequence is used as an input to Shellsort with
1’s. Neither 0’s at the beginning nor 1’s at the end are increments H . We know that at most m exchanges
involved in exchanges in the following k − i passes. are done for this sequence during Shellsort. Take a
Still in each pass we need to exchange 2 elements and stage of Shellsort using some increment h. In this stage
therefore there are at least 2(k − i) elements between h subsequences are sorted independently and in each
the left-most 1 and the right-most 0 (more precisely such sort at most m exchanges are done. It means that
there are at least k − i zeroes and at least k − i ones). each such subsequence contains at most m inversions.
 B. Brejová / Information Processing Letters 79 (2001) 223–227 225

√
Therefore it can be sorted√ using m passes of 1- Assume that the input sequence is some permuta-
shake (Lemma 1). Using m stages of Shaker sort tion π of the set {1, 2, . . . , n}. Let π  be a permutation
with increment h therefore sorts every subsequence which we obtain after all p stages of Dobosiewicz sort
achieving the same result as was achieved by Shellsort (before the final Insertion sort). Let X be the number
in one stage with increment h. Shellsort obtains a of inversions in π  . Then the final Insertion sort works
sorted sequence after all stages, Shaker sort therefore in (X) time. We will prove that X = (n2 /2p ) in
also obtains a sorted sequence. ✷ the average case.
We will give a description that uniquely describes
The best known upper bound for Shellsort is any permutation π of n elements. The description is
O(n log2 n) for the sequence of the length p = divided into two parts, the first one provides informa-
(log2 n) containing all numbers of the form 2i 3j tion which enables us to construct π once we know
smaller than n. This sequence was proposed by π  and the second one describes permutation π  . The
Vaughan Pratt in 1969, for details see, for exam- first part consists of p strings of length n. Bit j of the
ple, [7]. Let us apply the theorem for Pratt’s incre- ith string determines whether xj was exchanged with
ments (where p = O(log2 n) and m = O(n 2
√ log n)). xj −hi during the ith stage of the algorithm (where hi

We obtain sequence H of the length O( n log3 n) is the ith increment). Given this information and per-
which gives us O(n3/2 log3 n) time for Shaker sort. mutation π  we can simulate the stages backwards and
we can construct permutation π .
Theorem 3. There is a sequence of increments for Permutation π  is described using its inversion ta-
Shaker sort such that Shaker sort works in ble a1 , a2 , . . . , an where ai is the number of elements
O(n3/2 log3 n) time in the worst case. greater than πi which 

are to the left of πi in per-
mutation π  . Sum ni=1 ai is exactly X. There are
 
D(X) = X+n−1 n−1 sequences of n non-negative num-
3. Lower bound for average case bers with sum X and therefore π  can be described
as number X followed by the index of a1 , a2 , . . . , an
We prove a lower bound on the average case time in some fixed enumeration of all sequences with sum
complexity of Dobosiewicz sort and Shaker sort using X. To get prefix-free description we need log X +
incompressibility method. The argument is similar to log D(X) + 2 log log X bits. Together we need np +
the lower bound proof for the average case of Shellsort log X + log D(X) + 2 log log X bits to encode any per-
in [4]. We use conditional Kolmogorov complexity of
mutation π (denote  this value L(π)). Using the for-
a string, denoted C(x|y). It is the length of the shortest
mula for log ab given in [4] we get
binary string w such that if strings y and w are used as
 
an input for a fixed universal Turing machine, string x X+n−1
will be the result of the computation. For more details log D(X) = log
n−1
about the Kolmogorov complexity see [6].  
X+n

log
Theorem 4. Let H be a sequence of p increments. n
Dobosiewicz sort with increments H has the average- X+n X+n
= n log + X log
case running time (n2 /2p ), and Shaker sort with n X
increments H has the average-case running time 1 X+n
+ log + O(1).
(n2 /4p ). 2 Xn
The second term is equal to log(1 + Xn )X and it is
Proof. First let us prove the result for Dobosiewicz
at most n log e. The third term is O(log n) because
sort. Each comparison-based sorting algorithm needs
X
n2 . Using this we obtain the following bounds
(n log n) time in average case. For p > log n −
for log D(X) and L(π)
log log n we get the bound (n2 /2p ) directly from this
 
trivial lower bound. Therefore further we will assume X
that p < log n − log log n. log D(X)
n log + 1 + O(n),
n
226 B. Brejová / Information Processing Letters 79 (2001) 223–227

L(π)
np + log X + log D(X) + 2 log log X in case of Shaker sort. On the other hand, for p
 
X sufficiently small it gives even better lower bounds

np + n log + 1 + O(n). that the best known lower bounds for the worst case
n √
(n1+c/ p ) [7].
Now let P be a program that produces permutation
We have also shown that the worst-case running
π given n, p, H and our encoding. Let C(π|n, p,
time of Shaker sort is sub-quadratic for some incre-
H, P ) be the conditional Kolmogorov complexity
ment sequences. Unfortunately, the sequences con-
of π given n, p, H and P . From the definition
structed in the proof are not useful in practice, because
of Kolmogorov complexity we have that L(π) 
Shaker sort with such sequences runs for most inputs
C(π|n, p, H, P ). Now we will use the property, that
much longer than Shellsort.
most permutations do not have a short description
Our results provide at least some non-trivial bounds
(they are incompressible). More precisely, it is pos-
for the two variants of Shellsort, however there is still
sible to prove by a simple counting argument, that
a huge gap between the lower and upper bound in the
there are at least (1 − 1/n)n! permutations such that
worst case and no upper bound whatsoever is known
C(π|n, p, H, P )  log(n!) − log n (see [4]). Let π be
for the average case. Also we have no upper bound
such permutation. Then our description must have at
for Dobosiewicz sort leaving the possibility that it has
least log(n!) − log n = n log n − (n) bits and there-
worst-case complexity (n2 ) for any increments.
fore
  Comparing Shellsort with its variants, it seems
X
np + n log + 1 + O(n)  n log n − (n), that the variants do not have any advantage. They
n are about as difficult to implement as Shellsort and
2  2
n n practical tests give comparable results. Best known
X p −n= p . lower bounds for variants are slightly worse than
2 (1) 2
bounds for Shellsort and Shellsort has much better
The last step holds because p
log n − log log n and
upper bound.
which means n2 /2p  n log n.
However increment sequences for Shellsort were
The running time for permutation π is therefore
much more studied that sequences for its variants.
(n2 /2p ). This lower bound applies to at least (1 −
Therefore it might be possible that there is a sequence
1/n)n! permutations and therefore the average com-
for which Shaker sort outperforms the best increments
plexity is at least
   2  2 of Shellsort either in experimental results or in the
1 n n theoretical analysis of the average case.
1−  p = p .
n 2 2
The lower bound for Shaker sort can be proved
analogously, only we need 2n bits to encode one Acknowledgement
stage of the algorithm instead of n bits used for
Dobosiewicz sort. Thus we get lower bound of the I thank Ming Li for many useful comments.
form (n2 /4p ). ✷
References
4. Conclusion
[1] W. Dobosiewicz, An efficient variation of bubble sort, Inform.
Process. Lett. 11 (1) (1980) 5–6.
We have proved lower bound for the average- [2] J. Incerpi, R. Sedgewick, Practical variations of Shellsort,
case of the form (n2 /cp ) for both Shaker sort and Inform. Process. Lett. 26 (1) (1987) 37–43.
Dobosiewicz sort. This lower bound gives non-trivial [3] T. Jiang, M. Li, P. Vitányi, Average-case complexity of Shell-
results only for sort, in: Proc. 26th International Colloquium on Automata, Lan-
guages, and Programming, Prague, Czech Republic, 11–15 July,
p
log n − log log n 1999, Lecture Notes in Comput. Sci., Vol. 1644, Springer,
in case of Dobosiewicz sort, and Berlin, 1999, pp. 453–462.
[4] T. Jiang, M. Li, P. Vitányi, A lower bound on the average-case
p
12 (log n − log log n) complexity of Shellsort, J. ACM 47 (5) (2000) 905–911.
 B. Brejová / Information Processing Letters 79 (2001) 223–227 227

[5] D.E. Knuth, Sorting and Searching, The Art of Computer Annual European Symposium, Lecture Notes in Comput. Sci.,
Programming, Vol. 2, Addison-Wesley, Reading, MA, 1973. Vol. 1136, Barcelona, Spain, 25–27 September 1996, Springer,
[6] M. Li, P. Vitányi, An Introduction to Kolmogorov Complexity Berlin, 1996, pp. 1–11.
and its Applications, 2nd edn., Springer, New York, 1997. [9] M.A. Weiss, R. Sedgewick, Bad cases for Shaker-sort, Inform.
[7] B. Poonen, The worst case in Shellsort and related algorithms, Process. Lett. 28 (3) (1988) 133–136.
J. Algorithms 15 (1) (1993) 101–124.
[8] R. Sedgewick, Analysis of Shellsort and related algorithms,
in: J. Díaz, M. Serna (Eds.), Algorithms—ESA’96, Fourth