Newsgroups: comp.lang.smalltalk
Path: cantaloupe.srv.cs.cmu.edu!nntp.club.cc.cmu.edu!godot.cc.duq.edu!ddsw1!panix!bloom-beacon.mit.edu!gatech!howland.reston.ans.net!pipex!uunet!world!jhaungs
From: jhaungs@world.std.com (Jim Haungs)
Subject: Benchmarks: IBM VisualAge, Digitalk Visual ST, ParcPlace VisualWorks
Message-ID: <D1u7qy.5pF@world.std.com>
Summary: SLOP/SMOP benchmarks for three Smalltalk products
Keywords: benchmarks, performance, SLOP, SMOP, comparison
Organization: The World Public Access UNIX, Brookline, MA
Date: Tue, 3 Jan 1995 16:13:46 GMT
Lines: 118

Just for grins, I filed in Bruce Samuelson's SLOP and SMOP
benchmarks into several versions of the three major
Smalltalk products.  These benchmarks test a variety of low and
medium-level operations in an implementation-independent
manner.  Except for the differences in file-in formats 
between ParcPlace and the others, (methods vs methodsFor: category),
the source code is unchanged for all three systems.  

The source code for these is available on the Manchester
archives (StonesV.st) if anyone wants more details.

Methodologically, don't put too much stock in these 
benchmarks.  They only test low-level operations; they do
not test the windowing or disk subsystems at all, and 
they are not very representative of the mix of operations 
that occur in real applications.

I tested:
   Vw 2  - ParcPlace VisualWorks 2.0
   Vst 3 - Digitalk Visual Smalltalk 3.0
   Va 2  - IBM VisualAge 2.0

I also tested a couple of the previous versions:
   Vpm 2 - Smalltalk/V VOS2 (vm 2.11) 
   Vw 1  - VisualWorks 1.0 (vm 4.1D).

For VST, Int Add was decidedly faster in VOS2 2.0, but
almost everything else about VST seems to be faster
in the new version.  I also tested VST and PARTS 
separately, but the numbers were almost identical,
so loading all that extra "visual" baggage didn't seem
to affect the performance of the low-level stuff.

Most of the tests were about the same for VisualWorks 1.0 and 2.0,
with some features slightly faster, and others slightly slower.

As expected, the VisualAge performance is dismal
across the board.  Since this is the only byte-code
interpreter in the bunch (all the rest generate
machine code on the fly), this is to be expected.

Based on some non-scientific observations, each product's
video and windowing speed can be directly related to their 
use of native widgets.  Digitalk is fastest, with IBM coming 
in a close second, and ParcPlace a distant third. The respective 
video speed works against ParcPlace making it seem very slow, 
and it works for IBM, making its otherwise sluggish interpreter 
seem to put up windows very quickly.

I took a special look at IBM's high sorting speed
and discovered that IBM Smalltalk uses two different
sorting implementations depending on the size of the 
SortedCollection: one is an simple and slow insertion 
sort for small collections, and the other is a complex 
and fast heap sort for large collections. 
Since the SMOP benchmark creates a large collection to
stress the sorting algorithm, heap sort got chosen for the
implementation collection.  If the SMOP test used a smaller
collection, this number would drop significantly.

ParcPlace's sorting algorithm could be improved; it is by far
the slowest of all those tested.  I looked at the algorithm,
but the method is very poorly coded, and it does various things 
depending on the size of the collection.  It looks like it might be
a divide-and-conquer style like heapsort or quicksort, 
so it should be faster than it is, but I didn't have time 
to do a full analysis to determine the exact algorithm, 
or to find out where the speed penalty occurs.

Conclusions - for the OS/2 platform:
   Digitalk leads the pack on about half of the low-level tests, 
   and nearly all of the medium ones.  

   ParcPlace leads on the other half of the low-level tests, 
   and coasts squarely in the middle on all of the medium ones.  

   IBM is a really distant third on everything except sorting,
   where it nearly ties VST for the fastest.

An interesting comparison study could be done with
these three products in their MS Windows incarnations.
I'm told that the ParcPlace performance under Win32s
is substantially better than under OS/2, and I would be
interested to see if that's true, and to what extent, and
also whether it's true for the IBM and Digitalk products.

Eventually, VST and VA will be available on Unix, and these
too should be put through their paces.

Here's the raw data - note that none of this was done on a Pentium :-)
   Higher numbers mean better performance.
   Asterisks (*) follow particularly high numbers.
   At signs (@) follow particularly low numbers.
                       
Machine:  IBM ThinkPad laptop, 100mhz 486, 24megs RAM.
Oper Sys: OS/2 Warp 3.0

SLOP         Va 2  |   Vw 1    Vw 2  |  Vpm 2   Vst 3   
Int Add     0.710  |  3.085   2.646  |  3.846*  2.307   
Float Add   0.553  |  1.151   1.169  |  1.150   1.574   
Access      0.740  |  2.674   2.902  |  1.943   1.844   
Alloc       0.938  |  1.938   2.233  |  1.377   2.597   
Copy        0.839  |  1.708   1.408  |  1.256   1.410   
Perform     0.539  |  3.292*  3.292* |  0.319   1.821   
Eval Block  1.015  |  3.597*  3.597* |  0.408   0.658@   
Geom Mean   0.743  |  2.329   2.294  |  1.099   1.621   

SMOP         Va 2  |   Vw 1    Vw 2  |  Vpm 2   Vst 3
Fract       0.604  |  2.104   2.406  |  3.897   4.209   
Prime       1.056  |  2.246   2.994  |  2.955   3.302   
Streams     0.742  |  1.836   1.663  |  0.414@  2.948   
Strings     0.824  |  2.018   2.673  |  1.392   6.182*   
Sets        0.460  |  2.488   1.333  |  0.748   2.334   
Sorting     6.321* |  2.315@  2.492@ |  5.297   6.344   
Sorcerer    0.984  |  1.948   1.829  |  2.059   2.295   
Geom Mean   1.016  |  2.126   2.123  |  1.769   3.649   
                                    

