Newsgroups: comp.lang.smalltalk
Path: cantaloupe.srv.cs.cmu.edu!das-news2.harvard.edu!news2.near.net!news.mathworks.com!hookup!swrinde!howland.reston.ans.net!torn!nott!cunews!tina.mrco.carleton.ca!knight
From: knight@mrco.carleton.ca (Alan Knight)
Subject: Re: Digitalk... The end...
Message-ID: <knight.793951310@tina.mrco.carleton.ca>
Sender: news@cunews.carleton.ca (News Administrator)
Reply-To: knight@mrco.carleton.ca (Alan Knight)
Organization: Carleton University
References: <knight.793847358@tina.mrco.carleton.ca> <3itpsb$200@news1.delphi.com>
Date: Tue, 28 Feb 1995 06:01:50 GMT
Lines: 95

In <3itpsb$200@news1.delphi.com> jsutherland@BIX.com (Jeff Sutherland) writes:
>I still fail to see how benchmarks that relate to swapping overhead due to
>memory limitations are any worse than Smopstones which test low level math
>calculations and garbage collection that is not usually the performance
>bottleneck for business applications.

Then I guess I'll have to explain it. SMOPstones, while hardly ideal,
are at least attempting to measure the basic performance of the
execution engine. This has some chance of producing some meaningful
numbers. They aren't going to be nearly as meaningful as benchmarks of
your own application, but they're _much_ easier to get. One of the
thing I mean by meaningful is stable. I expect, for example, that
execution speed should vary more or less linearly with CPU speed.
Because of this property, I can abstract out a wide variety of
benchmark calculations on different machines to a few numbers which I
can think about as "speeds". e.g. I can say ParcPlace averages 5.23
IStones (Imaginy benchmark) on a Mac QT777 and 10.46 Istones on an IBM
Inkbat 999. Thus I can expect very roughly a factor of two in
performance between these two machines. This is useful.

Benchmarks related to swapping have no (well, almost no) relation to
CPU speed. Instead they vary discontinuously with memory on the
machine. This means that the most effective way to express them is to
say e.g. On a Zorch Dingbat 534, PowerVisualObjectStuff requires 100MB
to run. This is very simple, because you have one number, above which
performance can be measured normally, and below which it is
unacceptable. This is useful.

What is not useful is to exploit the difference in memory requirements
between two products and to compare execution speed without swapping
on one and with swapping on the other. This makes for either very bad
benchmarking, or for a deliberate attempt to confuse the issue.

>Business applications need snappy screen performance, good network
>connectivity, and fast native bindings to all major database systems.

First of all, when did we restrict ourselves to business applications.
In particular, I'd say you've restricted the domain to client-side
database-related client-server applications. Granted, that's where
your marketing effort is concentrated, but it is by no means the only
area where Smalltalk work is being done.

Specifically

>snappy screen performance

This is generally related to the operating system and to the
underlying execution speed. It's hard to display data faster than you
can compute it. Some implementations will get better screen
performance relative to their basic execution speed. This could be
because they've coded it better, because of differences in the
paradigm they're using (e.g. ParcPlace's emulation is generally slower
than native controls) or because they've sacrificed something to get
the speed (e.g. sacrifice some of Smalltalk's generality by
implementing your screen code in primitives).  
   Thus, it would be useful to have a benchmark that reflected speed
of the windowing system. I don't think SMOPstones include such a
benchmark because the windowing code is the least portable between
different Smaltlltalks. Since Bruce Samuelson does those benchmarks in
his spare time out of interest, I think he can be forgiven this. It
would be nice if someone could make such a benchmark, though.

>good network connectivity, and fast native bindings to all major
>database systems.

This is certainly true, and an important consideration. There are lots
of differences between the database support in different
implementations (and lots of third-party add-ons too). I'd like to see
some meaningful database benchmarks.

>  They
>need good design environments that lead to high productivity and well
>designed systems that scale to the enterprise.  Most MIS shops find a 16
>bit version of raw Smalltalk inadequate even if it will run fast in 4MB
>(irrespective of vendor).

Certainly these are useful things. In many cases, particularly
client-server database applications, they dwarf the importance of
basic execution speed. Unfortunately, they're also much harder to
quantify than basic execution speed. Nevertheless, that doesn't mean
these benchmarks are meaningless. If they're not important to a
particular app, ignore them. Don't make up distorted versions to compensate. 

Why would MIS shops find a small, fast 16-bit Windows version
unacceptable? In particular, why would they find it so if memory
footprint is their overwhelming consideration against Smalltalk?
Presumably lots of these small machines are still running 16-bit
Windows. They certainly aren't running NT, and OS/2 does just fine
with 16-bit Windows apps.

-- 
 Alan Knight                | The Object People
 knight@acm.org             | Smalltalk and OO Training and Consulting
 alan_knight@mindlink.bc.ca | 509-885 Meadowlands Dr.
 +1 613 225 8812            | Ottawa, Canada, K2C 3N2
