Subject: Re: Benchmark Results for CL's onWin32
From: Erik Naggum <erik@naggum.no>
Date: 1999/12/02
Newsgroups: comp.lang.lisp
Message-ID: <3153166270117604@naggum.no>

* "Martin Simmons" <martin@harlequin.co.uk>
| For best performance, you should also turn off all debugging support.

  all of these benchmarks seem to be testing fairly unusual conditions.
  the arguments we see here for "tuning" seem to me to imply that we have
  to chuck all the Lispy features we can think of to get a comparison.  I
  think this is completely bogus.  I'm not going to trust _myself_ enough
  to run my code with the kinds of optimization settings people advocate
  for these benchmarks.  I _want_ a lot of dynamism, debugging support,
  etc, and I'm not usually ready to run my code in production without these
  things, except in particular functions that I have determined to need it,
  and which have been specially written to be blindingly fast.  however, in
  the latter case, I'd be pretty damn stupid if I thought I could have 100%
  portable ANSI Common Lisp code _and_ super-duper maximal performance at
  the same time.  _sheesh_, guys.  if I had such needs as you imply with
  these declarations for "best performance", I would even hack the compiler
  output and look for instruction scheduling opportunities myself.  using a
  special function to turn off debugging, for instance, is just _one_ such
  measure to squeeze the bejeezus out of the system, and we're already
  outside of the portable ANSI Common Lisp domain.

  so in my view, you either want super-fast code and profile it and get the
  optimal results with hand-tweaked code that may not work anywhere else,
  or you use fully portable code under _normal_ condtions.  anything else
  is very close to cheating, rendering the benchmark results evidence of
  how good you are at cheating at random in each implementation with no way
  of comparing your skills and then _nothing_ else.  if you let instead
  consciously and openly do your extreme best to write fast code for each
  implementation, and _then_ compare the results, you at the very least get
  comparable results for comparable _people_ (i.e., experts).

  I suggest such benchmarks limit themselves to the narrowest possible type
  declarations, the highest SPEED and the lowest DEBUG and SAFETY, and
  either full local tweaking or no local tweaking.  if a declaration isn't
  enough, that's a factor that you need to take into account when and if
  you engage in _full_ local tweaking, and not at all until then.

#:Erik