Subject: Re: Lisp and Web Programming
From: (Rob Warnock)
Date: Fri, 17 Jun 2005 00:38:43 -0500
Newsgroups: comp.lang.lisp
Message-ID: <>
Fred Gilham <> wrote:
| Emre Sevinc <> wrote [or forwarded for a friend]:
| > The second problem is that CommonSQL has support for only few Common
| > Lisp interpreters and compilers. And those supported free ones does
| > interestingly not support Shebang Characters for CGI execution. Thus
| > I tried using modlisp for apache but that does not work as smooth as
| > I expected.
| If you google for Rob Warnock's postings to comp.lang.lisp you'll see
| that he has a method for dealing with this with CMU Lisp.  I suspect
| and hope that he will chime in shortly anyway. :-)

There have been many other useful answers in the past two weeks,
so I'll try not to duplicate their excellent advice, but here are
a few comments anyway specifically for the CMUCL environment:

1. If you're running on an operating system that supports more
   than one additional argument on a "#!" (e.g., FreeBSD), one
   of the ways to use CMUCL for CGI is to build an application-
   specific core image containing all of your CGI support & query-
   parsing & HTML-generation tools [whichever of the many excellent
   libraries you use]. You do this by loading all the stuff into
   a fresh CMUCL image and then use the function EXTENSIONS:SAVE-LISP
   to write out the image, specifying in the call a CGI application
   start-up function you've written (which just loads the CGI script,
   basically) that replaces the normal LISP::%TOP-LEVEL function, e.g.:

       ;; Build the core & exit
       (save-lisp "cgi.core"
		  :init-function #'cgi-init-function
		  :load-init-file nil          ; same as "-noinit"
		  :print-herald nil            ; same as CLISP's "-q"

   Then you can start your CGI script with:

       #!/usr/local/bin/cmucl -core /usr/local/lib/cmucl/lib/cgi.core

   Look at the "config.lisp" file in the CMUCL distribution for an
   example of saving a core, then tweak away. For a live example of
   CGI scripting with it, see <>.

   [Also see <> for the
   same thing done with CLISP.]

   NOTE: Despite the above, I no longer recommend this method. [See #2.]

2. Some of the objections to useing "mod_lisp" seemed to be along
   the lines of "But I don't control [the configuration of] my web
   server!" or "But what if the locally-available web server isn't
   Apache?". Well, besides FastCGI and selective HTTP-proxying (which
   Araneida can use behind Apache, for example) which others have
   already mentioned, you can write a *tiny* little C-based CGI program
   that just opens a socket [preferably a local Unix-domain socket,
   for security] to your persistent Lisp-based server [CMUCL or other]
   and passes it *EXACTLY* the same information that "mod_lisp" would
   pass it. That way, you can run your Lisp app behind *any* web server
   that supports CGI, but get (most of) the performance benefit of
   using "mod_lisp"... *and* be all set for the day the web server
   admin decides to allow you to install "mod_lisp" itself!

   I did that [mainly because of that web server admin issue], and
   called it "cgi_sock.c"; the executable is "sock.cgi" [so it doesn't
   have to be in /cgi-bin/, if the server has "*.cgi" enabled]. It's
   about 200 lines of C, but is that big only because it handles both
   GET & POST requests, and also contains a nice little compiled-in
   HTML error message that it serves up if there's any problem in
   connecting to the Lisp server socket.

   On a 1.4 Gz Athlon running FreeBSD, with Apache 1.3.26 executing
   it and with CMUCL-19a on the other end of the socket, "ab -n100"
   reports ~70 requests/sec for a simple "env.lhp" (Lisp-Handled Pages)
   that dumps the environment. [See <>, but
   please don't run perf tests; you'll eat my poor DSL line alive!]

   For comparison, a simple Perl-based "printenv" gets ~125 req/sec,
   and a /bin/sh script that just calls "env" gets ~175 req/sec.

   With "mod_lisp" you could probably get >200 such trivial reqs/sec,
   but for the apps I use "sock.cgi" for, even *1* req/sec is overkill.
   So it's quite "good enough" for my needs.

   [Compare the LHP version of <>
   to the above CGI examples. Note that ".lhp" pages can be just source,
   or compiled (this one is), and in either case are cached by the server
   for speed.]

To either method #1 or #2, add Tim Bradshaw's HTOUT or Edi Weitz's
CL-WHO for HTML generation, Eric Marsden's PG to talk to PostgreSQL,
and maybe Kevin Rosenberg's CL-MODLISP for the Lisp server front-end
(I had already written mine when CL-MODLISP appeared), and you have
a very nice web application environment.

3. Not for web/CGI per se, but just for basic "CL scripting", I hacked
   up "/usr/local/lib/cmucl/lib/site-init.lisp" [which gets run *before*
   the command-line switches are parsed!] to look for a new "-script"
   option, so you can say stuff like this:

       % cat ./test
       #!/usr/local/bin/cmucl -script
       (format t "hello world!~%")
       (loop for i from 1
	     and arg in *script-args*
	 do (format t "arg#~d = ~s~%" i arg))

       % ./test foo bar
       hello world!
       arg#1 = "foo"
       arg#2 = "bar"

   On that same 1.4 Gz Athlon under FreeBSD, that takes ~20 ms. to run,
   using an *unmodified* CMUCL-19a "lisp.core".  [Though note that
   to get the speed that low, I had to compile the "site-init.lisp"!
   And that time also assumes some other process on the system is
   also using (or has recently used) "lisp.core".]

   The same hack also supports concatenating "#!/usr/local/bin/cmucl -fasl"
   onto the front of a CMUCL ".x86f" file, for when the "script" starts
   getting too large to run fast enough when interpreted.

   [Yes, this *can* also be used for CGI scripting, but it's not as
   efficient as a CGI-specific core image (and *definitely* much
   less efficient than a persistent Lisp server daemon!), since with
   the "-script" approach you have to keep reloading your CGI/HTML/SQL
   libraries each time. Ugh.]


p.s. By the way, my apologies to the community for not having yet
made "cgi_sock.c" [and the "cl-modlisp"-like code that it talks to]
and the "site-init.lisp" "-script/-fasl" hacks generally available.
There seems to be a shortage of those round "tuit"s lately...

Rob Warnock			<>
627 26th Avenue			<URL:>
San Mateo, CA 94403		(650)572-2607