Buildapp - Create executables with SBCL or CCL

Buildapp is an application for SBCL or CCL that configures and saves an executable Common Lisp image. It is similar to cl-launch and hu.dwim.build. Buildapp is available under a BSD-style license. The latest version is 1.5.6, released on November 7th, 2015.

Download shortcut: http://www.xach.com/lisp/buildapp.tgz

Installation
Example Use
Overview
Limitations
Implementation
Feedback
License

Installation

Buildapp does not require any libraries. To compile it with SBCL you simply run make install. To compile with a different lisp, just specify it after the make command: make install LISP=<my_favorite_lisp>. For example, you wanted to compile it under CCL you'd say make install LISP=ccl By default, it is installed in /usr/local/bin; to use another location, use make DESTDIR=/path install.

You can also create the buildapp binary by loading the buildapp system with asdf and running (buildapp:build-buildapp).

Example Use

Here's a small application:

$ buildapp \
    --eval '(defun main (argv) (declare (ignore argv)) (write-line "Hello, world"))' \
    --entry main \
    --output hello-world
[undoing binding stack and other enclosing state... done]
[saving current Lisp image into hello-world:
writing 6352 bytes from the read-only space at 0x20000000
writing 4064 bytes from the static space at 0x20100000
writing 44834816 bytes from the dynamic space at 0x1000000000
done]

$ ./hello-world
Hello, world

The following creates a toy curl-like application. (It's not quite practical, because any errors will land you in the interactive debugger.)

$ buildapp --output lisp-curl --asdf-path ~/src/clbuild/systems/ \
    --load-system drakma \
    --eval '(defun main (args) (write-string (drakma:http-request (second args))))' \
    --entry main
;; loading system sb-grovel (needed by drakma)
;;  from /usr/local/lib/sbcl/sb-grovel/
;; loading system sb-posix (needed by cl+ssl)
;;  from /usr/local/lib/sbcl/sb-posix/
;; loading system trivial-gray-streams (needed by chunga, cl+ssl, flexi-streams)
;;  from /home/xach/src/clbuild/source/trivial-gray-streams/
;; loading system flexi-streams (needed by drakma, cl+ssl)
;;  from /home/xach/src/clbuild/source/flexi-streams/
;; loading system alexandria (needed by cffi, babel)
;;  from /home/xach/src/clbuild/source/alexandria/
;; loading system trivial-features (needed by cffi, babel)
;;  from /home/xach/src/clbuild/source/trivial-features/
;; loading system babel (needed by cffi)
;;  from /home/xach/src/clbuild/source/babel/
;; loading system cffi (needed by cl+ssl)
;;  from /home/xach/src/clbuild/source/cffi/
;; loading system cl+ssl (needed by drakma)
;;  from /home/xach/src/clbuild/source/cl+ssl/
;; loading system sb-bsd-sockets (needed by usocket)
;;  from /usr/local/lib/sbcl/sb-bsd-sockets/
;; loading system usocket (needed by drakma)
;;  from /home/xach/src/clbuild/source/usocket/
;; loading system chunga (needed by drakma)
;;  from /home/xach/src/clbuild/source/chunga/
;; loading system cl-base64 (needed by drakma)
;;  from /home/xach/src/clbuild/source/cl-base64/
;; loading system puri (needed by drakma)
;;  from /home/xach/src/clbuild/source/puri/
;; loading system drakma 
;;  from /home/xach/src/clbuild/source/drakma/
[undoing binding stack and other enclosing state... done]
[saving current Lisp image into lisp-curl:
writing 6352 bytes from the read-only space at 0x20000000
writing 5472 bytes from the static space at 0x20100000
writing 61722624 bytes from the dynamic space at 0x1000000000
done]

$ ./lisp-curl http://xach.com/
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
       "http://www.w3.org/TR/REC-html40/loose.dtd">
<HTML>
<HEAD>
<TITLE>www.xach.com</TITLE>
</HEAD>
...

Here's how the l1sp.org redirection service application is built:

$ make
buildapp --output l1sp --entry redirector:main \
                 --asdf-path /opt/l1sp/systems \
                 --require sb-aclrepl \
                 --eval '(pushnew :hunchentoot-no-ssl *features*)' \
                 --load-system swank \
                 --eval '(setf swank:*log-output* nil)' \
                 --load-system redirector
;; loading system sb-grovel
;;  from /usr/local/lib/sbcl/sb-grovel/
;; loading system sb-bsd-sockets
;;  from /usr/local/lib/sbcl/sb-bsd-sockets/
;; loading system sb-introspect
;;  from /usr/local/lib/sbcl/sb-introspect/
;; loading system sb-posix
;;  from /usr/local/lib/sbcl/sb-posix/
;; loading system sb-cltl2
;;  from /usr/local/lib/sbcl/sb-cltl2/
;; loading system swank
;;  from /opt/l1sp/src/slime/
;; loading system html-template (needed by redirector)
;;  from /opt/l1sp/src/html-template-0.9.1/
;; loading system sb-rotate-byte (needed by sb-md5)
;;  from /usr/local/lib/sbcl/sb-rotate-byte/
;; loading system sb-md5 (needed by redirector)
;;  from /usr/local/lib/sbcl/sb-md5/
;; loading system cl-who (needed by redirector)
;;  from /opt/l1sp/src/cl-who-0.11.1/
;; loading system cl-ppcre (needed by hunchentoot, redirector)
;;  from /opt/l1sp/src/cl-ppcre-2.0.0/
;; loading system url-rewrite (needed by hunchentoot)
;;  from /opt/l1sp/src/url-rewrite-0.1.1/
;; loading system rfc2388 (needed by hunchentoot)
;;  from /opt/l1sp/src/rfc2388/
;; loading system md5 (needed by hunchentoot)
;;  from /opt/l1sp/src/md5-1.8.5/
;; loading system cl-fad (needed by hunchentoot)
;;  from /opt/l1sp/src/cl-fad-0.6.2/
;; loading system cl-base64 (needed by hunchentoot)
;;  from /opt/l1sp/src/cl-base64-3.3.2/
;; loading system trivial-gray-streams (needed by flexi-streams)
;;  from /opt/l1sp/src/trivial-gray-streams-2006-09-16/
;; loading system flexi-streams (needed by chunga)
;;  from /opt/l1sp/src/flexi-streams-1.0.7/
;; loading system chunga (needed by hunchentoot)
;;  from /opt/l1sp/src/chunga-0.4.3/
;; loading system hunchentoot (needed by redirector)
;;  from /opt/l1sp/src/hunchentoot-0.15.7/
;; loading system redirector
;;  from /opt/l1sp/src/redirector/
[undoing binding stack and other enclosing state... done]
[saving current Lisp image into l1sp:
writing 6176 bytes from the read-only space at 0x20000000
writing 4064 bytes from the static space at 0x20100000
writing 61042688 bytes from the dynamic space at 0x1000000000
done]

$ ./l1sp
;; Swank started at port: 7717.
CL-USER(1):

The (setf swank:*log-output* nil) eval is needed to avoid problems when the image restarts.

redirector::main looks like this:

(defun main (argv)
  (declare (ignore argv))
  (load "/opt/l1sp/etc/init.lisp")
  (sb-impl::toplevel-repl nil))

Here's an example of the --dispatched-entry option, which was inspired by the desire to have a dozen different small utilities embedded in one big executable and called based on the binary name. First, the support files:

;;;; utils.lisp

(defpackage #:utils
  (:use #:cl))

(in-package #:utils)

(defun main (argv)
  (let ((name (pathname-name (first argv))))
    (format *error-output*
            "Unknown binary name ~S, try using cl-echo, cl-ls, or cl-true~%"
            name)
    (sb-ext:quit :unix-status 1)))

;;;; ls.lisp

(defpackage #:ls
  (:use #:cl))

(in-package #:ls)

(defun main (argv)
  (declare (ignore argv))
  (dolist (file (directory "*.*"))
    (write-line (namestring file))))

;;;; echo.lisp

(defpackage #:echo
  (:use #:cl))

(in-package #:echo)

(defun main (argv)
  (format t "~{~A~^ ~}~%" (rest argv)))

;;;; true.lisp

(defpackage #:true
  (:use #:cl))

(in-package #:true)

(defun main (argv)
  (declare (ignore argv))
  (sb-ext:quit :unix-status 0))

Buliding it all together looks like this:

$ buildapp --output utils \
   --load utils.lisp --dispatched-entry /utils:main \
   --load ls.lisp --dispatched-entry cl-ls/ls:main \
   --load echo.lisp --dispatched-entry cl-echo/echo:main \
   --load true.lisp --dispatched-entry cl-true/true:main
;; loading file #P"/tmp/demo/utils.lisp"
;; loading file #P"/tmp/demo/ls.lisp"
;; loading file #P"/tmp/demo/echo.lisp"
;; loading file #P"/tmp/demo/true.lisp"
[undoing binding stack and other enclosing state... done]
[saving current Lisp image into utils:
writing 6352 bytes from the read-only space at 0x20000000
writing 4064 bytes from the static space at 0x20100000
writing 45223936 bytes from the dynamic space at 0x1000000000
done]

$ ln -sf utils cl-ls

$ ln -sf utils cl-echo

$ ln -sf utils cl-true

$ ./cl-ls
/tmp/demo/cl-true
/tmp/demo/echo.lisp
/tmp/demo/ls.lisp
/tmp/demo/true.lisp
/tmp/demo/utils
/tmp/demo/utils.lisp

$ ./cl-echo Hello world
Hello world

$ ./cl-true && echo $?
0

Overview

Here is the usage output of buildapp:

Usage: buildapp --output OUTPUT-FILE [--flag1 value1 ...]

Required flags:
  --output OUTPUT-FILE      Use OUTPUT-FILE as the name of the executable
                              to create

Entry-point flags:
  --entry NAME              Use the function identified by NAME as the
                              executable's toplevel function. Called
                              with SB-EXT:*POSIX-ARGV* as its only
                              argument. If NAME has a colon, it is
                              treated as a package separator,
                              otherwise CL-USER is the implied
                              package.
  --dispatched-entry DNAME  Specify one possible entry function, depending
                              on the name of the file that is used to
                              start the application. The syntax of
                              DNAME is APPLICATION-NAME/ENTRY-NAME. If the
                              name used to start the executable matches
                              APPLICATION-NAME, use ENTRY-NAME as the
                              entry point. This can be used to choose
                              one of many possible entry points by
                              e.g. symlinking names to the application
                              executable. If APPLICATION-NAME is empty, the
                              specified ENTRY-NAME is used as a default
                              if no other application names match. There
                              may be any number of dispatched entry points,
                              but only one default.

Action flags:
  --load FILE               Load FILE. CL:*PACKAGE* is bound to the CL-USER
                              package before loading
  --load-system NAME        Load an ASDF system identified by NAME
  --require NAME            Use CL:REQUIRE to load NAME
  --eval CODE               Use CL:EVAL to evaulate CODE. The code is read
                              with CL:READ-FROM-STRING in the CL-USER package

There may be any number of load/load-system/require/eval flags. Each
is executed in command-line order before creating an executable.

Load path flags:
  --load-path DIRECTORY     When handling a --load, search DIRECTORY for
                              files to load
  --asdf-path DIRECTORY     When handling a --load-system, search DIRECTORY
                              for ASDF system files to load
  --asdf-tree DIRECTORY     When handling a --load-system, search DIRECTORY
                              and all its subdirectories for ASDF system
                              files to load
  --manifest-file FILE      When handling a --load-system, read a list of
                              ASDF system file pathnames from FILE as
                              possible matching systems.

There may be any number of load-path/asdf-path/asdf-tree/manifest-file
flags. They take priority in command-line order.

Other flags:
  --compress-core           Compress the core or executable; requires
                              configuration support in SBCL
  --core-only               Make a core file only, not an executable
  --dynamic-space-size MB   Pass a --dynamic-space-size option to SBCL
                              when building; value is megabytes
  --help                    Show this usage message
  --logfile FILE            Log compilation and load output to FILE
  --sbcl PATH-TO-SBCL       Use PATH-TO-SBCL instead of the sbcl program
                              found in your PATH environment variable

For the latest documentation, see http://www.xach.com/lisp/buildapp/

Limitations

Buildapp is limited in scope. It aims to make the following steps easy:

Define an application environment by loading files, loading ASDF systems, evaluating code, and using CL:REQUIRE
Dump an executable image with an arbitrary startup function

By design, it does not handle the following tasks:

Initialization tasks at app startup (use SB-EXT:*INIT-HOOKS* for that)
Debugger management (use *DEBUGGER-HOOK* for that)
Command-line processing (use a library such as COMMAND-LINE-ARGUMENTS for that)
Obtaining libraries (use clbuild, LibCL, cl-librarian, asdf-install, etc. for that)

Implementation

Buildapp works like this:

It processes the command-line and creates an object that captures the command-line requirements: the output file, any eval/load/load-system/require actions, the entry function, etc.
It creates a new Lisp file (the dumpfile) with all the commands needed to implement the command-line options.
It runs either "sbcl" with sb-ext:run-program or "ccl" with ccl:run-program. In either case lisp is invoked with no init files (it doesn't read user or system rc files) and loads the dumpfile:
1. The first few commands of the dumpfile establish a specialized loading environment:
  - The debugger is changed with *invoke-debugger-hook* to simply quit with a special exit code instead of entering the normal debugger (in CCL the same behavior is accomplished using the :application and :error-handler arguments to ccl:save-application)
  - Most output (ASDF system compilation output, low-level error messages, etc) is redirected to a log stream; that stream can be directed to a file with the --logfile argument
  - Stale compiled files are automatically recompiled with an :around method on asdf:perform
2. There are some sanity checks: Is the output file writable? Does this version of sbcl support the required :save-runtime-options argument?
3. The dumpfile performs the eval/load/load-system/require actions. Each operation is evaluated in the cl-user package and with a fresh binding of sb-ext:*invoke-debugger-hook* in sbcl or ccl::*debugger-hook* in ccl to the buildapp debugger function. If the binding is modified, the new value is saved.
4. The dumpfile clears itself out of the environment:
  - Remove extra ASDF methods with remove-method
  - Reset the debugger hook to the saved value or NIL if no value was saved.
  - Delete the dumpfile package with delete-package
5. The dumpfile then creates an executable with save-lisp-and-die in sbcl or ccl:save-application in ccl. This ends the lisp subprocess.
It deletes the dumpfile.

The ASDF central registry is temporarily extended with the --asdf-path and --asdf-tree arguments at load time, and reverts back to the default central registry value after that. To avoid conflicts with this behavior, changes to the central registry should be done at startup time instead of application load time.

Feedback

If you have any questions or comments about buildapp, please email me, Zach Beane.

License

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR 'AS IS' AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.