Subject: Re: A Proper Lexer
From: rpw3@rpw3.org (Rob Warnock)
Date: Sat, 13 Sep 2008 21:51:16 -0500
Newsgroups: comp.lang.lisp
Message-ID: <neudnbp4r7O54VHVnZ2dnUVZ_tLinZ2d@speakeasy.net>
Paul Tarvydas  <tarvydas@visualframeworksinc.com> wrote:
+---------------
| akopa wrote:
| > I was a little surprised to find no equivalent of Lex available for
| > Common Lisp.  There is http://www.geocities.com/mparker762/clawk but
| > it is string oriented, not stream oriented.
| 
| There is little incentive to use Lex in lisp, because the lisp reader *is*
| a lexer (tokenizer), hence, lisp is the language least likely to have a
| Lex-like library.  I've built many small languages and just used what lisp
| gives me, e.g. if you can design it so that the little language has a space
| between each token, you don't have to write a lexer.  I suppose you could
| get fancier by employing read macros, et al.
+---------------

In that regard, the OP might be interested in META-style parsers,
originally developed by D. V. Schorre circa 1962:

    http://en.wikipedia.org/wiki/META_II

     Schorre, D.V. "META II: A Syntax-Oriented Compiler Writing Language".
     Proc. 19'th Nat'l. Conf. of the ACM (Aug. 1964),D1.3-1-D1.3-11.

[I used Michael Green's implementation of Schorre's META II on the
DEC PDP-10 (part of Green's ALGOL-W compiler) in 1970 to write a tiny
BLISS compiler in a weekend.]

In 1991, Henry Baker did a very loose re-interpretation (or compilation)
of the META style into CL functions, macros, & readmacros to make META-
based parsers *very* concisely embeddable within a CL propram:

    http://home.pipeline.com/~hbaker1/Prag-Parse.html
    "Pragmatic Parsing in Common Lisp"

Jochen Schmidt typed in and reorganised Baker's code, and made it
available here:

    http://cclan.cvs.sourceforge.net/cclan/cclan/meta/

Another version by Fare Rideau is linked from here:

    http://www.cliki.net/meta

And, finally, yet a different version of Baker's code, assembled
by Henrik Motakef, is also available as a FreeBSD "port" under
the name "cl-meta".


-Rob

-----
Rob Warnock			<rpw3@rpw3.org>
627 26th Avenue			<URL:http://rpw3.org/>
San Mateo, CA 94403		(650)572-2607