Subject: Re: massive data analysis with lisp
From: rpw3@rpw3.org (Rob Warnock)
Date: Tue, 10 Oct 2006 19:17:32 -0500
Newsgroups: comp.lang.lisp
Message-ID: <N9qdnab9XdqBpbHYnZ2dnUVZ_oadnZ2d@speakeasy.net>
remixer <remixer1@gmail.com> wrote:
+---------------
| Rob Warnock wrote:
| > Note that by doing individual READ-LINEs and making sure that the
| > values were garbage by the time the next GC occurred, the above run
| > never used more than ~13 MB of additional heap at any one time,
| > despite the fact that the file was an order of magnitude larger.
| 
| Are there ways to make sure something is GC'ed ...
+---------------

"Don't hold onto it" is the only way I know. That is, overwrite any
variables or object slots that are holding a reference to the thing
you want to drop.

NOTA BENE: When working in the REPL, don't forget these:

    [CLHS]
    25.1.1 Top level loop
    ...

    *    +    /    -  
    **   ++   //      
    ***  +++  ///     
    Figure 25-1. Variables maintained by the Read-Eval-Print Loop

Fortunately, clearing them *all* out requires only typing "0<return>"
[or any other expression with minimal results] three or four times, e.g.:

    > (setf *print-length* 20)

    20
    > (make-array 10000000)

    #(0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ...)
    > (gc :full t)
    ; [GC threshold exceeded with 40,040,408 bytes in use.  Commencing GC.]
    ; [GC completed with 40,034,984 bytes retained and 5,424 bytes freed.]
    ; [GC will next occur when at least 52,034,984 bytes are in use.]

    NIL
    > (gc :full t)
    ; [GC threshold exceeded with 40,039,016 bytes in use.  Commencing GC.]
    ; [GC completed with 40,032,848 bytes retained and 6,168 bytes freed.]
    ; [GC will next occur when at least 52,032,848 bytes are in use.]

    NIL
    > (gc :full t)
    ; [GC threshold exceeded with 40,036,880 bytes in use.  Commencing GC.]
    ; [GC completed with 40,032,848 bytes retained and 4,032 bytes freed.]
    ; [GC will next occur when at least 52,032,848 bytes are in use.]

    NIL
    > (gc :full t)
    ; [GC threshold exceeded with 40,036,880 bytes in use.  Commencing GC.]
    ; [GC completed with 32,848 bytes retained and 40,004,032 bytes freed.]
    ; [GC will next occur when at least 12,032,848 bytes are in use.]

    NIL
    > 

And now it's finally gone!  [Hint: What was in "*" after each input?]

+---------------
| I tell Allegrocache about limits on cache, but that doesnt help --
| it is sluggish and heap blows after reading in less than 1% of the data.
+---------------

Sounds like an Allegrocache issue then, but since I don't use it
I can't be of any further help. (Sorry.)


-Rob

-----
Rob Warnock			<rpw3@rpw3.org>
627 26th Avenue			<URL:http://rpw3.org/>
San Mateo, CA 94403		(650)572-2607