Re: How to read lots of data from disk quickly?

Subject: Re: How to read lots of data from disk quickly?
From: Erik Naggum <erik@naggum.no>
Date: 1996/04/21
Newsgroups: comp.lang.lisp
Message-ID: <3039042460679868@arcana.naggum.no>

[Bruce L. Lambert, Ph.D.]

|   In the course of my information retrieval experiments, I often need to
|   read large data files in from disk.  I've got a couple of functions
|   that do this successfully, but they seem awful damn slow to me.  They
|   also seem really convoluted for what should be a more straightforward
|   task.

(this is not an answer to your question, but it could become one.)

the ANSI CL function `read-sequence' (`write-sequence') will replace
successive objects in a sequence (file) with objects in the file
(sequence).  the efficiency gains hinge on equality of the element-type of
the stream and the sequence.  `open' is not able to portable open a file
with anything but element-types that are subtypes of character, finite
subtypes of integer, signed-byte or unsigned-byte, barring an unusual
interpretation of :default.

would it be a non-conforming extension to pass floating-point types as the
element-type to `open' so one could do the following?

    (let* ((type 'double-float)
	   (vector (make-array <dimensions>
                              :element-type type
			      :initial-element (coerce 0 type))))
      (with-open-file (stream <filespec>
			      :element-type type)
	(read-sequence vector stream))
      vector)

IMHO, if this is a non-conforming extension, we have a problem.  if it is a
conforming extension, vendors should be encouraged to supply ways to allow
files to handle user-specified extended element-types.  it would perhaps
make sense to achieve consensus on the interface to such extensions.

(should this be posted to comp.std.lisp?  or is the group defunct?)

#<Erik>
-- 
errare umanum hest