ZCDB - Read and write cdb files from Common Lisp

ZCDB is a Common Lisp library for reading and writing D. J. Bernstein's fast and simple cdb database file format. It is available under a BSD-style license. The latest version is 1.0.4, released on March 12th, 2015.

Download shortcut: http://www.xach.com/lisp/zcdb.tgz

Contents

• Overview
• Example Use
• Limitations
• The ZCDB Dictionary
  • with-output-to-cdb
  • add-record
  • lookup
  • map-cdb
• Feedback

ZCDB is for reading and writing from CDB files. CDB files are described in more detail on D.J. Bernstein's CDB information page. The CDB file format makes it easy to quickly retrieve data associated with an arbitrary binary key.

Once written, CDB files cannot be updated with new records. In ZCDB, a CDB file is written out with WITH-OUTPUT-TO-CDB and ADD-RECORD, then read with LOOKUP.

Example Use

Here is a simple way to convert a Unix system password file to a CDB file:

(defpackage #:pwdb
  (:use #:cl #:zcdb))

(defun username (password-line)
  (let ((end (position #\: password-line)))
    (subseq password-line 0 end)))

(defun string-octets (string)
  (babel:string-to-octets string :encoding :utf-8))

(defun octets-string (octets)
  (babel:octets-to-string octets :encoding :utf-8))

(defun make-password-cdb (password-file cdb-file temp-file)
  (with-open-file (stream password-file)
    (with-output-to-cdb (cdb cdb-file temp-file)
      (loop for line = (read-line stream nil)
            while line do
            (let ((username (username line)))
              (add-record (string-octets username)
                          (string-octets line)
                          cdb))))))

(defun user-info (username cdb-file)
  (let ((entry (lookup (string-octets username) cdb-file)))
    (when entry
      (octets-string entry))))

Then the database can be created and queried:

* (make-password-cdb "/etc/passwd" "/tmp/pwdb.cdb" "/tmp/pwdb.cdb.tmp")
#P"/tmp/pwdb.cdb"

* (user-info "root" "/tmp/pwdb.cdb")
"root:*:0:0:System Administrator:/var/root:/bin/sh"

The CDB file format offers many opportunities for low-level interaction, including writing and reading records that don't fit in memory by writing or reading them incrementally. ZCDB currently only offers a simplified interface that works with keys and values that are fully loaded in memory as (unsigned-byte 8) vectors.

ZCDB does not provide any functions for serializing various data types (such as strings) to vectors. There are many other third-party libraries for that purpose.

The ZCDB Dictionary

ZCDB exports the following symbols.