ZS3 - Amazon S3 and CloudFront from Common Lisp
ZS3 is a Common Lisp library for working with Amazon's Simple Storage Service (S3) and CloudFront content delivery service. It is available under a BSD-style license; see LICENSE for details. Development of ZS3 is hosted on GitHub. The latest version is 1.3.3, released on September 29th, 2019.
Download shortcut: http://www.xach.com/lisp/zs3.tgz
Contents
- Installation
- Overview
- Example Use
- Limitations
- The ZS3 Dictionary
- References
- Acknowledgements
- Feedback
Installation
ZS3 depends on the following libraries:
- Closure XML
- Drakma 1.0.0 or newer
- Ironclad
- cl-base64
- Puri
The easiest way to install ZS3 and all its required libraries is with Quicklisp. After Quicklisp is installed, the following will fetch and load ZS3:
(ql:quickload "zs3")
For more information about incorporating ASDF-using libraries like
ZS3 into your own projects,
see this short
tutorial.
Overview
ZS3 provides an interface to two separate, but related, Amazon services: S3 and CloudFront
Using Amazon S3 involves working with two kinds of resources: buckets and objects.
Buckets are containers, and are used to organize and manage objects. Buckets are identified by their name, which must be unique across all of S3. A user may have up to 100 buckets in S3.
Objects are stored within buckets. Objects consist of arbitrary binary data, from 1 byte to 5 gigabytes. They are identified by a key, which must be unique within a bucket. Objects can also have associated S3-specific metadata and HTTP metadata.
For full documentation of the Amazon S3 system, see the Amazon S3 Documentation. ZS3 uses the REST interface for all its operations.
Using Amazon CloudFront involves working with distributions. Distributions are objects that associate an S3 bucket with primary cloudfront.net hostname and zero or more arbitrary CNAMEs. S3 objects requested through a CloudFront distribution are distributed to and cached in multiple locations throughout the world, reducing latency and improving throughput, compared to direct S3 requests.
For full documentation of the Amazon CloudFront system, see the Amazon CloudFront Documentation.
For help with using ZS3, please see
the zs3-devel
mailing list.
Example Use
* (asdf:oos 'asdf:load-op '#:zs3) => lots of stuff * (defpackage #:demo (:use #:cl #:zs3)) => #<PACKAGE "DEMO"> * (in-package #:demo) => #<PACKAGE "DEMO"> * (setf *credentials* (file-credentials "~/.aws")) => #<FILE-CREDENTIALS {100482AF91}> * (bucket-exists-p "zs3-demo") => NIL * (create-bucket "zs3-demo") => #<RESPONSE 200 "OK" {10040D3281}> * (http-code *) => 200 * (put-vector (octet-vector 8 6 7 5 3 0 9 ) "zs3-demo" "jenny") => #<RESPONSE 200 "OK" {10033EC2E1}> * (create-bucket "zs3 demo") Error: InvalidBucketName: The specified bucket is not valid. For more information, see: http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html [Condition of type INVALID-BUCKET-NAME] * (copy-object :from-bucket "zs3-demo" :from-key "jenny" :to-key "j2") => #<RESPONSE 200 "OK" {10040E3EA1}> * (get-vector "zs3-demo" "j2") => #(8 6 7 5 3 0 9), ((:X-AMZ-ID-2 . "Huwo...") (:X-AMZ-REQUEST-ID . "304...") (:DATE . "Sat, 27 Sep 2008 15:01:03 GMT") (:LAST-MODIFIED . "Sat, 27 Sep 2008 14:57:31 GMT") (:ETAG . "\"f9e71fe2c41a10c0a78218e98a025520\"") (:CONTENT-TYPE . "binary/octet-stream") (:CONTENT-LENGTH . "7") (:CONNECTION . "close") (:SERVER . "AmazonS3")) * (put-string "Nämen" "zs3-demo" "bork") => #<RESPONSE 200 "OK" {10047A3791}> * (values (get-vector "zs3-demo" "bork")) => #(78 195 164 109 101 110) * (values (get-file "zs3-demo" "bork" "bork.txt")) => #P"bork.txt" * (setf *distribution* (create-distribution "zs3-demo" :cnames "cdn.wigflip.com") => #<DISTRIBUTION X2S94L4KLZK5G0 for "zs3-demo.s3.amazonaws.com" [InProgress]> * (progn (sleep 180) (refresh *distribution*)) => #<DISTRIBUTION X2S94L4KLZK5G0 for "zs3-demo.s3.amazonaws.com" [Deployed]> * (domain-name *distribution*) => "x214g1hzpjm1zp.cloudfront.net" * (cnames *distribution*) => ("cdn.wigflip.com") * (put-string "Hello, world" "zs3-demo" "cloudfront" :public t) #<RESPONSE 200 "OK" {10042689F1}> * (drakma:http-request "http://x214g1hzpjm1zp.cloudfront.net/cloudfront") "Hello, world" 200 ((:X-AMZ-ID-2 . "NMc3IY3NzHGGEvV/KlzPgZMyDfPVT+ITtHo47Alqg00MboTxSX2f5XJzVTErfuHr") (:X-AMZ-REQUEST-ID . "52B050DC18638A00") (:DATE . "Thu, 05 Mar 2009 16:24:25 GMT") (:LAST-MODIFIED . "Thu, 05 Mar 2009 16:24:10 GMT") (:ETAG . "\"bc6e6f16b8a077ef5fbc8d59d0b931b9\"") (:CONTENT-TYPE . "text/plain") (:CONTENT-LENGTH . "12") (:SERVER . "AmazonS3") (:X-CACHE . "Miss from cloudfront") (:VIA . "1.0 ad78cb56da368c171e069e4444b2cbf6.cloudfront.net:11180") (:CONNECTION . "close")) #<PURI:URI http://x214g1hzpjm1zp.cloudfront.net/cloudfront> #<FLEXI-STREAMS:FLEXI-IO-STREAM {1002CE0781}> T "OK" * (drakma:http-request "http://x214g1hzpjm1zp.cloudfront.net/cloudfront") "Hello, world" 200 ((:X-AMZ-ID-2 . "NMc3IY3NzHGGEvV/KlzPgZMyDfPVT+ITtHo47Alqg00MboTxSX2f5XJzVTErfuHr") (:X-AMZ-REQUEST-ID . "52B050DC18638A00") (:DATE . "Thu, 05 Mar 2009 16:24:25 GMT") (:LAST-MODIFIED . "Thu, 05 Mar 2009 16:24:10 GMT") (:ETAG . "\"bc6e6f16b8a077ef5fbc8d59d0b931b9\"") (:CONTENT-TYPE . "text/plain") (:CONTENT-LENGTH . "12") (:SERVER . "AmazonS3") (:AGE . "311") (:X-CACHE . "Hit from cloudfront") (:VIA . "1.0 0d78cb56da368c171e069e4444b2cbf6.cloudfront.net:11180") (:CONNECTION . "close")) #<PURI:URI http://x214g1hzpjm1zp.cloudfront.net/cloudfront> #<FLEXI-STREAMS:FLEXI-IO-STREAM {100360A781}> T "OK"
Limitations
ZS3 supports many of the features of Amazon's S3 REST interface. Some features are unsupported or incompletely supported:
- No direct support for Amazon DevPay
- No support for checking the 100-Continue response to avoid unnecessary large requests; this will hopefully be fixed with a future Drakma release
- If a character in a key is encoded with multiple bytes in UTF-8, a bad interaction between PURI and Amazon's web servers will trigger a validation error.
The ZS3 Dictionary
The following sections document the symbols that are exported from
ZS3.
Credentials
*credentials*
*CREDENTIALS*
is the source of the Amazon Access Key and Secret Key for authenticated requests. Any object that has methods for the ACCESS-KEY and SECRET-KEY generic functions may be used.If
*CREDENTIALS*
is a cons, it is treated as a list, and the first element of the list is taken as the access key and the second element of the list is taken as the secret key.The default value of
*CREDENTIALS*
is NIL, which will signal an error. You must set*CREDENTIALS*
to something that follows the credentials generic function protocol to use ZS3.All ZS3 functions that involve authenticated requests take an optional
:CREDENTIALS
keyword parameter. This parameter is bound to*CREDENTIALS*
for the duration of the function call.The following illustrates how to implement a credentials object that gets the access and secret key from external environment variables.
(defclass environment-credentials () ()) (defmethod access-key ((credentials environment-credentials)) (declare (ignore credentials)) (getenv "AWS_ACCESS_KEY")) (defmethod secret-key ((credentials environment-credentials)) (declare (ignore credentials)) (getenv "AWS_SECRET_KEY")) (setf *credentials* (make-instance 'environment-credentials))
access-key
credentials
=> access-key-string
Returns the access key for credentials.
security-token
credentials
=> security-token-string
Returns the security token string for credentials, or NIL if there is no associated security token.
secret-key
credentials
=> secret-key-string
Returns the secret key for credentials.
file-credentials
pathname
=> credentials
Loads credentials on demand from pathname. The file named by pathname should be a text file with the access key on the first line and the secret key on the second line.
It can be used like so:
(setf *credentials* (file-credentials "/etc/s3.conf"))
Responses
Some operations return a response as an additional value. All response objects can be interrogated to obtain the HTTP code, headers and phrase.
The outcome of some requests — a very small proportion — will be an error internal to the AWS server. In these circumstances an exponential backoff policy operates; if this encounters too many failures then ZS3 signals an internal-error which can be interrogated to obtain the response object, and through that the HTTP response code and headers:
* e #<ZS3:INTERNAL-ERROR @ #x1000296bc92> * (setf r (zs3:request-error-response e)) #<ZS3::AMAZON-ERROR "InternalError"> * (zs3:http-code r) 500 * (zs3:http-headers r) ((:X-AMZ-REQUEST-ID . "3E20E3BAC24AB9AA") (:X-AMZ-ID-2 . "80sxu4PDKtx1BWLOcSrUVWD90mMMVaMx6y9c+sz5VBGa2eAES2YlNaefn5kqRsfvrbaF+7QGNXA=") (:CONTENT-TYPE . "application/xml") (:TRANSFER-ENCODING . "chunked") (:DATE . "Fri, 30 Sep 2016 10:10:11 GMT") (:CONNECTION . "close") (:SERVER . "AmazonS3")) * (zs3:http-phrase r) "Internal Server Error" *
*backoff*
Used as the default value of
:backoff
when submitting a request. The value should be a cons of two numbers: how many times to try before giving up, and how long to wait (in ms) before trying for the second time. Each subsequent attempt will double that time.The default value is
(3 . 100)
.If a requst fails more times than permitted by
*backoff*
, an error will be signalled. It is the application's responsibility to handle this error.
request-error-response
request-error
=> response
Returns the
response
object associated with a request-error.
http-headers
response
=> headers
Returns the HTTP headers associated with a
response
object.
http-phrase
response
=> phrase
Returns the HTTP phrase associated with a
response
object.
Operations on Buckets
With ZS3, you can put, get, copy, and delete buckets. You can also get information about the bucket.
all-buckets
&key
credentials backoff
=> bucket-vector
Returns a vector of all bucket objects. Bucket object attributes are accessible via NAME and CREATION-DATE.
creation-date
bucket-object
=> creation-universal-time
Returns the creation date of bucket-object, which must be a bucket object, as a universal time.
name
object
=> name-string
Returns the string name of object, which must be a key object or bucket object.
all-keys
bucket
&key
prefix
credentials
backoff
=> key-vector
Returns a vector of all key objects in bucket with names that start with the string prefix. If no prefix is specified, returns all keys. Keys in the vector are in alphabetical order by name. Key object attributes are accessible via NAME, SIZE, ETAG, LAST-MODIFIED, OWNER, and STORAGE-CLASS.
This function is built on QUERY-BUCKET and may involve multiple requests if a bucket has more than 1000 keys.
bucket-exists-p
bucket
&key
credentials
backoff
=> boolean
Returns true if bucket exists.
create-bucket
name
&key
access-policy
public
location
credentials
backoff
=> response
Creates a bucket named name.
If provided, access-policy should be one of the following:
:PRIVATE
- bucket owner is granted:FULL-CONTROL
; this is the default behavior if no access policy is provided:PUBLIC-READ
- all users, regardless of authentication, can query the bucket's contents:PUBLIC-READ-WRITE
- all users, regardless of authentication, can query the bucket's contents and create new objects in the bucket:AUTHENTICATED-READ
- authenticated Amazon AWS users can query the bucketFor more information about access policies, see Canned ACL in the Amazon S3 developer documentation.
If public is true, it has the same effect as providing an access-policy of
:PUBLIC-READ
. An error is signaled if both public and access-policy are provided.If location is specified, the bucket will be created in a region matching the given location constraint. If no location is specified, the bucket is created in the US. Valid locations change over time, but currently include "EU", "us-west-1", "us-west-2", "eu-west-1", "eu-central-1", "ap-southeast-1", "ap-southeast-2", "ap-northeast-1", and "sa-east-1". See Regions and Endpoints in the Amazon S3 developer documentation for the current information about location constraints.
delete-bucket
bucket &key
credentials
backoff
=> response
Deletes bucket. Signals a BUCKET-NOT-EMPTY error if the bucket is not empty, or a NO-SUCH-BUCKET error if there is no bucket with the given name.
bucket-location
bucket &key
credentials
backoff
=> location
Returns the location specified when creating a bucket, or NIL if no location was specified.
bucket-lifecycle
bucket
=> rules-list
Returns a list of lifecycle rules for bucket. Signals a NO-SUCH-LIFECYCLE-CONFIGURATION error if the bucket has no lifecycle rules configured.
Bucket lifecycle rules are used to control the automatic deletion of objects in a bucket. For more information about bucket lifecycle configuration, see Object Expiration in the Amazon S3 developer documentation.
(setf bucket-lifecycle)
rules bucket
=> rules, response
Sets the lifecycle configuration of bucket to the designator for a list of bucket lifecycle rules rules.
To create a bucket lifecycle rule, use LIFECYCLE-RULE. For example, to automatically delete objects with keys matching a "logs/" prefix after 30 days:
(setf (bucket-lifecycle "my-bucket") (lifecycle-rule :prefix "logs/" :days 30))To delete a bucket's lifecycle configuration, use an empty list of rules, e.g.
(setf (bucket-lifecycle "my-bucket") nil)
lifecycle-rule
&key
action
prefix
days
date
=> rule
Creates a rule object suitable for passing to (SETF BUCKET-LIFECYCLE).
action should be either :expire (the default) or :transition. For :expire, matching objects are deleted. For :transition, matching objects are transitioned to the GLACIER storage class. For more information about S3-to-Glacier object transition, see Object Archival (Transition Objects to the Glacier Storage Class) in the Amazon S3 Developer's Guide.
prefix is a string; all objects in a bucket with keys matching the prefix will be affected by the rule.
days is the number of days after which an object will be affected.
date is the date after which objects will be affected.
Only one of days or date may be provided.
restore-object
bucket
key
&body
days
credentials
backoff
=> response
Initiates a restoration operation on the object identified by bucket and key. A restoration operation can take several hours to complete. The restored object is temporarily stored with the reduced redundancy storage class. The status of the operation may monitored via OBJECT-RESTORATION-STATUS.
days is the number of days for which the restored object should be avilable for normal retrieval before transitioning back to archival storage.
Object restoration operation is only applicable to objects that have been transitioned to Glacier storage by the containing bucket's lifecycle configuration.
For more information, see POST Object restore in the S3 documentation.
object-restoration-status
bucket
key
&key
credentials
backoff
=> status-string
Returns a string describing the status of restoring the object identified by bucket and key. If no restoration is in progress, or the operation is not applicable, returns NIL.
Querying Buckets
S3 has a flexible interface for querying a bucket for information about its contents. ZS3 supports this interface via QUERY-BUCKET, CONTINUE-BUCKET-QUERY, and related functions.
query-bucket
bucket
&key
prefix
marker
max-keys
delimiter
credentials
backoff
=> response
Query bucket for key information. Returns a response object that has the result of the query. Response attributes are accessible via BUCKET-NAME, PREFIX, MARKER, DELIMITER, TRUNCATEDP, KEYS, and COMMON-PREFIXES.
Amazon might return fewer key objects than actually match the query parameters, based on max-keys or the result limit of 1000 key objects. In that case, TRUNCATEDP for response is true, and CONTINUE-BUCKET-QUERY can be used with response be used to get successive responses for the query parameters.
When prefix is supplied, only key objects with names that start with prefix will be returned in response.
When marker is supplied, only key objects with names occurring lexically after marker will be returned in response.
When max-keys is supplied, it places an inclusive upper limit on the number of key objects returned in response. Note that Amazon currently limits responses to at most 1000 key objects even if max-keys is greater than 1000.
When delimiter is supplied, key objects that have the delimiter string after prefix in their names are not returned in the KEYS attribute of the response, but are instead accumulated into the COMMON-PREFIXES attribute of the response. For example:
* (all-keys "zs3-demo") => #(#<KEY "a" 4> #<KEY "b/1" 4> #<KEY "b/2" 4> #<KEY "b/3" 4> #<KEY "c/10" 4> #<KEY "c/20" 4> #<KEY "c/30" 4>) * (setf *response* (query-bucket "zs3-demo" :delimiter "/")) => #<BUCKET-LISTING "zs3-demo"> * (values (keys *response*) (common-prefixes *response*)) => #(#<KEY "a" 4>), #("b/" "c/") * (setf *response* (query-bucket "zs3-demo" :delimiter "/" :prefix "b/")) => #<BUCKET-LISTING "zs3-demo"> * (values (keys *response*) (common-prefixes *response*)) => #(#<KEY "b/1" 4> #<KEY "b/2" 4> #<KEY "b/3" 4>), #()For more information about bucket queries, see GET Bucket in the Amazon S3 developer documentation.
continue-bucket-query
response
=> response
If response is a truncated response from a previous call to QUERY-BUCKET, continue-bucket-query returns the result of resuming the query at the truncation point. When there are no more results, continue-bucket-query returns NIL.
bucket-name
response
=> name
Returns the name of the bucket used in the call to QUERY-BUCKET that produced response.
keys
response
=> keys-vector
Returns the vector of key objects in response. Key object attributes are accessible via NAME, SIZE, ETAG, LAST-MODIFIED, and OWNER.
common-prefixes
response
=> prefix-vector
Returns a vector of common prefix strings, based on the delimiter argument of the QUERY-BUCKET call that produced response.
prefix
response
=> prefix-string
Returns the prefix given to the QUERY-BUCKET call that produced response. If present, all keys in response have prefix-string as a prefix.
marker
response
=> marker
Returns the marker given to the QUERY-BUCKET call that produced response. If present, it lexically precedes all key names in the response.
delimiter
response
=> delimiter
Returns the delimiter used in the QUERY-BUCKET call that produced response.
truncatedp
response
=> boolean
Returns true if response is truncated; that is, if there is more data to retrieve for a given QUERY-BUCKET query. CONTINUE-BUCKET-QUERY may be used to fetch more data.
last-modified
key-object
=> universal-time
Returns a universal time representing the last modified time of key-object.
owner
key-object
=> owner
Returns the owner of key-object, or NIL if no owner information is available.
Operations on Objects
Objects are the stored binary data in S3. Every object is uniquely identified by a bucket/key pair. ZS3 has several functions for storing and fetching objects, and querying object attributes.
get-object
bucket
key
&key
output start end
when-modified-since unless-modified-since
when-etag-matches unless-etag-matches
if-exists string-external-format credentials backoff => object
Fetch the object referenced by bucket and key. The secondary value of all successful requests is an alist of Drakma-style response HTTP headers.
If output is
:VECTOR
(the default), the object's octets are returned in a vector.If output is
:STRING
, the object's octets are converted to a string using the encoding specified by string-external-format, which defaults to:UTF-8
. See External formats in the FLEXI-STREAMS documentation for supported values for the string external format. Note that, even when output is:STRING
, the start and end arguments operate on the object's underlying octets, not the string representation in a particular encoding. It's possible to produce a subsequence of the object's octets that are not valid in the desired encoding.If output is a string or pathname, the object's octets are saved to a file identified by the string or pathname. The if-exists argument is passed to
WITH-OPEN-FILE
to control the behavior when the output file already exists. It defaults to:SUPERSEDE
.If output is
:STREAM
, a stream is returned from which the object's contents may be read.start marks the first index fetched from the object's data. end specifies the index after the last octet fetched. If start is NIL, it defaults to 0. If end is nil, it defaults to the total length of the object. If both start and end are provided, start must be less than or equal to end.
when-modified-since and unless-modified-since are optional. If when-modified-since is provided, the result will be the normal object value if the object has been modified since the provided universal time, NIL otherwise. The logic is reversed for unless-modified-since.
when-etag-matches and unless-etag-matches are optional. If when-etag-matches is provided, the result will be the normal object value if the object's etag matches the provided string, NIL otherwise. The logic is reversed for unless-etag-matches.
get-vector
bucket key
&key
start end
when-modified-since unless-modified-since
when-etag-matches unless-etag-matches
credentials
backoff
=> vector
get-vector is a convenience interface to GET-OBJECT. It is equivalent to calling:
(get-object bucket key :output :vector ...)
get-string
bucket key
&key
external-format
start end
when-modified-since
unless-modified-since
when-etag-matches
unless-etag-matches
credentials
backoff
=> string
get-string is a convenience interface to GET-OBJECT. It is equivalent to calling:(get-object bucket key :output :string :string-external-format external-format ...)
get-file
bucket key file
&key
start end
when-modified-since
unless-modified-since
when-etag-matches
unless-etag-matches
credentials
backoff
=> pathname
get-file is a convenience interface to GET-OBJECT. It is equivalent to calling:
(get-object bucket key :output file ...)
put-object
object bucket key
&key
access-policy
public
metadata
string-external-format
cache-control
content-encoding
content-disposition
content-type
expires
storage-class
tagging
credentials
backoff
=> response
Stores the octets of object in the location identified by bucket and key.
If object is an octet vector, it is stored directly.
If object is a string, it is converted to an octet vector using string-external-format, which defaults to
:UTF-8
, then stored. See External formats in the FLEXI-STREAMS documentation for supported values for the string external format.If object is a pathname, its contents are loaded in memory as an octet vector and stored.
If provided, access-policy should be one of the following:
:PRIVATE
- object owner is granted:FULL-CONTROL
; this is the default behavior if no access policy is provided:PUBLIC-READ
- all users, regardless of authentication, can read the object:AUTHENTICATED-READ
- authenticated Amazon AWS users can read the objectFor more information about access policies, see Canned ACL in the Amazon S3 developer documentation.
If public is true, it has the same effect as providing an access-policy of
:PUBLIC-READ
. An error is signaled if both public and access-policy are provided.If provided, metadata should be an alist of Amazon metadata to set on the object. When the object is fetched again, the metadata will be returned in HTTP headers prefixed with "x-amz-meta-".
The cache-control, content-encoding, content-disposition, content-type, and expires values are all used to set HTTP properties of the object that are returned with subsequent GET or HEAD requests. If content-type is not set, it defaults to "binary/octet-stream". The others default to NIL. If expires is provided, it should be a universal time.
If provided, storage-class should refer to one of the standard storage classes available for S3; currently the accepted values are the strings "STANDARD" and "REDUCED_REDUNDANCY". Using other values may trigger an API error from S3. For more information about reduced redundancy storage, see reduced Redundancy Storage in the Developer Guide.
If provided, tagging specifies the set of tags to be associated with the object. The set is given as an alist. For more information, see Object Tagging in the Developer Guide.
put-vector
vector
bucket
key &key
start end
access-policy
public metadata
content-disposition
content-encoding
content-type
expires
storage-class
tagging
credentials
backoff
=> response
put-vector is a convenience interface to PUT-OBJECT. It is similar to calling:
(put-object vector bucket key ...)
If one of start or end is provided, they are used as bounding index designators on the string, and only a subsequence is used.
put-string
string bucket key
&key
start end
external-format
access-policy
public metadata
content-disposition
content-encoding
content-type
expires
storage-class
tagging
credentials
backoff
=> response
put-string is a convenience interface to PUT-OBJECT. It is similar to calling:
(put-object string bucket key :string-external-format external-format ...)
If one of start or end is supplied, they are used as bounding index designators on the string, and only a substring is used.
put-file
file bucket key
&key
start end
access-policy
public metadata
content-disposition content-encoding content-type
expires
storage-class
tagging
credentials
backoff
=> response
put-file is a convenience interface to PUT-OBJECT. It is almost equivalent to calling:
(put-object (pathname file) bucket key ...)If key is T, the
FILE-NAMESTRING
of the file is used as the key instead of key.If one of start or end is supplied, only a subset of the file is used. If start is not NIL, start octets starting from the beginning of the file are skipped. If end is not NIL, octets in the file at and after end are ignored. An error of type CL:END-OF-FILE is signaled if end is provided and the file size is less than end.
put-stream
file bucket key
&key
start end
access-policy
public metadata
content-disposition content-encoding content-type
expires
storage-class
tagging
credentials
backoff
=> response
put-stream is similar to to PUT-OBJECT. It has the same effect as collecting octets from stream into a vector and using:
(put-object vector bucket key ...)If start is not NIL, start octets starting from the current position in the stream are skipped before collecting.
If end is NIL, octets are collected until the end of the stream is reached.
If end is not NIL, collecting octets stops just before reaching end in the stream. An error of type CL:END-OF-FILE is signaled if the stream ends prematurely.
copy-object
&key
from-bucket
from-key
to-bucket
to-key access-policy public
when-etag-matches unless-etag-matches
when-modified-since unless-modified-since
metadata public precondition-errors storage-class tagging credentials backoff => response
Copies the object identified by from-bucket and from-key to a new location identified by to-bucket and to-key. If to-bucket is NIL, from-bucket is used as the target. If to-key is nil, from-key is used as the target. An error is signaled if both to-bucket and to-key are NIL.
access-policy and public have the same effect on the target object as in PUT-OBJECT.
The precondition arguments when-etag-matches, unless-etag-matches, when-modified-since, and unless-modified-since work the same way they do in GET-OBJECT, but with one difference: if precondition-errors is true, an
PRECONDITION-FAILED
error is signaled when a precondition does not hold, instead of returning NIL.If metadata is explicitly provided, it follows the same behavior as with PUT-OBJECT. Passing NIL means that the new object has no metadata. Otherwise, the metadata is copied from the original object.
If tagging is explicitly provided, it follows the same behavior as with PUT-OBJECT. Passing NIL means that the new object has no tags. Otherwise, tagging is copied from the original object.
If storage-class is provided, it should refer to one of the standard storage classes available for S3; currently the accepted values are the strings "STANDARD" and "REDUCED_REDUNDANCY". Using other values may trigger an API error from S3. For more information about reduced redundancy storage, see Reduced Redundancy Storage in the Developer Guide.
delete-object
bucket
key
&key
credentials
backoff
=> response
Deletes the object identified by bucket and key.
If bucket is a valid bucket for which you have delete access granted, S3 will always return a success response, even if key does not reference an existing object.
delete-objects
bucket
keys
&key
credentials
backoff
=> deleted-count, errors
Deletes keys, which should be a sequence of keys, from bucket. The primary value is the number of objects deleted. The secondary value is a list of error plists; if there are no errors deleting any of the keys, the secondary value is NIL.
delete-all-objects
bucket
&key
credentials
backoff
=> count
Deletes all objects in bucket and returns the count of objects deleted.
object-metadata
bucket
key
&key
credentials
backoff
=> metadata-alist
Returns the metadata for the object identified by bucket and key, or NIL if there is no metadata. For example:
* (put-string "Hadjaha!" "zs3-demo" "hadjaha.txt" :metadata (parameters-alist :language "Swedish")) => #<RESPONSE 200 "OK" {1003BD2841}> * (object-metadata "zs3-demo" "hadjaha.txt") => ((:LANGUAGE . "Swedish"))
set-storage-class
bucket
key
storage-class
&key
credentials
backoff
=> response
Sets the storage class of the object identified by bucket and key to storage-class. This is a convenience function that uses COPY-OBJECT to make storage class changes.
The storage class of an object can be determined by querying the bucket with ALL-KEYS or QUERY-BUCKET and using STORAGE-CLASS on one of the resulting key objects.
Access Control
Each S3 resource has an associated access control list that is created automatically when the resource is created. The access control list specifies the resource owner and a list of permission grants.
Grants consist of a permission and a grantee. The permission must
be one of :READ
, :WRITE
,
:READ-ACL
, :WRITE-ACL
,
or :FULL-CONTROL
. The grantee should be a
person object, an acl-group object, or an acl-email object.
ZS3 has several functions that assist in reading, modifying, and storing access control lists.
get-acl
&key
bucket
key
credentials
backoff
=> owner, grants
Returns the owner and grant list for a resource as multiple values.
put-acl
owner grants
&key
bucket
key
credentials
backoff
=> response
Sets the owner and grant list of a resource.
grant
permission
&key
to
=> grant
Returns a grant object that represents a permission (one of
:READ
,:WRITE
,:READ-ACL
,:WRITE-ACL
, or:FULL-CONTROL
) for the grantee to. For example:* (grant :full-control :to (acl-email "bob@example.com")) => #<GRANT :FULL-CONTROL to "bob@example.com"> * (grant :read :to *all-users*) => #<GRANT :READ to "AllUsers">It can be used to create or modify a grant list for use with PUT-ACL.
acl-eqv
a b
=> boolean
Returns true if a and b are equivalent ACL-related objects (person, group, email, or grant).
*all-users*
This acl-group includes all users, including unauthenticated clients.
*aws-users*
This acl-group object includes only users that have an Amazon Web Services account.
*log-delivery*
This acl-group object includes the S3 system user that creates logfile objects. See also ENABLE-LOGGING-TO.
acl-email
email-address
=> acl-email
Returns an acl-email object, which can be used as a grantee for GRANT.
acl-person
id
&optional
display-name
=> acl-person
Returns an acl-person object for use as a resource owner (for PUT-ACL) or as a grantee (for GRANT). id must be a string representing the person's Amazon AWS canonical ID; for information about getting the canonical ID, see the Managing Access with ACLS in the Amazon S3 developer documentation. If display-name is provided, it is used only for printing the object in Lisp; it is ignored when passed to S3.
me
&key
credentials
backoff
=> acl-person
Returns the acl-person object associated with the current credentials.
This data requires a S3 request, but the result is always the same per credentials and is cached.
make-public
&key
bucket
key
credentials
backoff
=> response
Makes a resource publicly accessible, i.e. readable by the *ALL-USERS* group.
make-private
&key
bucket
key
credentials
backoff
=> response
Removes public access to a resource, i.e. removes all access grants for the *ALL-USERS* group.
Access Logging
S3 offers support for logging information about client requests. Logfile objects are delivered by a system user in the *LOG-DELIVERY* group to a bucket of your choosing. For more information about S3 access logging and the logfile format, see the Server Access Logging in the Amazon S3 developer documentation.
enable-logging-to
bucket
&key
credentials
backoff
=> response
Adds the necessary permission grants to bucket to allow S3 to write logfile objects into it.
disable-logging-to
bucket
&key
credentials
backoff
=> response
Changes the access control list of bucket to remove all grants for the *LOG-DELIVERY* group.
enable-logging
bucket
target-bucket
target-prefix
&key
target-grants
credentials
backoff
=> response
Enables logging of all requests involving bucket. Logfile objects are created in target-bucket and each logfile's key starts with target-prefix.
When a new logfile is created, its list of access control grants is extended with target-grants, if any.
If target-bucket does not have the necessary grants to allow logging, the grants are implicitly added by calling ENABLE-LOGGING-TO.
logging-setup
bucket
&key
credentials
backoff
=> target-bucket,
target-prefix,
target-grants
If logging is enabled for bucket, returns the target bucket, target prefix, and target grants as multiple values.
Object Tagging
In S3, a set of tags can be associated with each key and bucket. Tagging offers a way to categorize objects that is orthogonal to key prefixes. They resemble object metadata but, unlike metadata, tagging be used in access control, lifecycle rules, and metrics. For more information, please refer to the Object Tagging section on the S3 Developer Guide.
get-tagging
&key
bucket
key
credentials
backoff
=> tag-set
Returns the object's current tag set as an alist.
put-tagging
tag-set
&key
bucket
key
credentials
backoff
=> response
Sets the object's tagging resource to the given set of tags. The tags are given as an alist.
delete-tagging
&key
bucket
key
credentials
backoff
=> response
Deletes the tagging resource associated with the object.
Miscellaneous Operations
*use-keep-alive*
When true, HTTP keep-alives are used to reuse a single network connection for multiple requests.
make-post-policy
&key
expires
conditions
credentials
=> policy, signature
Returns an encoded HTML POST form policy and its signature as multiple values. The policy can be used to conditionally allow any user to put objects into S3.
expires must be a universal time after which the policy is no longer accepted.
conditions must be a list of conditions that the posted form fields must satisfy. Each condition is a list of a condition keyword, a form field name, and the form field value. For example, the following are all valid conditions:
(:starts-with "key" "uploads/")
(:eq "bucket" "user-uploads")
(:eq "acl" "public-read")
(:range "content-length-range" 1 10000)
These conditions are converted into a post policy description, base64-encoded, and returned as policy. The signature is returned as signature. These values can then be embedded in an HTML form and used to allow direct browser uploads.
For example, if policy is "YSBwYXRlbnRseSBmYWtlIHBvbGljeQ==" and the policy signature is "ZmFrZSBzaWduYXR1cmU=", you could construct a form like this:
<form action="http://user-uploads.s3.amazonaws.com/" method="post" enctype="multipart/form-data">
<input type="input" name="key" value="uploads/fun.jpg">
<input type=hidden name="acl" value="public-read">
<input type=hidden name="AWSAccessKeyId" value="8675309JGT9876430310">
<input type=hidden name="Policy" value="YSBwYXRlbnRseSBmYWtlIHBvbGljeQ==">
<input type=hidden name='Signature' value="ZmFrZSBzaWduYXR1cmU=">
<input name='file' type='file'>
<input type=submit value='Submit'>
</form>
For full, detailed documentation of browser-based POST uploads and policy documents, see Browser-Based Uploads Using POST in the Amazon S3 developer documentation.
head
&key
bucket
key
parameters
credentials
backoff
=> headers-alist,
status-code,
phrase
Submits a HTTP HEAD request for the resource identified by bucket and optionally key. Returns the Drakma headers, HTTP status code, and HTTP phrase as multiple values.
When parameters is supplied, it should be an alist of keys and values to pass as GET request parameters. For example:
* (head :bucket "zs3-demo" :parameters (parameters-alist :max-keys 0)) => ((:X-AMZ-ID-2 . "...") (:X-AMZ-REQUEST-ID . "...") (:DATE . "Sat, 27 Sep 2008 19:00:35 GMT") (:CONTENT-TYPE . "application/xml") (:TRANSFER-ENCODING . "chunked") (:SERVER . "AmazonS3") (:CONNECTION . "close")), 200, "OK"
authorized-url
&key
bucket
key
vhost
expires
ssl
sub-resource
credentials
=> url
Creates an URL that allows temporary access to a resource regardless of its ACL.
If neither bucket nor key is specified, the top-level bucket listing is accessible. If key is not specified, listing the keys of bucket is accessible. If both bucket and key are specified, the object specified by bucket and key is accessible.
expires is required, and should be the integer universal time after which the URL is no longer valid.
vhost controls the construction of the url. If vhost is nil, the constructed URL refers to the bucket, if present, as part of the path. If vhost is
:AMAZON
, the bucket name is used as a prefix to the Amazon hostname. If vhost is:FULL
, the bucket name becomes the full hostname of the url. For example:* (authorized-url :bucket "foo" :key "bar" :vhost nil) => "http://s3.amazonaws.com/foo/bar?..." * (authorized-url :bucket "foo" :key "bar" :vhost :amazon) => "http://foo.s3.amazonaws.com/bar?..." * (authorized-url :bucket "foo.example.com" :key "bar" :vhost :full) => "http://foo.example.com/bar?..."If ssl is true, the URL has "https" as the scheme, otherwise it has "http".
If sub-resource is specified, it is used as part of the query string to access a specific sub-resource. Example Amazon sub-resources include "acl" for access to the ACL, "location" for location information, and "logging" for logging information. For more information about the various sub-resources, see the Amazon S3 developer documentation.
resource-url
&key
bucket
key
vhost
ssl
sub-resource
=> url
Returns an URL that can be used to reference a resource. See AUTHORIZED-URL for more info.
Utility Functions
octet-vector
&rest
octets
=> octet-vector
Returns a vector of type
(simple-array (unsigned-byte 8) (*))
initialized with octets.
now+
delta
=> universal-time
Returns a universal time that represents the current time incremented by delta seconds. It's useful for passing as the
:EXPIRES
parameter to functions like PUT-OBJECT and AUTHORIZED-URL.
now-
delta
=> universal-time
Like NOW+, but decrements the current time instead of incrementing it.
file-etag
pathname
=> etag
Returns the etag of pathname. This can be useful for the conditional arguments
:WHEN-ETAG-MATCHES
and:UNLESS-ETAG-MATCHES
in GET-OBJECT and COPY-OBJECT.
parameters-alist
&rest
parameters
&key
&allow-other-keys
=> alist
Returns an alist based on all keyword arguments passed to the function. Keywords are converted to their lowercase symbol name and values are converted to strings. For example:
* (parameters-alist :name "Bob" :age 21) => (("name" . "Bob") ("age" . "21"))This can be used to construct Amazon metadata alists for PUT-OBJECT and COPY-OBJECT, or request parameters in HEAD.
clear-redirects
=> |
Clear ZS3's internal cache of redirections.
Most ZS3 requests are submitted against the Amazon S3 endpoint "s3.amazonaws.com". Some requests, however, are permanently redirected by S3 to new endpoints. ZS3 maintains an internal cache of permanent redirects, but it's possible for that cache to get out of sync if external processes alter the bucket structure
For example, if the bucket "eu.zs3" is created with a EU location constraint, S3 will respond to requests to that bucket with a permanent redirect to "eu.zs3.s3.amazonaws.com", and ZS3 will cache that redirect information. But if the bucket is deleted and recreated by a third party, the redirect might no longer be necessary.
CloudFront
CloudFront functions allow the creation and manipulation of distributions. In ZS3, distributions are represented by objects that reflect the state of a distributon at some point in time. It's possible for the distribution to change behind the scenes without notice, e.g. when a distribution's status is updated from "InProgress" to "Deployed".
The functions ENABLE, DISABLE, ENSURE-CNAME, REMOVE-CNAME, and SET-COMMENT are designed so that regardless of the state of the distribution provided, after the function completes, the new state of the distribution will reflect the desired update. The functions STATUS, CNAMES, and ENABLEDP do not automatically refresh the object and therefore might reflect outdated information. To ensure the object has the most recent information, use REFRESH. For example, to fetch the current, live status, use (status (refresh distribution)).
create-distribution
bucket-name
&key
cnames
enabled
comment
=> distribution
Creates and returns a new distribution object that will cache objects from the bucket named by bucket-name.If cnames is provided, it is taken as a designator for a list of additional domain names that can be used to access the distribution.
If enabled is NIL, the distribution is initially created in a disabled state. The default value is is T.
If comment is provided, it becomes part of the newly created distribution.
delete-distribution
distribution
=> |
Deletes distribution. Distributions must be disabled before deletion; see DISABLE.
refresh
distribution
=> distribution
Queries Amazon for the latest information regarding distribution and destructively modifies the instance with the new information. Returns its argument.
ensure-cname
distribution
cname
=> |
Adds cname to the CNAMEs of distribution, if necessary.
distributions-for-bucket
bucket-name
=> |
Returns a list of distributions that have bucket-name as the origin bucket.
distribution-error
All errors signaled as a result of a CloudFront request error are subtypes of distribution-error.
distribution-not-disabled
Distributions must be fully disabled before they are deleted. If they have not been disabled, or the status of the distribution is still "InProgress", distribution-not-disabled is signaled.
cname-already-exists
A CNAME may only appear on one distribution. If you attempt to add a CNAME to a distribution that is already present on some other distribution, cname-already-exists is signaled.
too-many-distributions
If creating a new distribution via CREATE-DISTRIBUTION would exceed the account limit of total distributions, too-many-distributions is signaled.
status
distribution
=> status
Returns a string describing the status of distribution. The status is either "InProgress", meaning that the distribution's configuration has not fully propagated through the CloudFront system, or "Deployed".
origin-bucket
distribution
=> origin-bucket
Returns the origin bucket for distribution. It is different from a normal ZS3 bucket name, because it has ".s3.amazonaws.com" as a suffix.
domain-name
distribution
=> domain-name
Returns the domain name through which CloudFront-enabled access to a resource may be made.
invalidate-paths
distribution
paths
=> invalidation
Initiates the invalidation of resources identified by paths in distribution. paths should consist of key names that correspond to objects in the distribution's S3 bucket.
The invalidation object reports on the status of the invalidation request. It can be queried with STATUS and refreshed with REFRESH.
* (invalidate-paths distribution '("/css/site.css" "/js/site.js")) #<INVALIDATION "I1HJC711OFAVKO" [InProgress]> * (progn (sleep 300) (refresh *)) #<INVALIDATION "I1HJC711OFAVKO" [Completed]>
References
Acknowledgements
Several people on freenode #lisp pointed out typos and glitches in this documentation. Special thanks to Bart "_3b" Botta for providing a detailed documentation review that pointed out glitches, omissions, and overly confusing passages.
James Wright corrected a problem with computing the string to
sign and URL encoding.
Feedback
If you have any questions or comments about ZS3, please email me, Zach Beane
For ZS3 announcements and development discussion, please see the zs3-devel mailing list.
2016-06-17
Copyright © 2008-2016 Zachary Beane, All Rights Reserved