# Streams

## Stream Extensions

### Stream External Format

`STREAM-EXTERNAL-FORMAT` can be applied to (and may return a non-null result for) open streams that are not `FILE-STREAM`s.

`(SETF STREAM-EXTERNAL-FORMAT)` can be used to change the external format of open streams created with `OPEN` or `MAKE-SOCKET`.

### Additional keywords for OPEN and MAKE-SOCKET

`OPEN` and `MAKE-SOCKET` have each been extended to take the additional keyword arguments: `:CLASS`, `:SHARING`, and `:BASIC`.

`:CLASS`

  A symbol that names the desired class of the stream. The specified class must inherit from `FILE-STREAM` for `OPEN`.

`:SHARING`

  Specifies how a stream can be used by multiple threads. The possible values are: `:PRIVATE`, `:LOCK` and `:EXTERNAL`. `:PRIVATE` is the default. `NIL` is also accepted as a synonym for `:EXTERNAL`.

  `:PRIVATE`

    Specifies that the stream can only be accessed by the thread that first tries to do I/O to it; that thread becomes the "owner" of the stream and is not necessarily the same thread as the one which created the stream. This is the default. (There was some discussion on openmcl-devel about the idea of "transferring ownership" of a stream; this has not yet been implemented.) Attempts to do I/O on a stream with :PRIVATE sharing from a thread other than the stream's owner yield an error.

  `:LOCK`

    Specifies that all access to the stream require the calling thread to obtain a lock. There are separate "read" and "write" locks for IO streams. This makes it possible for instance, for one thread to read from such a stream while another thread writes to it. (see also )

  `:EXTERNAL`

    Specifies that I/O primitives enforce no access protocol. This may be appropriate for some types of application which can control stream access via application-level protocols. Note that since even the act of reading from a stream changes its internal state (and simultaneous access from multiple threads can therefore lead to corruption of that state), some care must be taken in the design of such protocols.

`:BASIC`

  A boolean that indicates whether or not the stream is a Gray stream, i.e. whether or not the stream is an instance of `FUNDAMENTAL-STREAM` or `CCL::BASIC-STREAM`(see ). Defaults to `T`.

### Basic Versus Fundamental Streams

Gray streams (see ) all inherit from `FUNDAMENTAL-STREAM` whereas basic streams inherit from `CCL::BASIC-STREAM`. The tradeoff between FUNDAMENTAL and BASIC streams is entirely between flexibility and performance, potential or actual. I/O primitives can recognize BASIC-STREAMs and exploit knowledge of implementation details. FUNDAMENTAL stream classes can be subclassed and extended in a standard way (the Gray streams protocol).

For existing stream classes (FILE-STREAMs, SOCKETs, and the internal CCL::FD-STREAM classes used to implement file streams and sockets), a lot of code can be shared between the FUNDAMENTAL and BASIC implementations. The biggest difference should be that that code can be reached from I/O primitives like READ-CHAR without going through some steps that're there to support generality and extensibility, and skipping those steps when that support isn't needed can improve I/O performance.

The Gray stream method `STREAM-READ-CHAR` should work on appropriate `BASIC-STREAM`s. (There may still be cases where such methods are undefined; such cases should be considered bugs.) It is not guaranteed that Gray stream methods would ever be called by I/O primitives to read a character from a `BASIC-STREAM`, though there are still cases where this happens.

A simple loop reading 2M characters from a text file runs about 10X faster when the file is opened the new defaults `(:SHARING :PRIVATE :BASIC T)` than it had before these changes were made. That sounds good, until one realizes that the "equivalent" C loop can be about 10X faster still ...

### Stream Timeouts and Deadlines

A stream that is associated with a file descriptor has attributes and accessors: `STREAM-INPUT-TIMEOUT`, `STREAM-OUTPUT-TIMEOUT`, and `STREAM-DEADLINE`. All three accessors have corresponding `SETF` methods. `STREAM-INPUT-TIMEOUT` and `STREAM-OUTPUT-TIMEOUT` are specified in seconds and can be any positive real number less than one million. When a timeout is set and the corresponding I/O operation takes longer than the specified interval, an error is signalled. The error is `INPUT-TIMEOUT` for input and `OUTPUT-TIMEOUT` for output. `STREAM-DEADLINE` specifies an absolute time in internal-time-units. If an I/O operation on the stream does not complete before the deadline then a `COMMUNICATION-DEADLINE-EXPIRED` error is signalled. A deadline takes precedence over any input/output timeouts that may be set.

### Open File Streams

Historically, &CCL; and MCL maintained a list of open file streams in the value of `CCL:*OPEN-FILE-STREAMS*`. This functionality has been replaced with the thread-safe function: `CCL:OPEN-FILE-STREAMS` and its two helper functions: `CCL:NOTE-OPEN-FILE-STREAM` and `CCL:REMOVE-OPEN-FILE-STREAM`. Maintaining this list helps to ensure that streams get closed in an orderly manner when the lisp exits.

#### OPEN-FILE-STREAMS

Returns the list of file streams that are currently open.
Function

    	        open-file-streams
    	        => stream-list

##### Values

stream-list

  A list of open file streams. This is a copy of an internal list so it may be destructively modified without ill effect.

##### Description

Returns a list of open file streams.

##### See Also

#### NOTE-OPEN-FILE-STREAM

Adds a file stream to the internal list of open file streams that is returned by `note-open-file-stream`.
Function

    	        note-open-file-stream
    	        file-stream

##### Arguments

file-stream

  A file stream.

##### Description

Adds a file stream to the internal list of open file streams that is returned by `open-file-streams`. This function is thread-safe. It will usually only be called from custom stream code when a file-stream is created.

##### See Also

#### REMOVE-OPEN-FILE-STREAM

Removes file stream from the internal list of open file streams that is returned by `open-file-streams`.
Function

    	        remove-open-file-stream
    	        file-stream

##### Arguments

file-stream

  A file stream.

##### Description

Remove file stream from the internal list of open file streams that is returned by `open-file-streams`. This function is thread-safe. It will usually only be called from custom stream code when a file-stream is closed.

##### See Also

## Creating Your Own Stream Classes with Gray Streams

### Overview

This sect1 is still being written and revised, because it is woefully incomplete. The dictionary section currently only lists a couple functions. Caveat lector.

Gray streams are an extension to Common Lisp. They were proposed for standardization by David Gray (the astute reader now understands their name) quite some years ago, but not accepted, because they had not been tried sufficiently to find conceptual problems with them.

They have since been implemented by quite a few modern Lisp implementations. However, they do indeed have some inadequacies, and each implementation has addressed these in different ways. The situation today is that it's difficult to even find out how to get started using Gray streams. This is why standards are important.

Here's a list of some classes which you might wish for your new stream class to inherit from:

- fundamental-stream
- fundamental-input-stream
- fundamental-output-stream
- fundamental-character-stream
- fundamental-binary-stream
- fundamental-character-input-stream
- fundamental-character-output-stream
- fundamental-binary-input-stream
- fundamental-binary-output-stream
- ccl::buffered-stream-mixin
- ccl::buffered-input-stream-mixin
- ccl::buffered-output-stream-mixin
- ccl::buffered-io-stream-mixin
- ccl::buffered-character-input-stream-mixin
- ccl::buffered-character-output-stream-mixin
- ccl::buffered-character-io-stream-mixin
- ccl::buffered-binary-input-stream-mixin
- ccl::buffered-binary-output-stream-mixin
- ccl::buffered-binary-io-stream-mixin
- file-stream
- file-input-stream
- file-output-stream
- file-io-stream
- file-character-input-stream
- file-character-output-stream
- file-character-io-stream
- file-binary-input-stream
- file-binary-output-stream
- file-binary-io-stream
- ccl::fd-stream
- ccl::fd-input-stream
- ccl::fd-output-stream
- ccl::fd-io-stream
- ccl::fd-character-input-stream
- ccl::fd-character-output-stream
- ccl::fd-character-io-stream
- ccl::fd-binary-input-stream
- ccl::fd-binary-output-stream
- ccl::fd-binary-io-stream

All of these are defined in ccl/level-1/l1-streams.lisp, except for the ccl:file-* ones, which are in ccl/level-1/l1-sysio.lisp.

According to the original Gray streams proposal, you should inherit from the most specific of the fundamental-* classes which applies. Using &CCL;, though, if you want buffering for better performance, which, unless you know of some reason you wouldn't, you do, you should instead inherit from the appropriate ccl::buffered-* class The buffering you get this way is exactly the same as the buffering which is used on ordinary, non-Gray streams, and force-output will work properly on it.

Notice that -mixin suffix in the names of all the ccl::buffered-* classes? The suffix means that this class is not "complete" by itself; you still need to inherit from a fundamental-* stream, even if you also inherit from a *-mixin stream. You might consider making your own class like this. .... Except that they do inherit from the fundamental-* streams, that's weird.

If you want to be able to create an instance of your class with the :class argument to (open) and (with-open-file), you should make it inherit from one of the file-* classes. If you do this, it's not necessary to inherit from any of the other classes (though it won't hurt anything), since the file-* classes already do.

When you inherit from the file-* classes, you can use (call-next-method) in any of your methods to get the standard behavior. This is especially useful if you want to create a class which performs some simple filtering operation, such as changing everything to uppercase or to a different character encoding. If you do this, you will definitely need to specialize ccl::select-stream-class. Your method on ccl::stream-select-class should accept an instance of the class, but pay no attention to its contents, and return a symbol naming the class to actually be instantiated.

If you need to make your functionality generic across all the different types of stream, probably the best way to implement it is to make it a mixin, define classes with all the variants of input, output, io, character, and binary, which inherit both from your mixin and from the appropriate other class, then define a method on ccl::select-stream-class which chooses from among those classes.

Note that some of these classes are internal to the CCL package. If you try to inherit from those ones without the ccl:: prefix, you'll get an error which may confuse you, calling them "forward-referenced classes". That just means you used the wrong symbol, so add the prefix.

Here's a list of some generic functions which you might wish to specialize for your new stream class, and which ought to be documented at some point.

- stream-direction stream =>
- stream-device stream direction =>
- stream-length stream &optional; new =>
- stream-position stream &optional; new =>
- streamp stream => boolean
- stream-write-char output-stream char =>
- stream-write-entire-string output-stream string =>
- stream-read-char input-stream =>
- stream-unread-char input-stream char =>
- stream-force-output output-stream => nil
- stream-maybe-force-output output-stream => nil
- stream-finish-output output-stream => nil
- stream-clear-output output-stream => nil
- close stream &key; abort => boolean
- stream-fresh-line stream => t
- stream-line-length stream => length
- interactive-stream-p stream => boolean
- stream-clear-input input-stream => nil
- stream-listen input-stream => boolean
- stream-filename stream => string
- ccl::select-stream-class instance in-p out-p char-p => class

The following functions are standard parts of Common Lisp, but behave in special ways with regard to Gray streams.

- open-stream-p stream => generalized-boolean
- input-stream-p stream => generalized-boolean
- output-stream-p stream => generalized-boolean
- stream-element-type stream =>
- stream-error-stream =>
- open
- close
- with-open-file

Specifically, (open) and (with-open-file) accept a new keyword argument, :class, which may be a symbol naming a class; the class itself; or an instance of it. The class so given must be a subtype of 'stream, and an instance of it with no particular contents will be passed to ccl::select-stream-class to determine what class to actually instantiate.

The following are standard, and do not behave specially with regard to Gray streams, but probably should.

- stream-external-format

### Extending READ-SEQUENCE and WRITE-SEQUENCE

#### Overview

The "Gray Streams" API is based on an informal proposal that was made before ANSI CL adopted the READ-SEQUENCE and WRITE-SEQUENCE functions; as such, there is no "standard" way for the author of a Gray stream class to improve the performance of these functions by exploiting knowledge of the stream's internals (e.g., the buffering mechanism it uses.)

In the absence of any such knowledge, READ-SEQUENCE and WRITE-SEQUENCE are effectively just convenient shorthand for a loop which calls READ-CHAR/READ-BYTE/WRITE-CHAR/WRITE-BYTE as appropriate. The mechanism described below allows subclasses of FUNDAMENTAL-STREAM to define more specialized (and presumably more efficient) behavior.

#### Notes

READ-SEQUENCE and WRITE-SEQUENCE do a certain amount of sanity-checking and normalization of their arguments before dispatching to one of the methods above. If an individual method can't do anything particularly clever, CALL-NEXT-METHOD can be used to handle the general case.

#### Example

    (defclass my-string-input-stream (fundamental-character-input-stream)
      ((string :initarg :string :accessor my-string-input-stream-string)
       (index :initform 0 :accessor my-string-input-stream-index)
       (length)))

    (defmethod stream-read-vector ((stream my-string-input-stream) vector start end)
      (if (not (typep vector 'simple-base-string))
          (call-next-method)
          (with-slots (string index length)
    	      (do* ((outpos start (1+ outpos)))
                   ((or (= outpos end)
                        (= index length))
                    outpos))
            (setf (schar vector outpos)
                  (schar string index))
            (incf index)))))

### Multibyte I/O

All heap-allocated objects in &CCL; that cannot contain pointers to lisp objects are represented as ivectors. &CCL; provides low-level functions, and , to efficiently transfer data between buffered streams and ivectors. There's some overlap in functionality between the functions described here and the ANSI CL READ-SEQUENCE and WRITE-SEQUENCE functions.

As used here, the term "octet" means roughly the same thing as the term "8-bit byte". The functions described below transfer a specified sequence of octets between a buffered stream and an ivector, and don't really concern themselves with higher-level issues (like whether that octet sequence is within bounds or how it relates to the logical contents of the ivector.) For these reasons, these functions are generally less safe and more flexible than their ANSI counterparts.

### Gray Streams Dictionary

#### CCL:STREAM-READ-LIST

Generic Function

    stream-read-list
    	        stream list count

##### Arguments and Values

stream

  a stream, presumably a fundamental-input-stream.

list

  a list. When a STREAM-READ-LIST method is called by READ-SEQUENCE, this argument is guaranteed to be a proper list.

count

  a non-negative integer. When a STREAM-READ-LIST method is called by READ-SEQUENCE, this argument is guaranteed not to be greater than the length of the list.

##### Description

Should try to read up to count elements from stream into the list list, returning the number of elements actually read (which may be less than count in case of a premature end-of-file.)

#### CCL:STREAM-WRITE-LIST

Generic Function

    stream-write-list
    	        stream list count

##### Arguments and Values

stream

  a stream, presumably a fundamental-output-stream.

list

  a list. When a STREAM-WRITE-LIST method is called by WRITE-SEQUENCE, this argument is guaranteed to be a proper list.

count

  a non-negative integer. When a STREAM-WRITE-LIST method is called by WRITE-SEQUENCE, this argument is guaranteed not to be greater than the length of the list.

##### Description

should try to write the first count elements of list to stream. The return value of this method is ignored.

#### CCL:STREAM-READ-VECTOR

Generic Function

    stream-read-vector
    	        stream vector start end

##### Arguments and Values

stream

  a stream, presumably a fundamental-input-stream

vector

  a vector. When a STREAM-READ-VECTOR method is called by READ-SEQUENCE, this argument is guaranteed to be a simple one-dimensional array.

start

  a non-negative integer. When a STREAM-READ-VECTOR method is called by READ-SEQUENCE, this argument is guaranteed to be no greater than end and not greater than the length of vector.

end

  a non-negative integer. When a STREAM-READ-VECTOR method is called by READ-SEQUENCE, this argument is guaranteed to be no less than end and not greater than the length of vector.

##### Description

should try to read successive elements from stream into vector, starting at element start (inclusive) and continuing through element end (exclusive.) Should return the index of the vector element beyond the last one stored into, which may be less than end in case of premature end-of-file.

#### CCL:STREAM-WRITE-VECTOR

Generic Function

    stream-write-vector
    	        stream vector start end

##### Arguments and Values

stream

  a stream, presumably a fundamental-output-stream

vector

  a vector. When a STREAM-WRITE-VECTOR method is called by WRITE-SEQUENCE, this argument is guaranteed to be a simple one-dimensional array.

start

  a non-negative integer. When a STREAM-WRITE-VECTOR method is called by WRITE-SEQUENCE, this argument is guaranteed to be no greater than end and not greater than the length of vector.

end

  a non-negative integer. When a STREAM-WRITE-VECTOR method is called by WRITE-SEQUENCE, this argument is guaranteed to be no less than end and not greater than the length of vector.

##### Description

should try to write successive elements of vector to stream, starting at element start (inclusive) and continuing through element end (exclusive.)

#### CCL::STREAM-DEVICE

Returns the OS file descriptor associated with a given lisp stream.
Generic Function

    ccl::stream-device
    	        s direction

##### Method Signatures

    ccl::stream-device
    	        (s stream) direction => fd

##### Arguments and Values

s

  a stream.

direction

  either :INPUT or :OUTPUT.

fd

  a file descriptor, which is a non-negative integer used by the OS to refer to an open file, socket, or similar I/O connection. NIL if there is no file descriptor associated with `s` in the direction given by `direction`.

##### Description

Returns the file descriptor associated with `s` in the direction given by `direction`. It is necessary to specify `direction` because the input and output file descriptors may be different; the most common case is when one of them has been redirected by the Unix shell.

#### STREAM-READ-IVECTOR

Generic Function

    stream-read-ivector
    	        stream ivector start-octet max-octets

##### Description

Reads up to max-octets octets from stream into ivector, storing them at start-octet. Returns the number of octets actually read.

##### Arguments

stream

  An input stream. The method defined on BUFFERED-INPUT-STREAMs requires that the size in octets of an instance of the stream's element type is 1.

ivector

  Any ivector.

start-octet

  A non-negative integer.

max-octets

  A non-negative integer. The return value may be less than the value of this parameter if EOF was encountered.

#### STREAM-WRITE-IVECTOR

Generic Function

    stream-write-ivector stream
    	        ivector start-octet max-octets

##### Description

Writes max-octets octets to stream from ivector, starting at start-octet. Returns max-octets.

##### Arguments

stream

  An input stream. The method defined on BUFFERED-OUTPUT-STREAMs requires that the size in octets of an instance of the stream's element type is 1.

ivector

  Any ivector

start-octet

  A non-negative integer.

max-octet

  A non-negative integer.

##### Examples

    ;;; Write the contents of a (SIMPLE-ARRAY(UNSIGNED-BYTE 16) 3)
    ;;; to a character file stream. Read back the characters.
    (let* ((a (make-array 3
                          :element-type '(unsigned-byte 16)
                          :initial-contents '(26725 27756 28449))))
      (with-open-file (s "junk"
                         :element-type 'character
                         :direction :io
                         :if-does-not-exist :create
                         :if-exists :supersede)
        ;; Write six octets (three elements).
        (stream-write-ivector s a 0 6)
        ;; Rewind, then read a line
        (file-position s 0)
        (read-line s)))

    ;;; Write a vector of DOUBLE-FLOATs. Note that (to maintain
    ;;; alignment) there are 4 octets of padding before the 0th
    ;;; element of a (VECTOR DOUBLE-FLOAT) on 32-bit platforms.
    ;;; (Note that (= (- target::misc-dfloat-offset
    ;;;                  target::misc-data-offset) 4))
    (defun write-double-float-vector
        (stream vector &key (start 0) (end (length vector)))
         (check-type vector (vector double-float))
         (let* ((start-octet (+ (* start 8)
                                (- target::misc-dfloat-offset
                                   target::misc-data-offset)))
    	        (num-octets (* 8 (- end start))))
           (stream-write-ivector stream vector start-octet num-octets)))

## Lisp Standard Streams and OS Standard Streams

In normal interactive usage, the input and output sides of the bidirectional stream `*terminal-io*` are hooked up to the the operating system's standard input and standard output. The lisp streams `*standard-input*`, `*standard-output*`, and `*error-output*` are synonym streams for `*terminal-io*`.

In batch mode, this arrangement is modified slightly. The lisp streams `*standard-input*`, `*standard-output*`, and `*standard-error*` correspond directly to the operating system's standard input, standard output, and standard error. If the lisp can determine that it has access to an operating system tty, then `*terminal-io*` will be hooked up to that. Otherwise, the input and output streams of `*terminal-io*` will correspond to the operating system's standard input and standard output.