\chapter{The Foreign-Function Interface}

\section{Using Foreign Memory}

\CCL\ represents arbitrary memory addresses with a special Lisp object
called a {\it macptr}
\footnote{\CCL's ancestor first ran on the Apple Macintosh: a {\it
    macptr} is a ``Mac pointer'', which refers to an address is the
  Macintosh heap}.
Two macptr objects are \cd{eql} if they
encapsulate the same address.  A macptr may also be called a foreign
pointer.

It is often convenient to blur the distiction between a mactpr and and
the address it represents.  At times, though, one must keep the
distiction clear.  It is important to remember that a macptr is a
first-class Lisp object in the same sense a cons cell is: a macptr
will be GCed when it is no longer possible to reference it.  However,
the lifetime of a macptr generally has nothing to do with the lifetime
of the block of memory that the macptr's address points to.

A new macptr is created when calling a foreign function that returns
an address, or when referencing memory that is specified to contain an
address.  A mactpr may also be created explicitly from a specified
address or otherwise altered directly.

Primitive macptr-creating operations are usually open-coded by the
compiler: it has a fairly good notion of what low-level operations
produce macptrs and which operations consume the addresses that they
encapsulate.  The compiler will usually be able to optimize out the
introduction of intermediate mactpr objects in a simple expression.

Nevertheless, one consequence of the use of macptr objects to
encapsulate foreign addresses is that, conceptually, every reference
to a foreign address causes a new macptr to be allocated.

\subsection{Low-level Memory Accessors}

The following functions provide low-level access to foreign memory.
They are typically open-coded by the compiler.

\begin{defun}[Function]
\%get-signed-byte ptr &optional (offset 0) \\
\%get-unsigned-byte ptr &optional (offset 0) \\
\%get-signed-word ptr &optional (offset 0) \\
\%get-unsigned-word ptr &optional (offset 0) \\
\%get-signed-long ptr &optional (offset 0) \\
\%get-unsigned-long ptr &optional (offset 0) \\
\%\%get-signed-longlong ptr &optional (offset 0) \\
\%\%get-unsigned-longlong ptr &optional (offset 0) \\
\%get-ptr ptr &optional (offset 0) \\
\%get-single-float ptr &optional (offset 0) \\
\%get-double-float ptr &optional (offset 0)

These functions compute an effective byte address by adding {\it
  offset} to the address represented by {\it ptr}, and return the the
signed or unsigned 8-bit byte, signed or unsigned 16-bit word, signed
or unsigned 32-bit long word, signed or unsigned 64-bit long long
word, address, 32-bit single-float, or 64-bit double-float found
at that address.

All of these functions may be used with \cd{setf}.
\end{defun}

\subsection{Creating and Altering Foreign Pointers}

Sometimes it is useful to create or alter foreign pointers directly.

\begin{defun}[Function]
\%int-to-ptr int \\
\%ptr-to-int ptr

The function \cd{\%int-to-ptr} creates a new mactpr whose address
matches {\it int}.

The function \cd{\%ptr-to-int} does the reverse: it returns
the address encapsulated by the macptr {\it ptr}.
\end{defun}

\begin{defun}[Macro]
\%null-ptr

This macro is equivalent to writing \cd{(\%int-to-ptr 0)}.
\end{defun}

\begin{defun}[Constant]
+null-ptr+

This constant variable is bound to the result of \cd{(\%null-ptr)}.
\end{defun}

\begin{defun}[Function]
\%null-ptr-p thing

This function returns \cd{t} if {\it thing} is a null macptr (that
is, it encapsulates the address 0).  Otherwise, it returns \nil.
\end{defun}

\begin{defun}[Function]
\%inc-ptr ptr &optional delta

This function returns a new macptr whose address is the
address of {\it ptr} plus {\it delta} (default 1).  The idiom \cd{(\%inc-ptr ptr 0)}
may be used to copy a macptr.
\end{defun}

\begin{defun}[Function]
\%setf-macptr dest-ptr src-ptr \\
\%incf-macptr dest-ptr delta

These functions destructively modify {\it dest-ptr} by changing
the encapsulated address.

The \cd{\%setf-macptr} function causes {\it dest-ptr} to encapsulate
the same address that {\it src-ptr} does.  It then returns {\it dest-ptr}.

The \cd{\%incf-ptr} function adds {\it delta} (default 1) to
the address that {\it dest-ptr} encapsulates.  It then returns
{\it dest-ptr}.
\end{defun} 

\subsection{Stack-allocated Memory}

It is not uncommon for blocks of foreign memory to have well-defined,
short, lifetimes.

\begin{lstlisting}
(let ((p (#_malloc size)))
  (unwind-protect
      (use-foreign-memory p)
    (#_free p)))
\end{lstlisting}

This is not unreasoanble code, but it contains a number of
inefficiencies.  A foreign function call has a certain cost (as does
\cd{unwind-protect}), and library routines like malloc and free can be
fairly expensive in their own right.

In the code above, both {\it p} and the block of memory that it points
to have dynamic extent, and are therefore good candidates for stack
allocation.  \CCL\ provides the following macro that is useful for
such situations.

\begin{defun}[Macro]
\%stack-block ((var size)*) {\,form}*

This macro arranges to execute the provided series of forms with
specified variables bound to stack-allocated \cd{macptr}s which
encapsulate the addresses of stack-allocated blocks of specified
sizes.

The code above may be written as

\begin{lstlisting}
(%stack-block ((p size))
  (use-foreign-memory p))
\end{lstlisting}

which is a bit more efficient and a bit more concise.  Often, it is
possible to use slightly higher-level constructs such as \cd{rlet}
instead of using \cd{\%stack-block} directly.  When the size of the
block of memory is variable, however, \cd{\%stack-block} must be
used directly.
\end{defun}

\subsection{Stack-allocated macptrs}

Consider a code fragment like the following:

\begin{lstlisting}
(defun get-next-event ()
  "get the next event from a hypothetical window system"
  (loop
     (let ((event (#_get_next_event)))
       (unless (null-event-p event)
         (handle-event event)))))
\end{lstlisting}

As this is written, each call to the foreign function
\cd{\#\_get\_next\_event} will return a new macptr object.  Ignoring
for the sake of argument the question of whether this code fragment
exhibits a good way to poll for external events (it doesn't), it's not
hard to imagine that this loop could execute several million times per
second (producing several million macptrs per second.)  Clearly, this
naive approach is impractical in many cases.

If we knew that that neither \cd{null-event-p} nor \cd{handle-event}
cached or otherwise retained the {\it event} pointer, there'd be a few
alternatives to the naive approach. One of those approaches would be
to use the primitive function \cd{\%setf-macptr} to destructively
modify a macptr to change the value of the address it encapsulates.
The \cd{get-next-event} example could be rewritten as:

\begin{lstlisting}
(defun get-next-event ()
  (let ((event (%null-ptr))))
  (loop
    (%setf-macptr event (#_get_next_event))
    (unless (null-event-p event)
      (handle-event event))))
\end{lstlisting}

That version's a bit more realistic: it allocates a single macptr
outside if the loop, then changes its address to point to the current
address of the hypothetical event structure on each loop iteration. If
there are a million loop iterations per call to \cd{get-next-event},
we're allocating a million times fewer macptrs per call; that
sounds like a Good Thing.

An Even Better Thing would be to advise the compiler that the initial
value (the null macptr) bound to the variable event has dynamic extent
(that value won't be referenced once control leaves the extent of the
binding of that variable.) Common Lisp allows us to make such an
assertion via a dynamic-extent declaration; \CCL's compiler can
recognize the primitive macptr-creating operation involved and can
replace it with an equivalent operation that stack-allocates the
macptr object. If we're not worried about the cost of allocating that
macptr on every iteration (the cost is small and there's no hidden GC
cost), we could move the binding back inside the loop:

\begin{lstlisting}
(defun get-next-event ()
  (loop
     (let ((event (%null-ptr)))
       (declare (dynamic-extent event))
       (%setf-macptr event (#_get_next_event))
       (unless (null-event-p event)
         (handle-event event)))))
\end{lstlisting}

The idiom of binding one or more variables to stack-{}allocated
macptrs, then destructively modifying those macptrs before executing a
body of code is common enough that \CCL\ provides the macro
\cd{with-macptrs} that handles all of the gory details. The following
version of \cd{get-next-event} is semantically equivalent to the
previous version, but hopefully a bit more concise:

\begin{lstlisting}
(defun get-next-event ()
  (loop
     (with-macptrs ((event (#_get_next_event)))
       (unless (null-event-p event)
         (handle-event event)))))
\end{lstlisting}


\subsection{Caveats}

Reading from, writing to, allocating, and freeing foreign memory are
all potentially dangerous operations; this is no less true when these
operations are performed in \CCL\ than when they're done in C or some
other lower-level language. In addition, destructive operations on
Lisp objects be dangerous, as can stack allocation if it's abused (if
dynamic-extent declarations are violated.)  Correct use of the
constructs and primitives described here is reliable and safe; even
slightly incorrect use of these constructs and primitives can crash
\CCL.

