Interface Databases HOWTO
About Interface Databases
This HOWTO guide explains how to build interface databases
for Clozure CL (formerly OpenMCL).
Clozure CL's interface databases are
descriptions of foreign data structures and functions. Clozure
CL uses the interface databases to call C and Objective-C
functions, and to use C and Objective-C data
structures. Clozure CL provides the interface databases you
are most likely to need when writing Cocoa applications, but
it doesn't provide databases for every foreign library and
framework. For some applications, you may need to generate
interface databases yourself. This HOWTO shows how to do
that.
Online Documentation
The concepts explained here are also discussed in depth in
the OpenMCL online
documentation. See this
page for an explanation of how to generate interface
databases.
Interface Databases and the Foreign Function Interface
Clozure CL provides an integrated foreign function interface
that can call external C and Objective-C functions and methods,
define and instantiate Objective-C classes, and read and write
fields of C data structures. Using these features, a Lisp
program can interoperate freely with external libraries and
frameworks. Of special interest to Mac OS X programmers, Lisp
programs can link Mac OS X frameworks and use the features they
provide, including standard OS X features such as the AppKit,
CoreAudio, OpenGL, CoreAnimation, and so on.
Interface Databases and Frameworks
In order to use externally-defined functions and data
structures, Clozure CL needs descriptions of the entry points
and data fields, including their names and types. It reads
this information from the interface databases. A separate tool
called ffigen parses the header files that describe foreign
libraries and frameworks, such as the ones provided with
Apple's SDKs, to produce descriptions readable by Clozure CL's
parse-ffi subsystem. The parse-ffi subsystem then reads these
descriptions and writes the results of its processing to
files. These files, written by parse-ffi, are what we refer to
as interface databases.
Adding New Interface Databases
In order to generate a new set of interface databases, you
must follow these steps, explained in the following
sections:
Obtain and install the ffigen tool
Create an appropriately-named and -structured
subdirectory of the interfaces directory for your
platform
Write a script ("populate.sh") and then run it to populate the new interface
subdirectory
Using the parse-ffi subsystem, convert the ".ffi" files
created in the previous step to interface databases
1. Obtain and install ffigen
ffigen is a command-line tool, available from Clozure, that
parses C and Objective-C header files for use by the Clozure
CL parse-ffi subsystem. A "populate.sh" script drives the
ffigen tool to parse library or framework headers, and then
the Clozure CL parse-ffi subsystem converts the ffigen output
to interface databases. In order to generate interface
databases you must first obtain the latest version of ffigen
for the platform you are using.
ffigen is available from the Clozure ftp server
at clozure.com/pub/testing/. A
separate version of ffigen is available for each supported
platform. Make sure to get the latest version available for the
platform you are using. For example,
ffigen-apple-gcc-5465-x86-64-2007-11-06-00-00-59
supports Apple's C compiler ("gcc") on the 64-bit Intel
platform ("x86-64").
Once you have the appropriate version of ffigen, unpack and
install it in a location where it will be convenient to
use. You might, for example, install it in /usr/local/ffigen,
or in ~/ffigen.
The ffigen distribution unpacks into a directory with the
following structure:
bin/
h-to-ffi.sh
ffigen/
bin/
ffigen
include/
To install ffigen, unpack the distribution, then move the
unpacked directory to its install location. For example, to
install it in /usr/local/ffigen:
tar zxf ffigen-apple-gcc-5465-x86-64-2007-11-06-00-00-59.tar.gz
mv ffigen-apple-gcc-5465-x86-64-2007-11-06-00-00-59\
/usr/local/ffigen
You can now add a line to your shell's init script to add the
h-to-ffi.sh script to your PATH, and the script will find the
ffigen tool automatically.
For example, if your shell is bash, you can add this line to
your .bashrc or .bash_profile:
PATH="/usr/local/ffigen/bin:${PATH}"
2. Create An Interfaces Subdirectory
Clozure CL finds interface databases by consulting a
search-list. It initializes the search list when starting up,
and the exact contents of the search list depend on the
platform Clozure CL is running on. For example, a Mac OS X
version of Clozure CL running on the 64-bit Intel platform
searches "ccl/darwin-x86-headers64/", but a Mac OS X version
running on the 32-bit PowerPC platform searches
"ccl/darwin-headers/". We'll call this search directory
the interfaces directory. For the rest of this
discussion we'll assume we are running Clozure CL on Mac OS X
on the 64-bit Intel platform.
Each supported foreign library or framework has a
corresponding subdirectory of the interfaces directory. For
example, we can find a "cocoa" subdrectory of
"ccl/darwin-x86-headers64/"; it contains the interface databases
for the Cocoa framework. Similarly, the "gl" subdirectory
contains the interface databases for OpenGL.
In order to add support for a new framework, we must first
add a subdirectory for the framework to the interfaces
directory. For example, suppose we want to add support for the
TWAIN framework. This framework is one of the standard frameworks
provided in Mac OS X 10.5, but not one of those supported
out-of-the-box by Clozure CL.
To add TWAIN interfaces for use by Clozure CL, follow these
steps:
Create a subdirectory called "twain" in the interfaces directory
-
Create a subdirectory of "twain" called "C".
If you look inside any of the existing interfaces
subdirectories, you see that each one contains a similar
"C" subdirectory. The ffigen tool populates these "C"
subdirectories with ".ffi" files when it parses the
framework or library headers.
-
Inside the "C" subdirectory, create a file called
"populate.sh"
Again, you can see that the existing interfaces
subdirectories contain "populate.sh" files in their "C"
subdirectories. This file is the script that drives the
ffigen tool for the particular interface databases we are
creating. The next section explains what to put in the
"populate.sh" file.
3. Write and Run a "populate.sh" Script
If you look in the directory
ccl/darwin-x86-headers64/cocoa/C/
for example, you'll find a script named populate.sh. The
script sets up flags for the C compiler, and then calls
h-to-ffi.sh on each of several header files that define the
Objective-C data structures and entry points used by the Cocoa
framework. The first step in generating or regenerating the
interface databases is to run one of these populate.sh
scripts.
You can create a populate.sh script by copying an existing
one from another set of interfaces, then replacing the
references to the libraries or frameworks that the
populate.sh script processes. A good place to start is with
an existing populate.sh script that processes a library or
framework that is similar to, or related to, the one you want
to add.
For example, for our TWAIN interfaces, we can copy the
populate.sh script from the existing qtkit subdirectory. The
contents of the qtkit script are:
#!/bin/sh
# For now, things earlier than the 10.5 sdk are pointless
rm -rf System Developer usr
SDK=/Developer/SDKs/MacOSX10.5.sdk
CFLAGS="-m64 -fobjc-abi-version=2 -isysroot ${SDK} -mmacosx-version-min=10.5"; export CFLAGS
h-to-ffi.sh ${SDK}/System/Library/Frameworks/QTKit.framework/Headers/QTKit.h
Now we just need to change the reference to the QTKit header
file so that it refers instead to the TWAIN header file:
#!/bin/sh
# For now, things earlier than the 10.5 sdk are pointless
rm -rf System Developer usr
SDK=/Developer/SDKs/MacOSX10.5.sdk
CFLAGS="-m64 -fobjc-abi-version=2 -isysroot ${SDK} -mmacosx-version-min=10.5"; export CFLAGS
h-to-ffi.sh ${SDK}/System/Library/Frameworks/TWAIN.framework/Headers/TWAIN.h
Next you must run the script to generate the ".ffi" files
that will be used in the next step. Assuming you installed
ffigen as described in step 1, and made sure that it's on your
PATH, all you have to do is cd to the directory that contains
your new populate.sh script and execute it:
$ ./populate.sh
+++ /Developer/SDKs/MacOSX10.5.sdk/System/Library/Frameworks/TWAIN.framework/Headers/TWAIN.h
Now, if you look inside the C subdirectory of your twain
directory, you should find that it has been populated with
".ff" files:
oshirion:C mikel$ ls
./ ../ Developer/ populate.sh*
oshirion:C mikel$ ls Developer/SDKs/MacOSX10.5.sdk/System/Library/Frameworks/TWAIN.framework/Headers/
./ ../ TWAIN.ffi
4. Use parse-ffi to Generate Interface Databases
The ".ffi" files, created in the prevoius step, contain a
series of s-expressions readable by the Clozure CL parse-ffi
subsystem. Each s-expression describes a datastructure or
entry point in the foreign library.
Once you have created a set of these ".ffi" files by running
a populate.sh script, you must next use Clozure CL to convert
the ".ffi" files into interface database files.
First, launch Clozure CL
-
Next, evaluate this form to load the parse-ffi subsystem:
(require "PARSE-FFI")
-
Next, supposing you want to parse the TWAIN files
created in the previous section, evaluate the following form:
(ccl::parse-standard-ffi-files :twain)
The parse-ffi subsystem looks for a subdirectory named
"twain" in the interface databases directory for the
current platform.
The first time you run this expression, you'll see
warnings like this one:
; Warning: Interface file #P"ccl:darwin-x86-headers64;twain;types.cdb.newest" does not exist.
These warnings are normal.
-
Finally, evaluate the following form once more:
(ccl::parse-standard-ffi-files :twain)
In order to correctly describe some foreign definitions,
the parse-ffi subsystem needs information provided by the
first parse. Thus, to produce a complete and corrrect set of
interface databases, you should always evaluate
the parse-standard-ffi-files
form twice.
Conclusion
Assuming all went well, you should now find a new set of
interface database files in the twain subdirectory of the
interface databases directory for your current
platform. Congratulations! You have added a new set of
interface databases to your Clozure CL installation.
For more information about ffigen and the interface
translation process, refer to the online documentation for the
Interface
Translator.