# Building &CCL; from its Source Code

&CCL;, like many other Lisp implementations, consists of a kernel and a heap image. The kernel is an ordinary C program, and is built with a C compiler. It provides very basic and fundamental facilities, such as memory management, garbage collection, and bootstrapping. All the higher-level features are written in Lisp, and compiled into the heap image. Both parts are needed to have a working Lisp implementation; neither the kernel nor the heap image can stand alone.

You may already know that, when you have a C compiler which is written in C, you need a working C compiler to build the compiler. Similarly, the &CCL; heap image includes a Lisp compiler, which is written in Lisp. You therefore need a working Lisp compiler in order to build the Lisp heap image.

Where will you get a working Lisp compiler? No worries; you can use a precompiled copy of a (slightly older and compatible) version of &CCL;. This section explains how to do all this.

In principle it should be possible to use another implementation of Common Lisp as the host compiler, rather than an older &CCL;; this would be a challenging and experimental way to build, and is not described here.

## Building Definitions

The following terms are used in subsequent sections; it may be helpful to refer to these definitions.

fasl files are the object files produced by `compile-file`. fasl files store the machine code associated with function definitions and the external representation of other lisp objects in a compact, machine-readable form. fasl is short for “`FAS`t `L`oading”. &CCL; uses different pathname types (extensions) to name fasl files on different platforms; see

The
Lisp kernel is a C program with a fair amount of platform-specific assembly language code. Its basic job is to map a lisp heap image into memory, transfer control to some compiled lisp code that the image contains, handle any exceptions that occur during the execution of that lisp code, and provide various other forms of runtime support for that code. &CCL; uses different filenames to name the lisp kernel files on different platforms; see .

A
heap image is a file that can be quickly mapped into a process's address space. Conceptually, it's not too different from an executable file or shared library in the OS's native format (ELF or Mach-O/dyld format); for historical reasons, &CCL;'s own heap images are in their own (fairly simple) format. The term `full heap image` refers to a heap image file that contains all of the code and data that comprise &CCL;. &CCL; uses different filenames to name the standard full heap image files on different platforms; see .

A
bootstrapping image is a minimal heap image used in the process of building &CCL; itself. The bootstrapping image contains just enough code to load the rest of &CCL; from fasl files. It may help to think of the bootstrapping image as the egg and the full heap image as the chicken. &CCL; uses different filenames to name the standard bootstrapping image files on different platforms; see .

Each supported platform (and possibly a few as-yet-unsupported ones) has a uniquely named subdirectory of `ccl/lisp-kernel/`; each such
contains a Makefile and may contain some auxiliary files (linker scripts, etc.) that are used to build the lisp kernel on a particular platform.The platform-specific name of the kernel build directory is described in .

### Platform-specific filename conventions

#### Platform-specific filename conventions

Platform | kernel | full-image | boot-image | fasl extension | kernel-build directory
--- | --- | --- | --- | --- | ---
DarwinPPC32 | dppccl | dppccl.image | ppc-boot.image | .dfsl | darwinppc
LinuxPPC32 | ppccl | ppccl.image | ppc-boot | .pfsl | linuxppc
DarwinPPC64 | dppccl64 | dppccl64.image | ppc-boot64.image | .d64fsl | darwinppc64
LinuxPPC64 | ppccl64 | ppccl64.image | ppc-boot64 | .p64fsl | linuxppc64
LinuxX8664 | lx86cl64 | lx86cl64.image | x86-boot64 | .lx64fsl | linuxx8664
LinuxX8632 | lx86cl | lx86cl.image | x86-boot32 | .lx32fsl | linuxx8632
DarwinX8664 | dx86cl64 | dx86cl64.image | x86-boot64.image | .dx64fsl | darwinx8664
DarwinX8632 | dx86cl | dx86cl.image | x86-boot32.image | .dx32fsl | darwinx8632
FreeBSDX8664 | fx86cl64 | fx86cl64.image | fx86-boot64 | .fx64fsl | freebsdx8664
FreeBSDX8632 | fx86cl | fx86cl.image | fx86-boot32 | .fx32fsl | freebsdx8632
SolarisX64 | sx86cl64 | sx86cl64.image | sx86-boot64 | .sx64fsl | solarisx64
SolarisX86 | sx86cl | sx86cl.image | sx86-boot32 | .sx32fsl | solarisx86
Win64 | wx86cl64.exe | sx86cl64.image | wx86-boot64.image | .wx64fsl | win64
Win32 | wx86cl.exe | wx86cl.image | wx86-boot32.image | .wx32fsl | win32

## Setting Up to Build

At a given time, there are generally two versions of &CCL; that you might want to use (and therefore might want to build from source):

- The released version

- The development version, called the "trunk", which may contain both interesting new features and interesting new bugs

All versions are available for download from svn.clozure.com via the Subversion source control system.

For example, to get a released version (1.7 in this example), use a command like:

    svn co http://svn.clozure.com/publicsvn/openmcl/release/1.7/xxx/ccl

To get the trunk version, use:

    svn co http://svn.clozure.com/publicsvn/openmcl/trunk/xxx/ccl

Change the `xxx` to one of the following names: `darwinx86`, `linuxx86`, `freebsdx86`, `solarisx86`, `windows`, `linuxppc`, or `darwinppc`.

Tarball distributions of released versions are also available for download via ftp from: . For additional information about availability of source and distributions see the Clozure CL Trac.

Subversion client programs are pre-installed on Mac OS X 10.5 and later and are typically either pre-installed or readily available on Linux and FreeBSD platforms. The Subversion web page contains links to Subversion client programs for many platforms. Users of Mac OS X 10.4 or later can also install Subversion clients via Fink or MacPorts. On Debian Linux (and on related Linux distros such as Ubuntu) run `apt-get install subversion` or equivalent in the command-line or interactive package manager.

## Kernel Build Prerequisites

The &CCL; kernel can be built with the following widely available tools:

- cc or gcc — the GNU C compiler

- ld — the GNU linker

- m4 or gm4 — the GNU m4 macro processor

- as — the GNU assembler (version 2.10.1 or later)

- make — either GNU make or, on FreeBSD, the default BSD make program

In general, the more recent the versions of those tools, the better; some versions of gcc 3.x on Linux have difficulty compiling some of the kernel source code correctly (so gcc 4.0 should be used, if possible.) On Mac OS X, the versions of the tools distributed with Xcode should work fine; on Linux, the versions of the tools installed with the OS (or available through its package management system) should work fine if they're "recent enough". On FreeBSD, the installed version of the `m4` program doesn't support some features that the kernel build process depends on; the GNU version of the m4 macroprocessor (called `gm4` on FreeBSD) should be installed.

In order to build the lisp kernel on Mac OS X 10.6 Snow Leopard, you must install the optional 10.4 support when installing Xcode.

## Building Everything

You now have everything you need. Start up &CCL; with the `-n` or `--no-init` option to avoid potential interference from code in your init file, and evaluate the following form to bring your Lisp system up to date.

    ? (ccl:rebuild-ccl :full t)

That call to the function `rebuild-ccl` performs the following steps:

- Deletes all fasl files and other object files in the `ccl` directory tree

- Runs an external process that does a `make` in the current platform's kernel build directory to create a new kernel. This step can only work if the C compiler and related tools are installed; see .

- Does `(compile-ccl t)` in the running lisp, to produce a set of fasl files from the “higher level” lisp sources.

- Does `(xload-level-0 :force)` in the running lisp, to compile the lisp sources in the “ccl:level-0;” directory into fasl files and then create a bootstrapping image from those fasl files.

- Runs another external process, which causes the newly compiled lisp kernel to load the new bootstrapping image. The bootstrapping image then loads the “higher level” fasl files and a new copy of the platform's full heap image is then saved.

If all goes well, it'll all happen without user intervention and with some simple progress messages. If anything goes wrong during execution of either of the external processes, the process output is displayed as part of a lisp error message.

`rebuild-ccl` is essentially just a short cut for running all the individual steps involved in rebuilding the system. You can also execute these steps individually, as described below.

## Building the Kernel

The Lisp kernel is the executable that you run to use Lisp. It doesn't actually contain the entire Lisp implementation; rather, it loads a heap image which contains the specifics—the "library", as it might be called if this was a C program. The kernel also provides runtime support to the heap image, such as garbage collection, memory allocation, exception handling, and the OS interface.

The Lisp kernel file has different names on different platforms. See . On all platforms the lisp kernel sources reside in `ccl/lisp-kernel`.

This section gives directions on how to rebuild the Lisp kernel from its source code. Most &CCL; users will rarely have to do this. You probably will only need to do it if you are attempting to port &CCL; to a new architecture or extend or enhance its kernel in some way. As mentioned above, this step happens automatically when you do

    ? (rebuild-ccl :full t)

### Using "make" to build the lisp kernel

With those tools in place, do:

    shell> cd ccl/lisp-kernel/PLATFORM
    shell> make

That'll assemble several assembly language source files, compile several C source files, and link ../../the kernel.

## Building the Heap Image

The initial heap image is loaded by the Lisp kernel, and provides most of the language implementation The heap image captures the entire state of a running Lisp (except for external resources, such as open files and TCP sockets). After it is loaded, the contents of the new Lisp process's memory are exactly the same as those of the old Lisp process when the image was created.

The heap image is how we get around the fact that we can't run Lisp code until we have a working Lisp implementation, and we can't make our Lisp implementation work until we can run Lisp code. Since the heap image already contains a fully-working implementation, all we need to do is load it into memory and start using it.

If you're building a new version of &CCL;, you need to build a new heap image.

(You might also wish to build a heap image if you have a large program that is very complicated or time-consuming to load, so that you will be able to load it once, save an image, and thenceforth never have to load it again. At any time, a heap image capturing the entire memory state of a running Lisp can be created by calling the function `ccl:save-application`.)

### Development cycle

Creating a new &CCL; full heap image consists of the following steps:

1. Using your existing &CCL;, create a bootstrapping image

2. Using your existing &CCL;, recompile your updated &CCL; sources

3. Invoke &CCL; with the bootstrapping image you just created (rather than with the existing full heap image).

When you invoke &CCL; with the bootstrapping image, it starts up, loads all of the &CCL; fasl files, and saves out a new full heap image. Voila. You've created a new heap image.

A few points worth noting:

- There's a circular dependency between the full heap image and the bootstrapping image, in that each is used to build the other.

- There are some minor implementation differences, but the environment in effect after the bootstrapping image has loaded its fasl files is essentially equivalent to the environment provided by the full heap image; the latter loads a lot faster and is easier to distribute, of course.

- If the full heap image doesn't work (because of an OS compatibilty problem or other bug), it's very likely that the bootstrapping image will suffer the same problems.

Given a bootstrapping image and a set of up-to-date fasl files, the development cycle usually involves editing lisp sources (or updating those sources via `svn update`), recompiling modified files, and using the bootstrapping image to produce a new heap image.

### Generating a bootstrapping image

The bootstrapping image isn't provided in &CCL; distributions. It can be built from the source code provided in distributions (using a lisp image and kernel provided in those distributions) using the procedure described below.

The bootstrapping image is built by invoking a special utility inside a running &CCL; heap image to load files contained in the `ccl/level-0` directory. The bootstrapping image loads several dozen fasl files. After it's done so, it saves a heap image via `save-application`. This process is called "cross-dumping".

Given a source distribution, a lisp kernel, and a heap image, one can produce a bootstrapping image by first invoking &CCL; from the shell:

    shell> ccl
    Welcome to &CCL; .... !
    ?

then calling `ccl:xload-level-0` at the lisp prompt:

    ? (ccl:xload-level-0)

This function compiles the lisp sources in the `ccl/level-0` directory if they're newer than the corresponding fasl files and then loads the resulting fasl files into a simulated lisp heap contained in data structures inside the running lisp. That simulated heap image is then written to disk.

`xload-level-0` should be called whenever your existing boot image is out-of-date with respect to the source files in `ccl:level-0;` — For example:

    ? (ccl:xload-level-0 :force)

forces recompilation of the level-0 sources.

### Generating fasl files

Calling:

    ? (ccl:compile-ccl)

at the lisp prompt compiles any fasl files that are out-of-date with respect to the corresponding lisp sources; `(ccl:compile-ccl t)` forces recompilation. `ccl:compile-ccl` reloads newly-compiled versions of some files; `ccl:xcompile-ccl` is analogous, but skips this reloading step.

Unless there are bootstrapping considerations involved, it usually doesn't matter whether these files are reloaded after they're recompiled.

Calling `compile-ccl` or `xcompile-ccl` in an environment where fasl files don't yet exist may produce warnings to that effect whenever files are `require`d during compilation; those warnings can be safely ignored. Depending on the maturity of the &CCL; release, calling `compile-ccl` or `xcompile-ccl` may also produce several warnings about undefined functions, etc. They should be cleaned up at some point.

### Building a full image from a bootstrapping image

To build a full image from a bootstrapping image, just invoke the kernel with the bootstrapping image as an argument

    $ cd ccl                        # wherever your ccl directory is
    $ ./KERNEL --image-name BOOT_IMAGE --no-init

Where KERNEL and BOOT_IMAGE are the names of the kernel and boot image appropriate to the platform you are running on. See

That should load a few dozen fasl files (printing a message as each file is loaded.) If all of these files successfully load, the lisp will print a prompt. You should be able to do essentially everything in that environment that you can in the environment provided by a "real" heap image. If you're confident that things loaded OK, you can save that image:

    ? (ccl:save-application "image_name") ; Overwriting the existing heap image

Where image_name is the name of the full heap image for your platform. See .

If things go wrong in the early stages of the loading sequence, errors are often difficult to debug; until a fair amount of code (CLOS, the CL condition system, streams, the reader, the read-eval-print loop) is loaded, it's generally not possible for the lisp to report an error. Errors that occur during these early stages ("the cold load") sometimes cause the lisp kernel debugger (see ) to be invoked; it's primitive, but can sometimes help one to get oriented.