<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"[
          <!ENTITY rest "<varname>&amp;rest</varname>">
          <!ENTITY key "<varname>&amp;key</varname>">
          <!ENTITY optional "<varname>&amp;optional</varname>">
          <!ENTITY body "<varname>&amp;body</varname>">
          <!ENTITY aux "<varname>&amp;aux</varname>">
          <!ENTITY allow-other-keys "<varname>&amp;allow-other-keys</varname>">
          <!ENTITY CCL "Clozure CL">
          ]>
<chapter id="building-ccl-from-source"><title>Building &CCL; from its Source Code</title>
  <anchor id="Building-CCL"/>
  <para>&CCL;, like many other Lisp implementations, consists of a
    kernel and a heap image.  The kernel is an ordinary C program, and
    is built with a C compiler.  It provides very basic and
    fundamental facilities, such as memory management, garbage
    collection, and bootstrapping.  All the higher-level features are
    written in Lisp, and compiled into the heap image.  Both parts are
    needed to have a working Lisp implementation; neither the kernel
    nor the heap image can stand alone.</para>

  <para>You may already know that, when you have a C compiler which
    is written in C, you need a working C compiler to build the
    compiler. Similarly, the &CCL; heap image includes a Lisp
    compiler, which is written in Lisp. You therefore need a working
    Lisp compiler in order to build the Lisp heap image.</para>
  
  <para>Where will you get a working Lisp compiler?  No worries; you
    can use a precompiled copy of a (slightly older and compatible)
    version of &CCL;. This section explains how to do all this.</para>

  <para>In principle it should be possible to use another
    implementation of Common Lisp as the host compiler, rather than an
    older &CCL;; this would be a challenging and experimental way to
    build, and is not described here.</para>

  <!-- =========================================================== -->
  <sect1 id="building-definitions">
    <title>Building Definitions</title>
    <para>The following terms are used in subsequent sections; it
      may be helpful to refer to these definitions.</para>

    <para><indexterm><primary>fasl
          files</primary></indexterm><glossterm linkend="fasl-file">fasl
        files</glossterm> are the object files produced
      by <literal>compile-file</literal>.  fasl files store the
      machine code associated with function definitions and the
      external representation of other lisp objects in a compact,
      machine-readable form. fasl is short for
      &ldquo;<literal>FAS</literal>t
      <literal>L</literal>oading&rdquo;. &CCL; uses different pathname
      types (extensions) to name fasl files on different platforms;
      see
      <xref linkend="Platform-specific-filename-conventions"/> </para>

    <para>The <indexterm><primary>lisp
          kernel</primary></indexterm> <glossterm linkend="lisp_kernel">Lisp
        kernel</glossterm> is a C program with a fair amount of
      platform-specific assembly language code. Its basic job is to
      map a lisp heap image into memory, transfer control to some
      compiled lisp code that the image contains, handle any
      exceptions that occur during the execution of that lisp code,
      and provide various other forms of runtime support for that
      code. &CCL; uses different filenames to name the lisp kernel
      files on different platforms; see
      <xref linkend="Platform-specific-filename-conventions"/>.</para>

    <para>A <indexterm><primary>heap
          image</primary></indexterm> <glossterm linkend="lisp_image">heap
        image</glossterm> is a file that can be quickly mapped into a
      process's address space. Conceptually, it's not too different
      from an executable file or shared library in the OS's native
      format (ELF or Mach-O/dyld format); for historical reasons,
      &CCL;'s own heap images are in their own (fairly simple)
      format. The term <literal>full heap image</literal> refers to a
      heap image file that contains all of the code and data that
      comprise &CCL;. &CCL; uses different filenames to name the
      standard full heap image files on different platforms; see
      <xref linkend="Platform-specific-filename-conventions"/>.</para>

    <para>A <indexterm><primary>bootstrapping
          image</primary></indexterm> bootstrapping image is a minimal
      heap image used in the process of building &CCL; itself.  The
      bootstrapping image contains just enough code to load the rest
      of &CCL; from fasl files.  It may help to think of the
      bootstrapping image as the egg and the full heap image as the
      chicken. &CCL; uses different filenames to name the standard
      bootstrapping image files on different platforms; see
      <xref linkend="Platform-specific-filename-conventions"/>
      .</para>

    <para>Each supported platform (and possibly a few
      as-yet-unsupported ones) has a uniquely named subdirectory of
      <literal>ccl/lisp-kernel/</literal>; each such
      <indexterm><primary>kernel build directory</primary></indexterm>
      contains a Makefile and may contain some auxiliary files (linker
      scripts, etc.) that are used to build the lisp kernel on a
      particular platform.The platform-specific name of the kernel
      build directory is described in
      <xref linkend="Platform-specific-filename-conventions"/>.</para>

    <!-- ******************************************************** -->
    <sect2 id="filename_conventions">
      <title>Platform-specific filename conventions</title>
      <table id ="Platform-specific-filename-conventions">
	    <title>Platform-specific filename conventions</title>
	    <tgroup cols="6">
	      <thead>
            <row>
              <entry>Platform</entry>
              <entry>kernel</entry>
              <entry>full-image</entry>
              <entry>boot-image</entry>
              <entry>fasl extension</entry>
              <entry>kernel-build directory</entry>
	        </row>
	      </thead>
	      <tbody>
	        <row>
	          <entry>DarwinPPC32</entry>
                  <entry>dppccl</entry>
                  <entry>dppccl.image</entry>
                  <entry>ppc-boot.image</entry>
                  <entry>.dfsl</entry>
                 <entry>darwinppc</entry>
	        </row>
	        <row>
	          <entry>LinuxPPC32</entry>
                  <entry>ppccl</entry>
	          <entry>ppccl.image</entry>
                  <entry>ppc-boot</entry>
                  <entry>.pfsl</entry>
                  <entry>linuxppc</entry>
	        </row>
	        <row>
	          <entry>DarwinPPC64</entry>
	          <entry>dppccl64</entry>
	          <entry>dppccl64.image</entry>
	          <entry>ppc-boot64.image</entry>
	          <entry>.d64fsl</entry>
	          <entry>darwinppc64</entry>
                </row>
                <row>
	          <entry>LinuxPPC64</entry>
                  <entry>ppccl64</entry>
                  <entry>ppccl64.image</entry>
                  <entry>ppc-boot64</entry>
                  <entry>.p64fsl</entry>
                  <entry>linuxppc64</entry>
                </row>
	        <row>
	          <entry>LinuxX8664</entry>
                  <entry>lx86cl64</entry>
                  <entry>lx86cl64.image</entry>
                  <entry>x86-boot64</entry>
                  <entry>.lx64fsl</entry>
                  <entry>linuxx8664</entry>
                </row>
	        <row>
	          <entry>LinuxX8632</entry>
                  <entry>lx86cl</entry>
                  <entry>lx86cl.image</entry>
                  <entry>x86-boot32</entry>
                  <entry>.lx32fsl</entry>
                  <entry>linuxx8632</entry>
               </row>
	       <row>
	          <entry>DarwinX8664</entry>
    	          <entry>dx86cl64</entry>
                  <entry>dx86cl64.image</entry>
                  <entry>x86-boot64.image</entry>
                  <entry>.dx64fsl</entry>
                  <entry>darwinx8664</entry>
               </row>
	       <row>
	          <entry>DarwinX8632</entry>
	          <entry>dx86cl</entry>
                  <entry>dx86cl.image</entry>
                  <entry>x86-boot32.image</entry>
                  <entry>.dx32fsl</entry>
                  <entry>darwinx8632</entry>
               </row>
	       <row>
	          <entry>FreeBSDX8664</entry>
                  <entry>fx86cl64</entry>
                  <entry>fx86cl64.image</entry>
                  <entry>fx86-boot64</entry>
                  <entry>.fx64fsl</entry>
                  <entry>freebsdx8664</entry>
               </row>
	       <row>
                  <entry>FreeBSDX8632</entry>
                  <entry>fx86cl</entry>
                  <entry>fx86cl.image</entry>
                  <entry>fx86-boot32</entry>
                  <entry>.fx32fsl</entry>
                  <entry>freebsdx8632</entry>
               </row>
               <row>
	          <entry>SolarisX64</entry>
                  <entry>sx86cl64</entry>
                  <entry>sx86cl64.image</entry>
                  <entry>sx86-boot64</entry>
                  <entry>.sx64fsl</entry>
                  <entry>solarisx64</entry>
               </row>
	       <row>
		  <entry>SolarisX86</entry>
                  <entry>sx86cl</entry>
                  <entry>sx86cl.image</entry>
                  <entry>sx86-boot32</entry>
                  <entry>.sx32fsl</entry>
                  <entry>solarisx86</entry>
               </row>
               <row>
	          <entry>Win64</entry>
                  <entry>wx86cl64.exe</entry>
                  <entry>sx86cl64.image</entry>
                  <entry>wx86-boot64.image</entry>
                  <entry>.wx64fsl</entry>
                  <entry>win64</entry>
               </row>
	       <row>
		  <entry>Win32</entry>
                  <entry>wx86cl.exe</entry>
                  <entry>wx86cl.image</entry>
                  <entry>wx86-boot32.image</entry>
                  <entry>.wx32fsl</entry>
                  <entry>win32</entry>
               </row>
	      </tbody>
	    </tgroup>
      </table>
    </sect2>
  </sect1>

  <!-- =========================================================== -->
  <sect1 id="Setting-Up-to-Build">
    <title>Setting Up to Build</title>
    <para>At a given time, there are generally two versions of &CCL; that
    you might want to use (and therefore might want to build from
      source):</para>
    <itemizedlist>
      <listitem><para>The released version</para></listitem>
      <listitem><para>The development version, called the "trunk", which
      may contain both interesting new features and interesting new bugs
      </para></listitem>
    </itemizedlist>
    <para>All versions are available for download from svn.clozure.com via
    the Subversion source control system.</para>
    <para>
      For example, to get a released version (1.7 in this example),
      use a command like:
      <programlisting>svn co http://svn.clozure.com/publicsvn/openmcl/release/1.7/<replaceable>xxx</replaceable>/ccl</programlisting>
    </para>
    <para>
      To get the trunk version, use:
      <programlisting>svn co http://svn.clozure.com/publicsvn/openmcl/trunk/<replaceable>xxx</replaceable>/ccl</programlisting>
    </para>
    <para>
      Change the <emphasis><literal>xxx</literal></emphasis> to one of the following names:
      <literal>darwinx86</literal>,
      <literal>linuxx86</literal>,
      <literal>freebsdx86</literal>,
      <literal>solarisx86</literal>,
      <literal>windows</literal>,
      <literal>linuxppc</literal>,
      or 
      <literal>darwinppc</literal>.
    </para>
    <para>
      Tarball distributions of released versions are also available for download via ftp from:
      <ulink url="ftp://clozure.com/pub/release/"/>.
      For additional information about availability of source and distributions see the
      <ulink url="http://trac.clozure.com/ccl/">Clozure CL Trac</ulink>.
    </para>
    <para>Subversion client programs are pre-installed on Mac OS X 10.5 and
      later and are typically either pre-installed or readily available
      on Linux and FreeBSD platforms.  The <ulink url="http://subversion.tigris.org">Subversion web page</ulink> contains links to Subversion client programs
      for many platforms.
      Users of Mac OS X 10.4 or later can also
      install Subversion clients via Fink or MacPorts.
      On Debian Linux (and on related Linux distros such as Ubuntu) run
      <literal>apt-get install subversion</literal> or equivalent in the command-line or interactive package manager.
    </para>
    
  </sect1>

  
  <!-- ******************************************************** -->
  <sect1 id="Kernel-build-prerequisites">
    <title>Kernel Build Prerequisites</title>
        <para>The &CCL; kernel can be built with the following widely
          available tools:</para>
    <itemizedlist>
      <listitem><para>cc or gcc &mdash; the GNU C compiler</para></listitem>
      <listitem><para>ld &mdash; the GNU linker</para></listitem>
      <listitem><para>m4 or gm4 &mdash; the GNU m4 macro processor</para></listitem>
      <listitem><para>as &mdash; the GNU assembler (version 2.10.1 or later)</para></listitem>
      <listitem><para>make &mdash; either GNU make or, on FreeBSD, the default BSD make program</para></listitem>
        </itemizedlist>
        <para> In general, the more recent the versions of those
          tools, the better; some versions of gcc 3.x on Linux have
          difficulty compiling some of the kernel source code correctly
          (so gcc 4.0 should be used, if possible.)  On Mac OS X, the
          versions of the tools distributed with Xcode should work fine;
          on Linux, the versions of the tools installed with the OS (or
          available through its package management system) should work
          fine if they're "recent enough".  On FreeBSD, the installed
          version of the <literal>m4</literal> program doesn't support
          some features that the kernel build process depends on; the
          GNU version of the m4 macroprocessor (called
          <literal>gm4</literal> on FreeBSD) should be installed.
        </para>
        <note><para>In order to build the lisp kernel on Mac OS X
        10.6 Snow Leopard, you must install the optional 10.4
        support when installing Xcode.</para>
        </note>
  </sect1>

  <!-- =========================================================== -->
  <sect1 id="Building-Everything">
    <title>Building Everything</title>
    <para>You now have everything you need.  Start up
      &CCL; with the <literal>-n</literal> or <literal>--no-init</literal>
      option to avoid potential interference from code in your init file,
      and evaluate the following form to bring your Lisp system
      up to date.</para>
    <programlisting>
? (ccl:rebuild-ccl :full t)
    </programlisting>
    <para>That call to the function <literal>rebuild-ccl</literal>
      performs the following steps:</para>
    <itemizedlist>
      <listitem>
	    <para>Deletes all fasl files and other object files in the
	      <literal>ccl</literal> directory tree</para>
	  </listitem>
      <listitem>
	    <para>Runs an external process that does a
	      <literal>make</literal> in the current platform's kernel
	      build directory to create a new kernel.  
             This step can only work if the C compiler and related
             tools are installed; see <xref linkend="Kernel-build-prerequisites"/>. 
           </para>
	  </listitem>
      <listitem>
	    <para>Does <literal>(compile-ccl t)</literal> in the running
	      lisp, to produce a set of fasl files from the &ldquo;higher
	      level&rdquo; lisp sources.</para>
	  </listitem>
      <listitem>
	    <para>Does <literal>(xload-level-0 :force)</literal> in the
	      running lisp, to compile the lisp sources in the
	      &ldquo;ccl:level-0;&rdquo; directory into fasl files and
	      then create a bootstrapping image from those fasl
	      files.</para>
	  </listitem>
      <listitem>
	    <para>Runs another external process, which causes the newly
	      compiled lisp kernel to load the new bootstrapping image.
	      The bootstrapping image then loads the &ldquo;higher
	      level&rdquo; fasl files and a new copy of the platform's
	      full heap image is then saved.</para>
	  </listitem>
    </itemizedlist>
    <para>If all goes well, it'll all happen without user
      intervention and with some simple progress messages.  If
      anything goes wrong during execution of either of the external
      processes, the process output is displayed as part of a lisp
      error message.</para>
    <para><literal>rebuild-ccl</literal> is essentially just a short
      cut for running all the individual steps involved in rebuilding
      the system.  You can also execute these steps individually, as
      described below.</para>
  </sect1>

  <!-- =========================================================== -->
  <sect1 id="Building-the-kernel">
    <title>Building the Kernel</title>
    <para>The Lisp kernel is the executable that you run to use
      Lisp.  It doesn't actually contain the entire Lisp
      implementation; rather, it loads a heap image which contains the
      specifics&mdash;the "library", as it might be called if this was a C
      program.  The kernel also provides runtime support to the heap
      image, such as garbage collection, memory allocation, exception
      handling, and the OS interface.</para>

    <para>The Lisp kernel file has different names on different
      platforms. See
      <xref linkend="Platform-specific-filename-conventions"/>. On all
      platforms the lisp kernel sources reside
      in <literal>ccl/lisp-kernel</literal>.</para>

    <para>This section gives directions on how to rebuild the Lisp
      kernel from its source code.  Most &CCL; users will rarely
      have to do this.  You probably will only need to do it if you are
      attempting to port &CCL; to a new architecture or extend or enhance
      its kernel in some way.  As mentioned above, this step happens
      automatically when you do
      <programlisting>
? (rebuild-ccl :full t)
      </programlisting>
    </para>

    <!-- ******************************************************** -->
    <sect2 id="kernel-build-command">
	  <title>Using "make" to build the lisp kernel</title>
      <para>With those tools in place, do:
        <programlisting>
shell> cd ccl/lisp-kernel/<replaceable>PLATFORM</replaceable>
shell> make
	    </programlisting>
	  </para>
      <para>That'll assemble several assembly language source files,
        compile several C source files, and link
        ../../<replaceable>the kernel</replaceable>.
	  </para>
    </sect2>
  </sect1>

  <!-- =========================================================== -->
  <sect1 id="Building-the-heap-image">
    <title>Building the Heap Image</title>
    <para>The initial heap image is loaded by the Lisp kernel, and
      provides most of the language implementation The heap image
      captures the entire state of a running Lisp (except for external
      resources, such as open files and TCP sockets).  After it is
      loaded, the contents of the new Lisp process's memory are
      exactly the same as those of the old Lisp process when the image
      was created.</para>
    <para>The heap image is how we get around the fact that we can't
      run Lisp code until we have a working Lisp implementation, and
      we can't make our Lisp implementation work until we can run Lisp
      code.  Since the heap image already contains a fully-working
      implementation, all we need to do is load it into memory and
      start using it.</para>
    <para>If you're building a new version of &CCL;, you need to
      build a new heap image.</para>
    <para>(You might also wish to build a heap image if you have a
      large program that is very complicated or time-consuming to
      load, so that you will be able to load it once, save an image,
      and thenceforth never have to load it again. At any time, a heap
      image capturing the entire memory state of a running Lisp can be
      created by calling the function
      <literal>ccl:save-application</literal>.)</para>

    <!-- ******************************************************** -->
    <sect2 id="Development-cycle">
	  <title>Development cycle</title>
      <para>Creating a new &CCL; full heap image consists of the
        following steps:</para>
      <orderedlist>
        <listitem><para>Using your existing &CCL;, create a
            bootstrapping image</para></listitem>
        <listitem><para>Using your existing &CCL;, recompile your
            updated &CCL; sources</para></listitem>
        <listitem><para>Invoke &CCL; with the bootstrapping image
            you just created (rather than with the existing full heap
            image).</para></listitem>
	  </orderedlist>
	  <para>When you invoke &CCL; with the bootstrapping image, it
	    starts up, loads all of the &CCL; fasl files, and saves out a
	    new full heap image.  Voila.  You've created a new heap
	    image.</para>
      <para>A few points worth noting:</para>
	  <itemizedlist>
        <listitem>
	      <para>There's a circular dependency between the full heap
	        image and the bootstrapping image, in that each is used to
	        build the other.</para>
	    </listitem>
        <listitem>
	      <para>There are some minor implementation
	        differences, but the environment in effect after the
	        bootstrapping image has loaded its fasl files is essentially
	        equivalent to the environment provided by the full heap
	        image; the latter loads a lot faster and is easier to
	        distribute, of course.</para>
	    </listitem>
        <listitem>
	      <para>If the full heap image doesn't work (because
	        of an OS compatibilty problem or other bug), it's very likely
	        that the bootstrapping image will suffer the same
	        problems.</para>
	    </listitem>
	  </itemizedlist>
      <para>Given a bootstrapping image and a set of up-to-date fasl
        files, the development cycle usually involves editing lisp
        sources (or updating those sources via <literal>svn update</literal>),
        recompiling modified files, and using the bootstrapping image
        to produce a new heap image.</para>
    </sect2>

    <!-- ******************************************************** -->
    <sect2 id="Generating-a-bootstrapping-image">
      <title>Generating a bootstrapping image</title>
      <para>The bootstrapping image isn't provided in &CCL;
        distributions. It can be built from the source code provided
        in distributions (using a lisp image and kernel provided in
        those distributions) using the procedure described
        below.</para>

      <para>The bootstrapping image is built by invoking a special
        utility inside a running &CCL; heap image to load files
        contained in the <literal>ccl/level-0</literal> directory. The
        bootstrapping image loads several dozen fasl files.  After
        it's done so, it saves a heap image via
        <literal>save-application</literal>. This process is called
        "cross-dumping".</para>

      <para>Given a source distribution, a lisp kernel, and a heap
        image, one can produce a bootstrapping image by first invoking
        &CCL; from the shell:</para>
      <programlisting>
shell&gt; ccl
Welcome to &CCL; .... !
?
	  </programlisting>
	  <para>then calling <literal>ccl:xload-level-0</literal> at the
	    lisp prompt:</para>
	  <programlisting>
? (ccl:xload-level-0)
	  </programlisting>
      <para>This function compiles the lisp sources in the <literal>ccl/level-0</literal>
        directory if they're newer than the corresponding fasl files
        and then loads the resulting fasl files into a simulated lisp
        heap contained in data structures inside the running
        lisp. That simulated heap image is then written to
        disk.</para>
      <para><literal>xload-level-0</literal> should be called
        whenever your existing boot image is out-of-date with respect
        to the source files in <literal>ccl:level-0;</literal>
        &mdash; For example:</para>
      <programlisting>
? (ccl:xload-level-0 :force)
      </programlisting>
      <para>forces recompilation of the level-0 sources.</para>
    </sect2>

    <!-- ******************************************************** -->
    <sect2 id="Generating-fasl-files">
      <title>Generating fasl files</title>
	  <para> Calling:</para>
      <programlisting>
? (ccl:compile-ccl)
	  </programlisting>
	  <para>at the lisp prompt compiles any fasl files that are
	    out-of-date with respect to the corresponding lisp sources;
	    <literal>(ccl:compile-ccl t)</literal> forces
	    recompilation. <literal>ccl:compile-ccl</literal> reloads
	    newly-compiled versions of some files;
	    <literal>ccl:xcompile-ccl</literal> is analogous, but skips
	    this reloading step.</para>
      <para>Unless there are bootstrapping considerations involved, it
        usually doesn't matter whether these files are reloaded after
        they're recompiled.</para>
      <para>Calling <literal>compile-ccl</literal> or
        <literal>xcompile-ccl</literal> in an environment where fasl
        files don't yet exist may produce warnings to that effect
        whenever files are <literal>require</literal>d during
        compilation; those warnings can be safely ignored. Depending
        on the maturity of the &CCL; release, calling
        <literal>compile-ccl</literal> or
        <literal>xcompile-ccl</literal> may also produce several
        warnings about undefined functions, etc. They should be
        cleaned up at some point.</para>
    </sect2>

    <!-- ******************************************************** -->
    <sect2 id="Building-a-full-image-from-a-bootstrapping-image">
	  <title>Building a full image from a bootstrapping image</title>
	  <para>To build a full image from a bootstrapping image, just
	    invoke the kernel with the bootstrapping image as an
	    argument</para>
      <programlisting>
$ cd ccl                        # wherever your ccl directory is
$ ./<replaceable>KERNEL</replaceable> --image-name <replaceable>BOOT_IMAGE</replaceable> --no-init
	  </programlisting>
      <para>Where <replaceable>KERNEL</replaceable> and
        <replaceable>BOOT_IMAGE</replaceable> are the names of
        the kernel and boot image appropriate to the platform you are
        running on.  See <xref linkend="Platform-specific-filename-conventions"/></para>
      <para>That should load a few dozen fasl files (printing a
        message as each file is loaded.) If all of these files
        successfully load, the lisp will print a prompt. You should be
        able to do essentially everything in that environment that you
        can in the environment provided by a "real" heap image. If
        you're confident that things loaded OK, you can save that
        image:</para>
      <programlisting>
? (ccl:save-application "<replaceable>image_name</replaceable>") ; Overwriting the existing heap image
	  </programlisting>
	  <para>Where <replaceable>image_name</replaceable> is the name of
        the full heap image for your platform. See
        <xref linkend="Platform-specific-filename-conventions"/>.</para>
      <para>If things go wrong in the early stages of the loading
        sequence, errors are often difficult to debug; until a fair
        amount of code (CLOS, the CL condition system, streams, the
        reader, the read-eval-print loop) is loaded, it's generally
        not possible for the lisp to report an error.  Errors that
        occur during these early stages ("the cold load") sometimes
        cause the lisp kernel debugger (see ) to be invoked; it's
        primitive, but can sometimes help one to get oriented.</para>
    </sect2>
  </sect1>
</chapter>
