
Title:	How to generate Win32 libraries

Author:	David N. Gray

Date:	September 26, 1995; updated October 5 and December 14, 1995
	April 7, 1997 - a few clarifications and minor updates.
	March 11, 1998 - update for the switch to the new header files.
	April 2, 1998 - switch from subset to full API.

Status:	describes work previously done

---------------------------

This document explains how the Dylan FFI declarations were generated for
the Win32-common library and the other "win32-*" libraries that use it.
(See file "/u/dylan/doc/lib/win32-library.text" for the user-level
specification of these libraries.)

The ".dylan" files in the libraries are of three kinds:

  * interface files
    Each of these was automatically generated from one (or more) of the
    Microsoft ".h" files.  These files can be easily recognized because
    they begin with a comment of the form:
      // This file is automatically generated from "foo.h"; do not edit.

  * library and module definition files
    Each of these is automatically generated from a corresponding
    ".src" file by inserting the `export' lists from "*.exp" files that are
    generated as a side-effect of translating the header files.

  * other
    There are a few ".dylan" files that were hand-coded; these include
    "first.dylan", and "more*.dylan".  Generally these implement
    equivalents for C macros that couldn't be mechanically translated.

Since the Win32 libraries are needed before Collage will be available, a
different approach has been taken to converting the ".h" files to Dylan FFI
declarations.  This is a more ad-hoc approach that is sufficient to handle
particular patterns of usage, but is not intended to be as general purpose
as Collage will need to be.  

The converter consists of the following components (in directory
"dylan/lib/win32-common" unless otherwise indicated):

  * A general-purpose public-domain text processing utility named "gema".
    (Version 1.2 or later is required.)  Lacking a better place, it is
    currently installed as "/u/dylan/local/gema" for SparcStations.
    There is also a Windows NT/Windows 95 image on host "fog" in
    file "\users\gray\NTutil\gema.exe".  For documentation of this program,
    see URL: http://webhost.functionalobjects.com/~gray/gema-intro.html 
    If you really want hard-copy, there are PostScript versions in file:
    zephyr.menlo:/u/gray/ftp/gema-1-3/postscript.tar.gz
    The source code has been placed under Hope in directory
    "~dylan/tools/gema/".  The program is available for public distribution
    by anonymous FTP from "ftp://ftp.functionalobjects.com/pub/gema/".
    There is also a WinHelp version of the documentation available there.

  * Pattern files "patterns.pat", "winpat.pat", and "wincom.pat", which
    define the text transformation rules to be applied by "gema" in order
    to read a ".h" file and emit corresponding Dylan FFI declarations and a
    "*.exp" file of exports.  Some of the other libraries have their own
    pattern files of additional special case rules.

  * Text files listing C names which are not to be translated.
    These files are read by "gema" under the direction of rules in the
    "patterns.pat" file (using the `-exclude' option).  Names may be
    separated by either commas or whitespace, comments may be included
    (using either "//" or "/*...*/"), and "*" may be used for wildcarded
    names.   These lists were hand-generated from information in the Win32
    SDK online documentation and MSDN library.

    . "obsolete-names.text" is a list of C identifiers whose
      definitions are not to be translated because they are known to be
      obsolete.  This includes anything that exists only for compatibility
      with Win16.

    . "nt-only.text" lists names that are only supported on Windows NT,
      not on Windows 95.

    . "w95-only.text" lists names that are only supported on Windows 95,
       not on Windows NT.

    . "ce-only.text" lists names that are only supported on Windows CE,
      not on Windows 95 or Windows NT.

    (The exclusion of names that are new for Windows 98 and/or NT 5.0 is
    handled by the pattern files, usually by processing of the conditional
    directives, and occasionally by special rules for individual unguarded
    declarations.)

  * File "appnames.text" is the list of C identifiers naming functions,
    types, and constants that were included in the initial subset libraries
    (before April 1998).  The syntax is the same as for the exclude files
    above.  Additional features can be added to the subset library by
    adding their names to this list and re-generating the Dylan files.
    This list was originally generated by a semi-mechanical collection of
    names appearing in some sample Windows programs; consequently it may also
    include some harmless junk which are not API names.  The list happens
    to be sorted as a result of merging and eliminating duplicates, but it
    is not required to be in any particular order.  For names where Foo is
    an alias for either FooA or FooW, you can just say Foo here instead of
    FooA.  An entry such as "FOO_BAR" (all upper-case letters and an
    underscore) will cause inclusion of all constants whose names begin
    with "FOO_".  An entry may include an asterisk to denote a wildcard;
    for example, use "Foo*" to include all names beginning with "Foo".

  * File "include.pat" is used in the generation of the library definition
    files.  It is a pattern file that tells "gema" how to merge the "*.src"
    and "*.exp" files.

  * File "make-sources" in each of the library directories is the make file
    that controls generation of the ".dylan" files.  (It isn't called
    "Makefile" here because a file with that name is generated by the C
    back-end compiler.)  For example:

       make -f make-sources

    will re-generate the sources for the library in the current directory.
    This currently assumes that the Microsoft header files can be found in
    "../microsoft-headers/", but that can be overridden like:
 
       make MSH=header-dir/ -f make-sources ...

  * File "sources.mak" in "win32-common" is included by each of the
    "make-sources" files; it is simply the portion of the make file that is
    the same for all of the libraries.

The choice of generating the subset or full API libraries instead of the
"appnames.text" subset, is done by editing "sources.mak" to comment out
one of the following sets of lines:
  OPTIONS=$(SUBSET-OPTIONS)
  OPTDEP=$(COMDIR)/appnames.text
or:
  OPTIONS=$(FULL-OPTIONS)
  OPTDEP=$(COMDIR)/obsolete-names.text $(COMDIR)/w95-only.text \
   $(COMDIR)/nt-only.text $(COMDIR)/ce-only.text


The header files used are the "WIN*.H" files taken from the Microsoft
Developer Network distribution of the Win32 SDK (typically installed in
"\MSTOOLS\INCLUDE\").  The 1997 beta release of Functional Developer was
based on the header files from the August 1995 distribution of the Win32
SDK with the final Windows 95 release.  These differ only in insignificant
ways from the files in the Visual C++ 2.2 update (installed in
"\MSVC20\INCLUDE\").  These same files, which carry the date "7/11/95", can
be found on the 1997 MSDN CD-ROM number 6, labelled "Win32 SDK (for Windows
95 and Windows NT Worksation 3.51) July, 1996".  As of March 17, 1998, the
1.0 release is based on the header files from the October 1997 distribution
of the Platform SDK for Windows 95 and NT 4.0; these files are not
significantly different from those in the January 1998 distribution.  They
are different from (newer than) the headers supplied with Visual C++ 5.0.

The Microsoft header files are not archived in HOPE because that would
appear to violate Microsoft's license agreement.  Each user needs to have
their own licensed copy.

The Microsoft header files were being kept in a separate directory from the
Dylan sources because when the old Dylan compiler (SCC) was run, it would
create new .h files which would overwrite the originals.  However, that is
not a problem with the new compiler (DFMC).

A typical rule in the Makefile is:

  wingdi.dylan: $(MSH)wingdi.h patterns.pat common-slots.pat appnames.text
	gema -f patterns.pat -f common-slots.pat -module Win32-GDI \
		-out $@ -only appnames.text -exports wingdi.exp $(MSH)wingdi.h

The command line options "-module", "-only", "-exports", and "-exclude" are
actually defined in the "patterns.pat" file (see the lines beginning
"ARGV:"), so don't look for them in the "gema" documentation.  The "-f"
options direct "gema" to load rules from the files "patterns.pat" and
"common-slots.pat"; the latter file is automatically generated by the
Makefile and simply specifies structure slot names that are imported
instead of exported.  The "-module" option specifies what to put in the
"module:" line at the beginning of the generated file.  The "-out" option
specifies the output file as being the make target, namely "wingdi.dylan"
in this case.  The "-only" option names the file telling which names are to
be included; alternatively, the "-exclude" option can be used for names to
not be included.  There is a second output file generated which is the
`export' lists that need to be inserted into the module definition; here
the "-exports" option is used to specify that this file will be named
"wingdi.exp".  Finally, the file that doesn't have any option before it
(although "-in" could be used to make it explicit) is the input file, in
this case, "wingdi.h".

So if you want to build your own extended version of one of the libraries
"win32-user", "win32-kernel", or "win32-gdi", you need to check out a copy
of both that directory and "win32-common", and put the Microsoft headers in
another sibling directory.  Then edit "appnames.text" to add the names of
whatever additional features you need (remember, this is the C identifier,
not the resulting Dylan identifier) and (on a SparcStation) run "make -f
make-sources" to generate the Dylan source files.  The process may need to
be repeated if when compiling the library, you find that the functions you
added require data types or macros that weren't already included.  There
may be a few warning messages written during the conversion about things
that are not yet supported, but you don't need to worry about them unless
"gema" terminates with a non-zero exit code or the warnings are in features
that you really want to use.  Note though that currently there are warnings
only for a few low-level oddities (such as bit fields in a structure) but
that top-level forms that don't fit any pattern are silently ignored.
Thus, for example, if one of the names you wanted is actually defined as a
non-trivial macro, you won't get it.

For example, you want to add the function 'ScreenToClient' to Win32-User.
You should:
 - claim both Win32-Common and Win32-User
 - add 'ScreenToClient' to the bottom of "appnames.text"
 - "cd" to win32-common, and do 'make -f make-sources'.
   The first time, you may need to do 'make -f make-sources clean' first in
   order to generate some auxiliary files that are not in HOPE.
 - "cd" to win32-user, and do 'make -f make-sources'
 - it's best to be paranoid -- do a "hope diff" to verify that the expected
   changes took place.  (Note that "gema" renames the original output file
   with a ".bak" suffix, so you can also compare to that for a quick check
   of what the last "make" did.)
 - check back in both compounds
 - abandon the stray claims you probably have

Note that "win32-common" and other libraries such as "win32-dialog",
"win32-controls", and "win32-version" 
do not use the "appnames.text" file.  If you need something that is missing
from one of those libraries, some tweaking of the pattern files will be
needed.  Generally, "wincom.pat" controls what is excluded from
"win32-common".  Any other omissions from these libraries would be
things that the existing patterns fail to recognize rather than any
deliberate subsetting.

Some quick hints for reading the pattern files:
 
  "!" introduces a comment.
 
  Lines beginning with "@" are used to set option switches and initialize
  variables. 

  The rest of the lines are transformation rules of the form
	pattern=replacement 
  or
	domain:pattern=replacement 
  Within the designated domain (the default is the top-level of the input
  file), input file text that matches the pattern is replaced in the output
  file by the result of evaluating what follows the "=".

  Angle brackets ("<...>") denote variable syntactic elements, like in BNF.

  @foo{...} is a function call.

  For example, the following rule handles integer constants:

    \L\#define <relevant-constant> <number>\W<endline>=\
	\Ndefine constant @export{\$$1} @tab{53}\=@wrap{\s@right{4;$2}\;}\n

  This says (glossing over minor details) that if the input contains
  `#define' followed by a suitable identifier (according to the "-only" or
  "-exclude" option) and a numeric constant, then a corresponding `define
  constant' is emitted.  `$1' is the value of the `<relevant-constant>' and
  `$2' is the converted value of the `<number>'.  Rules prefixed by
  `number:' define the recognition and conversion of a `<number>'.  The
  `@export' function converts the name to Dylan syntax (replacing "_" with
   "-") and has the side-effect of adding the name to the module's
  export list, and the other functions are just for aligning the spacing in
  the output.

  See the "gema" documentation for a complete explanation of the notation
  used.


