<?xml version='1.0' ?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
               "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd" [
<!ENTITY % myvars SYSTEM "../variables.ent">
%myvars;
]>
<chapter id="maintenance-web">
  <title>Production Environments</title>
    <authorblurb>
      <para>by <ulink url="mailto:joel@aufrecht.org">Joel Aufrecht</ulink></para>

    </authorblurb>
    
    <sect1 id="install-openacs-keepalive" xreflabel="Keep AOLserver alive">
	<title>Starting and Stopping an OpenACS instance.</title> 

    <para>The simplest way to start and stop and OpenACS site is to run the startup shell script provided, <computeroutput>/var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/daemontools/run</computeroutput>.  This runs as a regular task, and logs to the logfile.  To stop the site, kill the script.</para>
    <para>A more stable way to run OpenACS is with a "keepalive" mechanism of some sort, so that whenever the server halts or is stopped for a reset, it restarts automatically.  This is recommended for development and production servers.</para>
      <para>The Reference Platform uses Daemontools to control AOLserver.  A simpler method, using <computeroutput>init</computeroutput>, is <link linkend="install-openacs-inittab">here</link>.</para>
      <orderedlist>
        <listitem>
          <para>Daemontools must already be installed.  If not, <link linkend="install-daemontools">install it</link>.</para>
        </listitem>
          <listitem>
            <para>Each service controlled by daemontools must have a
            directory in <computeroutput>/service</computeroutput>.  That
            directory must have a file called
            <computeroutput>run</computeroutput>.  It works like this:</para>
            <itemizedlist>
              <listitem><para>The <computeroutput>init</computeroutput> program starts every
              time the computer is booted. </para></listitem>
              <listitem><para>A line in <computeroutput>init</computeroutput>'s configuration
              file, <computeroutput>/etc/inittab</computeroutput>, tells init to
              run, and to restart if necessary,
              <computeroutput>svscanboot</computeroutput>.</para></listitem>
              <listitem><para><computeroutput>svscanboot</computeroutput> checks
              the directory <computeroutput>/service</computeroutput>
              every few seconds.</para>
              </listitem>
              <listitem><para>If it sees a subdirectory there, it
              looks for a file in the subdirectory called
              <computeroutput>run</computeroutput>.  If it finds a run file, it creates a <computeroutput>supervise</computeroutput> process</para>
              </listitem>
              <listitem>
                <para><computeroutput>supervise </computeroutput> executes the run script.  Whenever the run script stops, <computeroutput>supervise</computeroutput> executes it again.  It also creates additional control files in the same directory. </para>
              </listitem>
            </itemizedlist>
            <para>Hence, the AOLserver
	  instance for your development server is started by the file
	  <computeroutput>/service/$OPENACS_SERVICE_NAME/run</computeroutput>.
	  But we use a symlink to make it easier to add and remove
	  stuff from the <computeroutput>/service</computeroutput>, so
	  the actual location is
          <computeroutput>/var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>etc/daemontools/run</computeroutput>.</para>

          <para>Daemontools creates additional files and directories to track status and
        log.  A daemontools directory is included in the OpenACS tarball at
        <computeroutput>/var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/daemontools</computeroutput>.  To use it, first ill any existing AOLserver instances.  As root, link the <computeroutput>daemontools</computeroutput> directory into the <computeroutput>/service</computeroutput> directory.  Daemontools' <computeroutput>svscan</computeroutput> process checks this directory every five seconds, and will quickly execute <computeroutput>run</computeroutput>.</para>
            
            <screen>[$OPENACS_SERVICE_NAME etc]$ <userinput>killall nsd</userinput>
nsd: no process killed
[$OPENACS_SERVICE_NAME etc]$ <userinput>emacs /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/daemontools/run</userinput>
[$OPENACS_SERVICE_NAME etc]$ <userinput>exit</userinput>

[root root]# <userinput>ln -s /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/daemontools/ /service/<replaceable>$OPENACS_SERVICE_NAME</replaceable></userinput></screen>
            <para>Verify that AOLserver is running.</para>
            <screen>[root root]#<userinput> ps -auxw | grep nsd</userinput>
<replaceable>$OPENACS_SERVICE_NAME</replaceable>   5562 14.4  6.2 22436 15952 ?       S    11:55   0:04 /usr/local/aolserver/bin/nsd -it /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/config.tcl -u serve
root      5582  0.0  0.2  3276  628 pts/0    S    11:55   0:00 grep nsd
[root root]#</screen>
          </listitem>
          <listitem>
            <para>The user <replaceable>$OPENACS_SERVICE_NAME</replaceable> can now control the service <replaceable>$OPENACS_SERVICE_NAME</replaceable> with these commands:</para>
            <itemizedlist>
              <listitem><para>

            <computeroutput>svc -d /service/<replaceable>$OPENACS_SERVICE_NAME</replaceable></computeroutput> -
            Bring the server down

          </para></listitem>
              
              <listitem><para>

            <computeroutput>svc -u /service/<replaceable>$OPENACS_SERVICE_NAME</replaceable></computeroutput> -
            Start the server up and leave it in keepalive mode.

          </para></listitem>
              
              <listitem><para>

            <computeroutput>svc -o /service/<replaceable>$OPENACS_SERVICE_NAME</replaceable></computeroutput> -
            Start the server up once. Do not restart it if it stops.

          </para></listitem>

        <listitem><para>

            <computeroutput>svc -t /service/<replaceable>$OPENACS_SERVICE_NAME</replaceable></computeroutput> -
            Stop and immediately restart the server.

          </para></listitem>

        <listitem><para>
        
            <computeroutput>svc -k /service/<replaceable>$OPENACS_SERVICE_NAME</replaceable></computeroutput> -
            Sends the server a KILL signal. This is like KILL -9. AOLserver
            exits immediately. If svc -t fails to fully kill AOLserver, use
            this option.  This does not take the server out of keepalive mode, so it should still bounce back up immediately.

          </para></listitem>
        </itemizedlist>
      </listitem>

        <listitem>
          <para>Install a script to automate the stopping and starting
          of AOLserver services via daemontools.  You can then restart a service via <computeroutput>restart-aolserver <replaceable>$OPENACS_SERVICE_NAME</replaceable></computeroutput></para>
          <screen>[root root]# <userinput>cp /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/packages/acs-core-docs/www/files/restart-aolserver-daemontools.txt /usr/local/bin/restart-aolserver</userinput>
[root root]# <userinput>chmod 755 /usr/local/bin/restart-aolserver</userinput>
[root root]#</screen>
        </listitem>
      <listitem>
        <para>
        At this point, these commands will work only for the
        <computeroutput>root</computeroutput> user.  Grant permission for the <computeroutput>web</computeroutput> group to use <computeroutput>svc</computeroutput> commands on the <emphasis><replaceable>$OPENACS_SERVICE_NAME</replaceable></emphasis> server.</para>
            <screen>[root root]# <userinput>/usr/local/bin/svgroup web /service/<replaceable>$OPENACS_SERVICE_NAME</replaceable></userinput>
[root root]#</screen>
          </listitem>
          <listitem>
            <para>Verify that the controls work.  You may want to <computeroutput>tail -f /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/log/<replaceable>$OPENACS_SERVICE_NAME</replaceable>-error.log</computeroutput> in another window, so you can see what happens when you type these commands.
      </para>
            <para>

        Most of this information comes from Tom Jackson's <ulink
          url="http://zmbh.com/daemontools-aolserver/daemontools-aolserver.html">AOLserver+Daemontools
          Mini-HOWTO</ulink>.
</para>
        </listitem>
      </orderedlist>

      <table frame='all'><title>How it Works</title>
	<tgroup cols='6' align='center' colsep='1' rowsep='1'>
	  <thead>
	    <row>
	      <entry>Program</entry>
	      <entry>Invoked by this program ...</entry>
	      <entry>... using this file</entry>
	      <entry>Where to find errors</entry>
	      <entry>Log goes to</entry>
	      <entry>Use these commands to control it</entry>
	    </row>
	  </thead>
	  <tbody>
	    <row>
	      <entry>svscanboot
      </entry>
	      <entry>init</entry>
	      <entry>/etc/inittab</entry>
	      <entry>ps -auxw | grep readproctitle</entry>
	      <entry>n/a</entry>
	      <entry></entry>
	    </row>
	    <row>
	      <entry>aolserver</entry>
	      <entry><computeroutput></computeroutput>supervise
(a child of svscanboot)</entry>
	      <entry>/service/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/run</entry>
	      <entry>/var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/log/error.log</entry>
	      <entry>/var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/log/$OPENACS_SERVICE_NAME.log</entry>
            <entry>svc -k /service/<replaceable>$OPENACS_SERVICE_NAME</replaceable></entry>
	    </row>
	    <row>
	      <entry>postgresql</entry>
	      <entry>Redhat init scripts during boot</entry>
	      <entry>/etc/init.d/postgresql</entry>
	      <entry>/usr/local/pgsql/data/server.log</entry>
	      <entry></entry>
	      <entry>service postgresql start (Red Hat), /etc/init.d/postgresql start (Debian)</entry>
	    </row>
	  </tbody>
	</tgroup>
      </table>

    </sect1>
    <sect1 id="install-openacs-inittab" xreflabel="Editing inittab">
      <title>AOLserver keepalive with inittab</title>
      
      <para>This is an alternative method for keeping the AOLserver
		process running.  The recommended method is to <link
		linkend="install-openacs-keepalive">run AOLserver
		supervised</link>.</para>
      
      <para>
		This step should be completed as root. This can break every service
		on your machine, so proceed with caution.
	  </para>
      
      <itemizedlist>
        <listitem><para>
			There are 2 general steps to getting this working. 
		  </para>
          <orderedlist>
            <listitem><para> 
				Install a script called
				<computeroutput>restart-aolserver</computeroutput>. This
				script doesn't actually restart AOLserver - it just kills
				it.  
			  </para></listitem>
            
            <listitem><para>
				Ask the OS to restart our service whenever it's not
				running. We do this by adding a line to
				<computeroutput>/etc/inittab</computeroutput>.
			  </para></listitem>
          </orderedlist>
          
          <para>
			Calling <computeroutput>restart-aolserver</computeroutput>
			kills our service. The OS notices that our service is not
			running, so it automatically restarts it. Thus, calling
			<computeroutput>restart-aolserver</computeroutput> effectively
			restarts our service.  
		  </para></listitem>
        
		<listitem><para> 
			Copy this <ulink
			  url="files/restart-aolserver.txt">file</ulink> into
			<computeroutput>/var/tmp/restart-aolserver.txt</computeroutput>.
		  </para></listitem>

		<listitem><para> 
			This script needs to be SUID-root, which means
			that the script will run as root. This is necessary to ensure
			that the AOLserver processes are killed regardless of who owns
			them. However the script should be executable by the
			<computeroutput>web</computeroutput> group to ensure that the
			users updating the web page can use the script, but that
			general system users cannot run the script. You also need to
			have Perl installed and also a symbolic link to it in
			<computeroutput>/usr/local/bin</computeroutput>. 
		  </para>
		<programlisting>
[joeuser ~]$ su - 
Password: ***********
[root ~]# cp /var/tmp/restart-aolserver.txt /usr/local/bin/restart-aolserver
[root ~]# chown root.web /usr/local/bin/restart-aolserver
[root ~]# chmod 4750 /usr/local/bin/restart-aolserver
[root ~]# ln -s /usr/bin/perl /usr/local/bin/perl
[root ~]# exit</programlisting>
      </listitem>

      <listitem><para> 
			Test the <computeroutput>restart-aolserver</computeroutput>
			script. We'll first kill all running servers to clean the
			slate. Then, we'll start one server and use
			<computeroutput>restart-aolserver</computeroutput> to kill
			it. If it works, then there should be no more servers
			running. You should see the following lines. </para>

		<programlisting>
[joeuser ~]$ killall nsd
nsd: no process killed
[joeuser ~]$ /usr/local/aolserver/bin/nsd-postgres -t ~/var/lib/aolserver/<emphasis>birdnotes</emphasis>/nsd.tcl
[joeuser ~]$ restart-aolserver <emphasis>birdnotes</emphasis>
Killing 23727 
[joeuser ~]$ killall nsd
nsd: no process killed</programlisting>

        <para>
            The number 23727 indicates the process id(s) (PIDs) of the
            processes being killed. It is important that <emphasis
            role="strong">no processes are killed</emphasis> by the second
            call to <computeroutput>killall</computeroutput>. If there are
            processes being killed, it means that the script is not
            working.</para></listitem>

      <listitem><para> 
          Assuming that the <computeroutput>restart-aolserver</computeroutput>
          script worked, login as root and open
          <computeroutput>/etc/inittab</computeroutput> for
          editing. </para>
        <programlisting>
[joeuser ~]$ su -
Password: ************
[root ~]# emacs -nw /etc/inittab</programlisting>
      </listitem>

      <listitem><para> 
            Copy this line into the bottom of the file as a template,
            making sure that the first field
            <computeroutput>nss1</computeroutput> is unique.
          </para>
          <programlisting>
nss1:345:respawn:/usr/local/aolserver/bin/nsd-postgres -i -u nobody -g web -t /home/<emphasis>joeuser</emphasis>/var/lib/aolserver/<emphasis>birdnotes</emphasis>/nsd.tcl</programlisting>
        </listitem>

        <listitem><para>
            <emphasis role="strong">Important:</emphasis> Make sure there is a
            newline at the end of the file. If there is not a newline at
            the end of the file, the system may suffer catastrophic
            failures.  
          </para></listitem>

        <listitem><para>
            Still as root, enter the following command to re-initialize
            <computeroutput>/etc/inittab</computeroutput>. </para>

          <programlisting>
[root ~]# killall nsd    
nsd: no process killed
[root ~]# /sbin/init q</programlisting>        
        </listitem>

        <listitem><para> 
            See if it worked by running the
            <computeroutput>restart-aolserver</computeroutput> script
            again. </para>

          <programlisting>
[root ~]# restart-aolserver <emphasis>birdnotes</emphasis>
Killing 23750</programlisting>
        </listitem>
      </itemizedlist>

      <para>
        If processes were killed, congratulations, your server is now
        automated for startup and shutdown. 
      </para>
    </sect1>
  
  <sect1 id="install-next-add-server">
    <title>Running multiple services on one machine</title>
      <formalpara>
        <title>Services on different ports</title>
        <para>To run a different service on another port but the same
        ip, simply repeat <xref linkend="openacs"/> replacing
        <replaceable>$OPENACS_SERVICE_NAME</replaceable>, and change the
<programlisting>set httpport              8000
set httpsport             8443 </programlisting>
 to different values.</para>
      </formalpara>
      <formalpara>
        <title>Services on different host names</title>
        <para>For example, suppose you want to support
<computeroutput>http://service0.com</computeroutput> and
    <computeroutput>http://bar.com</computeroutput> on the same
    machine.  The easiest way is to assign each one a different ip
    address.  Then you can install two services as above, but with
        different values for
<programlisting>set hostname               [ns_info hostname]
set address                127.0.0.1 </programlisting>
</para>
    </formalpara>

    <para>If you want to install two services with different host
    names sharing the same ip, you'll need nsvhr to redirect requests
    based on the contents of the tcp headers.  See <ulink
    url="http://borkware.com/rants/aolserver-vhosting/">AOLserver
    Virtual Hosting with TCP</ulink> by <ulink url="mailto:markd@borkware.com">markd</ulink>.
</para>

  </sect1>

  <sect1 id="high-avail">
    <title>High Availability/High Performance Configurations</title>
    <para>See also <xref linkend="remote-postgres"/>.</para>
    <figure>
      <title>Multiple-server configuration</title>
      <mediaobject>
        <imageobject>
          <imagedata fileref="images/hpha.png" format="PNG" align="center"/>
        </imageobject>
      </mediaobject>
    </figure>

  </sect1>

    <sect1 id="maintenance-deploy">
  <title>Staged Deployment for Production Networks</title>
    <authorblurb>
  <para><phrase role="cvstag">($Id: maintenance.xml,v 1.30 2010/12/11 23:36:32 ryang Exp $)</phrase></para>      
<para>By <ulink url="mailto:joel@aufrecht.org">Joel Aufrecht</ulink></para>
    </authorblurb>
  <para>This section describes two minimal-risk methods for deploying changes on a production network.  The important characteristics of a safe change deployment include:  (THIS SECTION IN DEVELOPMENT)</para>
  <itemizedlist>
    <listitem>
      <para>Control: You know for sure that the change you are making is the change that you intend to make and is the change that you tested.</para>
    </listitem>
    <listitem>
      <para>Rollback: If anything goes wrong, you can return to the previous working configuration safely and quickly.</para>
    </listitem>
  </itemizedlist>

    <sect2>
      <title>Method 1: Deployment with CVS</title>
      <para>With this method, we control the files on a site via
      CVS. This example uses one developmental server (service0-dev) and one
      production server (service0). Depending on your needs, you can also
      have a staging server for extensive testing before you go
      live. The only way files should move between the server
      instances is via cvs.</para>

      <para>To set up a developmental installation, first set up
      either your developmental installation or your production
      installation, and follow the instructions for committing your
      files to CVS. We'll assume in this example that you set up the
      production server (service0). To set up the developmental instance,
      you then follow the intall guide again, this time creating a new
      user (service0-dev) that you'll use for the new installation. To get
      the files for service0-dev, you check them out from cvs (check out
      service0). </para>

<programlisting>
su - service0-dev
co -d /cvsroot service0
mv service0 /var/lib/aolserver/service0-dev
ln -s /home/service0-dev/web /var/lib/aolserver/service0-dev
emacs web/etc/config.tcl
emacs web/etc/daemontools/run
</programlisting>

      <para>In the config.tcl file, you'll probably want to pay attention
      the rollout support section. That will ensure that email on your
      developmental server will not be sent out to the general world.</para>

      <para>Also, instead of going through the OpenACS online
      installer, you'll actually load live data into your production
      server. </para>

      <para>You can even automate the process of getting live data
      from your production server. Copy something like this to
      /home/service0-dev/bin and put it in service0-dev's crontab to
      run once a night. You'll need to make sure the database backups
      are set up in service0's crontab, and that if the servers are on
      different physical machines, that the database backup is copied
      to the developmental machine once per night.</para>

<programlisting>
/usr/local/bin/svc -d /service/service0-dev
/bin/sleep 60
# this deletes the dev database!
/usr/local/pgsql/bin/dropdb service0-dev
/usr/local/pgsql/bin/createdb -E UNICODE service0-dev
# this is not necessary from Postgres 7.4 on
/usr/local/pgsql/bin/psql -f /var/lib/aolserver/service0-dev/packages/acs-kernel/sql/postgresql/postgresql.sql service0
mv /var/lib/aolserver/service0/database-backup/service0-nightly-backup.dmp.gz /var/lib/aolserver/service0-dev/database-backup/service0-nightly-backup-old.dmp.gz
/bin/gunzip /var/lib/aolserver/service0-dev/database-backup/service0-nightly-backup.dmp.gz
/usr/bin/perl -pi -e "s/^\\connect service0$/\\connect service0-dev/" /var/lib/aolserver/service0-dev/database-backup/service0-nightly-backup.dmp
/usr/local/pgsql/bin/psql service0-dev &lt; /var/lib/aolserver/service0-dev/database-backup/service0-nightly-backup.dmp
/usr/local/bin/svc -u /service/service0-dev
/bin/gzip /var/lib/aolserver/service0-dev/database-backup/service0-nightly-backup-old.dmp
</programlisting>      

      <para>Your developmental server will always have data about a day
      old.</para>

      <para>To make changes on service0-dev:</para>

<programlisting>
1) change the file on service0-dev as desired
2) test the new file
3) commit the file: 
if the file is /var/lib/aolserver/service0-dev/www/index.adp, do: 

cd /var/lib/aolserver/service0-dev/www
cvs diff index.adp (this is optional; it's just a
reality check)
the lines starting &gt; will be added and the lines
starting &lt; will be removed, when you commit
if that looks okay, commit with: 
cvs -m "changing text on front page for February conference" index.adp
the stuff in -m "service0" is a comment visible only from within cvs commands
</programlisting>

        <para>To make these changes take place on service0:</para>

<programlisting>
4) update the file on production:
cd /var/lib/aolserver/service0/www
cvs up -Pd index.adp</programlisting>

      <para>If you make changes that require changes to the database,
      test them out first on service0-dev, using either -create.sql or
      upgrade scripts. Once you've tested them, you then update and
      run the upgrade scripts from the package manager. </para>

<para>The production site can run "HEAD" from cvs.</para>

<para>The drawback to using HEAD as the live code is that you cannot commit new work on the development server without erasing the definition of 'working production code.'  So a better method is to use a tag.  This guarantees that, at any time in the future, you can retrieve exactly the same set of code.  This is useful for both of the characteristics of safe change deployment.  For control, you can use tags to define a body of code, test that code, and then know that what you are deploying is exactly that code.  For rollback, you can use return to the last working tag if the new tag (or new, untagged changes) cause problems.  .... example of using tags to follow ...</para>


    </sect2>
    <sect2>
      <title>Method 2: A/B Deployment</title>
  <para>The approach taken in this section is to always create a new service with the desired changes, running in parallel with the existing site.  This guarantees control, at least at the final step of the process: you know what changes you are about to make because you can see them directly.  It does not, by itself, guarantee the entire control chain.  You need additional measures to make sure that the change you are making is exactly and completely the change you intended to make and tested previously, and nothing more.  Those additional measures typically take the form of source control tags and system version numbers.  The parallel-server approach also guarantees rollback because the original working service is not touched; it is merely set aside.</para>
  <para>This approach can has limitations.  If the database or file system regularly receiving new data, you must interrupt this function or risk losing data in the shuffle.  It also requires extra steps if the database will be affected.</para>
  <sect3>
    <title>Simple A/B Deployment: Database is not changed</title>
    <figure>
      <title>Simple A/B Deployment - Step 1</title>
      <mediaobject>
        <imageobject>
          <imagedata fileref="images/simple-deploy-1.png" format="PNG" align="center"/>
        </imageobject>
      </mediaobject>
    </figure>
     <figure>
      <title>Simple A/B Deployment - Step 2</title>
      <mediaobject>
        <imageobject>
          <imagedata fileref="images/simple-deploy-2.png" format="PNG" align="center"/>
        </imageobject>
      </mediaobject>
    </figure>
     <figure>
      <title>Simple A/B Deployment - Step 3</title>
      <mediaobject>
        <imageobject>
          <imagedata fileref="images/simple-deploy-3.png" format="PNG" align="center"/>
        </imageobject>
      </mediaobject>
    </figure>
  </sect3>
  <sect3>
    <title>Complex A/B Deployment: Database is changed</title>
    <figure>
      <title>Complex A/B Deployment - Step 1</title>
      <mediaobject>
        <imageobject>
          <imagedata fileref="images/complex-deploy-1.png" format="PNG" align="center"/>
        </imageobject>
      </mediaobject>
    </figure>
     <figure>
      <title>Complex A/B Deployment - Step 2</title>
      <mediaobject>
        <imageobject>
          <imagedata fileref="images/complex-deploy-2.png" format="PNG" align="center"/>
        </imageobject>
      </mediaobject>
    </figure>
     <figure>
      <title>Complex A/B Deployment - Step 3</title>
      <mediaobject>
        <imageobject>
          <imagedata fileref="images/complex-deploy-3.png" format="PNG" align="center"/>
        </imageobject>
      </mediaobject>
    </figure>
  </sect3>
  </sect2>
</sect1>
    <sect1 id="install-ssl">
      <title>Installing SSL Support for an OpenACS service</title>
    <para>Debian Users: <computeroutput>apt-get install openssl</computeroutput> before proceeding.</para>
        <orderedlist>
          <listitem>
            <para>Make sure nsopenssl.so is <link linkend="install-nsopenssl">installed</link> for AOLserver.</para>
          </listitem>
          <listitem>
            <para>Uncomment this line from <computeroutput>config.tcl</computeroutput>.</para>
            <programlisting>#ns_param   nsopenssl       ${bindir}/nsopenssl.so
</programlisting>
          </listitem>
          <listitem id="ssl-certificates" xreflabel="Generate ssl certificates">
            <para>Prepare a certificate directory for the service.</para>
            <screen>[$OPENACS_SERVICE_NAME etc]$ <userinput>mkdir /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/certs</userinput>
[$OPENACS_SERVICE_NAME etc]$ <userinput>chmod 700 /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/certs</userinput>
[$OPENACS_SERVICE_NAME etc]$ 
<action>mkdir /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/certs
chmod 700 /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/certs</action></screen>            
          </listitem>
          <listitem>
            <para>It takes two files to support an SSL connection.  The certificate is the public half of the key pair - the server sends the certificate to browser requesting ssl.  The key is the private half of the key pair.  In addition, the certificate must be signed by Certificate Authority or browsers will protest.  Each web browser ships with a built-in list of acceptable Certificate Authorities (CAs) and their keys.  Only a site certificate signed by a known and approved CA will work smoothly.  Any other certificate will cause browsers to produce some messages or block the site.  Unfortunately, getting a site certificate signed by a CA costs money.  In this section, we'll generate an unsigned certificate which will work in most browsers, albeit with pop-up messages.</para>
            <para>Use an OpenSSL perl script to generate a certificate and key.</para>

        <para>
          Debian users: use /usr/lib/ssl/misc/CA.pl instead of /usr/share/ssl/CA
        </para>

        <para>
          Mac OS X users: use perl /System/Library/OpenSSL/misc/CA.pl -newcert instead of /usr/share/ssl/CA
        </para>

            <screen>[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>cd /var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/certs</userinput>
[$OPENACS_SERVICE_NAME certs]$ <userinput>perl /usr/share/ssl/misc/CA -newcert</userinput>
Using configuration from /usr/share/ssl/openssl.cnf
Generating a 1024 bit RSA private key
...++++++
.......++++++
writing new private key to 'newreq.pem'
Enter PEM pass phrase:</screen>
            <para>Enter a pass phrase for the CA certificate.  Then, answer the rest of the questions.  At the end you should see this:</para>
        <screen>Certificate (and private key) is in newreq.pem
[$OPENACS_SERVICE_NAME certs]$</screen>
        <para><computeroutput>newreq.pem</computeroutput> contains our certificate and private key.  The key is protected by a passphrase, which means that we'll have to enter the pass phrase each time the server starts.  This is impractical and unnecessary, so we create an unprotected version of the key.  <emphasis>Security implication</emphasis>: if anyone gets access to the file keyfile.pem, they effectively own the key as much as you do.  Mitigation: don't use this key/cert combo for anything besides providing ssl for the web site.</para>
            <screen>[root misc]# <userinput>openssl rsa -in newreq.pem -out keyfile.pem</userinput>
read RSA key
Enter PEM pass phrase:
writing RSA key
[$OPENACS_SERVICE_NAME certs]$ </screen>
        <para>To create the certificate file, we take the combined file, copy it, and strip out the key.</para>
        <screen>[$OPENACS_SERVICE_NAME certs]$ <userinput>cp newreq.pem certfile.pem</userinput>
[root misc]# <userinput>emacs certfile.pem</userinput></screen>
        <para>Strip out the section that looks like</para>
        <programlisting>-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,F3EDE7CA1B404997
S/Sd2MYA0JVmQuIt5bYowXR1KYKDka1d3DUgtoVTiFepIRUrMkZlCli08mWVjE6T
<emphasis>(11 lines omitted)</emphasis>
1MU24SHLgdTfDJprEdxZOnxajnbxL420xNVc5RRXlJA8Xxhx/HBKTw==
-----END RSA PRIVATE KEY-----</programlisting>
        </listitem>
      <listitem><para>
        If you start up using the etc/daemontools/run script, you will
        need to edit this script to make sure the ports are bound for
        SSL. Details of this are in the run script.
        </para>
      </listitem>
      </orderedlist>
    </sect1>

    <sect1 id="analog-setup">
    <title>Set up Log Analysis Reports</title>
    
    <para>Analog is a program with processes webserver access logs,
      performs DNS lookup, and outputs HTML reports.  Analog should
      <link linkend="analog-install">already be
      installed.</link>  A modified configuration file is included in
      the OpenACS tarball.</para>
    <orderedlist>
      <listitem>
        <screen>[root src]# <userinput>su - $OPENACS_SERVICE_NAME</userinput>
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>cd /var/lib/aolserver/$OPENACS_SERVICE_NAME</userinput>
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>mkdir www/log</userinput>
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>cp -r /usr/share/analog-5.32/images www/log/</userinput>
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <action>
su - $OPENACS_SERVICE_NAME
cd /var/lib/aolserver/$OPENACS_SERVICE_NAME
cp /var/lib/aolserver/$OPENACS_SERVICE_NAME/packages/acs-core-docs/www/files/analog.cfg.txt etc/analog.cfg
mkdir www/log
cp -r /usr/share/analog-5.32/images www/log/</action></screen>
        <para>Edit
<computeroutput>/var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/analog.cfg</computeroutput> and change the variable in <computeroutput>HOSTNAME "[my
organisation]"</computeroutput> to reflect your website title.  If you
don't want the traffic log to be publicly visible, change
<computeroutput>OUTFILE /var/lib/aolserver/$OPENACS_SERVICE_NAME/www/log/traffic.html</computeroutput> to use a private
directory. You'll also need to edit all instances of service0 to your $OPENACS_SERVICE_NAME.</para>
      </listitem>
      <listitem>
        <para>Run it.</para>
        <screen>[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>/usr/share/analog-5.32/analog -G -g/var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/analog.cfg</userinput>
/usr/share/analog-5.32/analog: analog version 5.32/Unix
/usr/share/analog-5.32/analog: Warning F: Failed to open DNS input file
  /home/$OPENACS_SERVICE_NAME/dnscache: ignoring it
  (For help on all errors and warnings, see docs/errors.html)
/usr/share/analog-5.32/analog: Warning R: Turning off empty Search Word Report
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$</screen>
        <para>Verify that it works by browing to <computeroutput>http://yourserver.test:8000/log/traffic.html</computeroutput></para>
      </listitem>
      <listitem>
        <para>Automate this by creating a file in
          <computeroutput>/etc/cron.daily</computeroutput>.</para>
          <screen>[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>exit</userinput>
logout

[root root]# <userinput>emacs /etc/cron.daily/analog</userinput></screen>
        <para>Put this into the file:</para>
        <programlisting>#!/bin/sh

/usr/share/analog-5.32/analog -G -g/var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/analog.cfg</programlisting>
        <screen>[root root]# <userinput>chmod 755 /etc/cron.daily/analog</userinput></screen>
        <para>Test it by running the script.</para>
        <screen>[root root]# <userinput>sh /etc/cron.daily/analog</userinput></screen>
        <para>Browse to <computeroutput>http://<replaceable>yourserver.test</replaceable>/log/traffic.html</computeroutput></para>
      </listitem>
    </orderedlist>
  </sect1>
  <sect1 id="uptime">
    <title>External uptime validation</title>
    <para>The <ulink url="http://uptime.openacs.org/uptime/">OpenACS uptime site</ulink> can monitor your site and send you an email whenever your site fails to respond.  If you test the url <computeroutput>http://<replaceable>yourserver.test</replaceable>/SYSTEM/dbtest.tcl</computeroutput>, you should get back the string <computeroutput>success</computeroutput>.</para>
  </sect1>
<sect1 id="maint-performance">
  <title>Diagnosing Performance Problems</title>
  <itemizedlist>
    <listitem>
      <para>Did performance problems happen overnight, or did they sneak up on
    you? Any clue what caused the performance problems (e.g. loading 20K
    users into .LRN)</para>
    </listitem>
    <listitem>
      <para>Is the file system out of space?  Is the machine swapping to disk constantly?</para>
    </listitem>
    <listitem>
      <para>Isolating and solving database problems.</para>
    <itemizedlist>
      <listitem>
          <para>Without daily internal maintenance, most databases slowly degrade in performance.  For PostGreSQL, see <xref linkend="install-next-nightly-vacuum"/>.  For Oracle, use <computeroutput>exec dbms_stats.gather_schema_stats('SCHEMA_NAME')</computeroutput> (<ulink url="http://www.piskorski.com/docs/oracle.html">Andrew Piskorski's Oracle notes</ulink>).</para>
      </listitem>
      <listitem>
        <para>You can track the exact amount of time each database query on a page takes:</para>
          <orderedlist>
            <listitem>
              <para>Go to <ulink url="/acs-admin/install">Main Site : Site-Wide Administration : Install Software</ulink></para>
              </listitem>
              <listitem>
                <para>Click on "Install New Application" in "Install from OpenACS Repository"</para>
            </listitem>
            <listitem>
              <para>Choose "ACS Developer Support"></para>
            </listitem>
            <listitem>
              <para>After install is complete, restart the server.</para>
            </listitem>
            <listitem>
              <para>Browse to Developer Support, which is automatically mounted at <computeroutput><ulink url="/ds">/ds</ulink></computeroutput>.
              </para>
            </listitem>
            <listitem>
              <para>Turn on Database statistics</para>
            </listitem>
            <listitem>
              <para>Browse directly to a slow page and click "Request Information" at the bottom of the page.</para>
            </listitem>
            <listitem>
              <para>This should return a list of database queries on the page, including the exact query (so it can be cut-paste into psql or oracle) and the time each query took.</para>
              <figure>
                <title>Query Analysis example</title>
              <mediaobject>
                <imageobject>
                  <imagedata fileref="images/query-duration.png" format="PNG"/>
                </imageobject>
              </mediaobject>
              </figure>
            </listitem>
          </orderedlist>
        </listitem>


        <listitem>
          <para>Identify a runaway Oracle query: first, use <userinput>ps aux</userinput> or <userinput>top</userinput> to get the UNIX process ID of a runaway Oracle process.</para>
        <para>Log in to SQL*Plus as the admin:</para>
        <screen>[<replaceable>$OPENACS_SERVICE_NAME</replaceable> ~]$ svrmgrl

Oracle Server Manager Release 3.1.7.0.0 - Production

Copyright (c) 1997, 1999, Oracle Corporation.  All Rights Reserved.

Oracle8i Enterprise Edition Release 8.1.7.3.0 - Production
With the Partitioning option
JServer Release 8.1.7.3.0 - Production

SVRMGR> <userinput>connect internal</userinput>	        
Password:</screen>
        <para>See all of the running queries, and match the UNIX PID:</para>
        <programlisting>select p.spid  -- The UNIX PID
       ,s.sid  ,s.serial#
       ,p.username  as os_user
       ,s.username  ,s.status
       ,p.terminal  ,p.program
  from v$session s  ,v$process p
 where p.addr = s.paddr
 order by s.username ,p.spid ,s.sid ,s.serial# ;</programlisting>
        <para>See the SQL behind the oracle processes:</para>
        <programlisting>select s.username
       ,s.sid  ,s.serial#
       ,sql.sql_text
  from v$session s, v$sqltext sql
 where sql.address    = s.sql_address
   and sql.hash_value = s.sql_hash_value
 --and upper(s.username) like 'USERNAME%'
 order by s.username ,s.sid ,s.serial# ,sql.piece ;</programlisting>
        <para>To kill a troubled process:</para>
        <programlisting>alter system kill session 'SID,SERIAL#';  --substitute values for SID and SERIAL#</programlisting>
        <para>(See <ulink url="http://www.piskorski.com/docs/oracle.html">Andrew Piskorski's Oracle notes</ulink>)</para>
        </listitem>

        <listitem>
          <para>Identify a runaway Postgres query.  First, logging must be enabled in the database.  This imposes a performance penalty and should not be done in normal operation.</para>
          <para>Edit the file <computeroutput>postgresql.conf</computeroutput> - its location depends on the PostGreSQL installation - and change</para>
          <programlisting>#stats_command_string = false</programlisting>
          <para>to</para>
          <programlisting>stats_command_string = true</programlisting>
          <para>Next, connect to postgres (<computeroutput>psql <replaceable>service0</replaceable></computeroutput>) and <computeroutput>select * from pg_stat_activity;</computeroutput>.  Typical output should look like:</para>
          <programlisting>
  datid   |   datname   | procpid | usesysid | usename |  current_query
----------+-------------+---------+----------+---------+-----------------
 64344418 | openacs.org |   14122 |      101 | nsadmin | &lt;IDLE&gt;
 64344418 | openacs.org |   14123 |      101 | nsadmin |
                                                         delete
                                                         from acs_mail_lite_queue
                                                         where message_id = '2478608';
 64344418 | openacs.org |   14124 |      101 | nsadmin | &lt;IDLE&gt;
 64344418 | openacs.org |   14137 |      101 | nsadmin | &lt;IDLE&gt;
 64344418 | openacs.org |   14139 |      101 | nsadmin | &lt;IDLE&gt;
 64344418 | openacs.org |   14309 |      101 | nsadmin | &lt;IDLE&gt;
 64344418 | openacs.org |   14311 |      101 | nsadmin | &lt;IDLE&gt;
 64344418 | openacs.org |   14549 |      101 | nsadmin | &lt;IDLE&gt;
(8 rows)
openacs.org=&gt;</programlisting>
        </listitem>
      </itemizedlist>
    </listitem>
  </itemizedlist>

  <sect2 id="install-oracle-monitoring">
    <title>Creating an appropriate tuning and monitoring environment</title>
    <para>
      The first task is to create an appropriate environment for finding out
      what is going on inside Oracle. Oracle provides Statspack, a package to
      monitor and save the state of the v$ performance views. These reports
      help finding severe problems by exposing summary data about the Oracle
      wait interface, executed queries. You'll find the installation
      instructions in $ORACLE_HOME/rdbms/admin/spdoc.txt. Follow the
      instructions carefully and take periodic snapshots, this way you'll be
      able to look at historical performance data.
    </para>

    <para>
      Also turn on the timed_statistics in your init.ora file, so that
      Statspack reports (and all other Oracle reports) are timed, which makes
      them a lot more meaningful. The overhead of timing data is about 1% per
      Oracle Support information.
    </para>


    <para>
      To be able to get a overview of how Oracle executes a particular query,
      install "autotrace". I usually follow the instructions here <ulink
      url="http://asktom.oracle.com/~tkyte/article1/autotrace.html">http://asktom.oracle.com/~tkyte/article1/autotrace.html</ulink>.
    </para>
    <sect3>
      <title>Make sure, that the Oracle CBO works with adequate statistics</title>
  
      <para>
    The Oracle Cost Based optimizer is a piece of software that tries to find
    the "optimal" execution plan for a given SQL statement. For that it
    estimates the costs of running a SQL query in a particular way (by default
    up to 80.000 permutations are being tested in a Oracle 8i). To get an
    adequate cost estimate, the CBO needs to have adequate statistics. For
    that Oracle supplies the <ulink
    url="http://download-west.oracle.com/docs/cd/B10501_01/appdev.920/a96612/d_stats.htm#999107">dbms_stats
    package</ulink>.
      </para>
    </sect3>

  </sect2>


  </sect1>
</chapter>
   
