<?xml version='1.0' ?>
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN"
"http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd" [
<!ENTITY % myvars SYSTEM "../variables.ent">
%myvars;
]>
  <chapter id="backup-recovery">
  <title>Backup and Recovery</title>
   <authorblurb>
  <para><phrase role="cvstag">($Id: recovery.xml,v 1.17 2010/12/11 23:36:32 ryang Exp $)</phrase></para>

    <para>By <ulink url="mailto:dhogaza@pacifier.com">Don Baccus</ulink> with additions
      by <ulink url="mailto:joel@aufrecht.org">Joel Aufrecht</ulink></para>
    <para>We will cover some basic backup and recovery strategies.  These are intended to 
    be robust but simple enough to set up.  For a large scale production site you would 
    probably need to create your own backup strategies (in particular full dumps from 
    oracle, while easy to set up, are far from the best solution).
    </para>
  
  <para>There are three basic things which need to be backed up, the database data, the server 
    source tree, and the acs-content-repository (which is in the server source tree).</para>
  <para>
    <figure>
      <title>Backup and Recovery Strategy</title>
      <mediaobject>
        <imageobject>
          <imagedata fileref="images/backup.png" format="PNG" align="center"/>
        </imageobject>
      </mediaobject>
    </figure>
  </para>
   </authorblurb>


  <sect1 id="install-next-backups">
    <title>Backup Strategy</title>
    <para>
      The purpose of backup is to enable recovery.  Backup and
      recovery are always risky; here are some steps that minimize the
      chance recovery is necessary:
    </para>
    <itemizedlist>
      <listitem><para>
      Store everything on a fault-tolerant disk array (RAID 1 or 5
      or better).
      </para></listitem>
      <listitem><para>
      Use battery backup.
      </para></listitem>
      <listitem><para>
      Use more reliable hardware, such as SCSI instead of IDE.
      </para></listitem>
    </itemizedlist>

    <para>These steps improve the chances of successful recovery:</para>
    <itemizedlist>
      <listitem><para>
      Store backups on a third disk on another controller
      </para></listitem>
      <listitem><para>
      Store backups on a different computer on a different network
      in a different physical location.  (Compared to off-line
      backup such as tapes and CDRs, on-line backup is faster and
      more likely to succeed, but requires maintenance of another machine.)
      </para></listitem>
      <listitem><para>
      Plan and configure for recovery from the beginning.
      </para></listitem>
      <listitem><para>
      Test your recovery strategy from time to time.
      </para></listitem>
      <listitem><para>
      Make it easy to maintain and test your recovery strategy, so
      that you are more likely to do it.
      </para></listitem>
    </itemizedlist>

    <para>
      OpenACS installations comprise files and database contents.
      If you follow the reference install and put all files,
      including configuration files, in
      <filename>/var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/</filename>,
      and back up the database nightly to a file in
      <filename>/var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/database-backup</filename>,
      then you can apply standard file-based backup strategies to
      <filename>/var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable></filename>
    </para>
  </sect1>
  
  <sect1 id="snapshot-backup">
    <title>Manual backup and recovery</title>
    
    <para>This section describes how to make a one-time backup and
    restore of the files and database.  This is useful for rolling
    back to known-good versions of a service, such as at initial
    installation and just before an upgrade.  First, you back up the
    database to a file within the file tree.  Then, you back up the
    file tree.  All of the information needed to rebuild the site,
    including the AOLserver config files, is then in tree for regular
    file system backup.</para>

    <orderedlist>
      <listitem>
        <formalpara>
          <title>Back up the database to a file</title>
          <para></para>
        </formalpara>
        <itemizedlist>
          <listitem>
            <formalpara id="oracle-snapshot-backup">
              <title>Oracle</title>
              <para></para>
            </formalpara>
            <itemizedlist>
              <listitem><para> 
              Download the backup script. Save the file <ulink
              url="files/export-oracle.txt">export-oracle.txt</ulink> as
              <filename>/var/tmp/export-oracle.txt</filename>
              </para></listitem>
              
              <listitem><para>
              Login as root. The following commands will install the export script:
              </para> 
              <programlisting>[joeuser ~]$ <userinput>su -</userinput>
[root ~]# <userinput>cp /var/tmp/export-oracle.txt /usr/sbin/export-oracle</userinput>
[root ~]# <userinput>chmod 700 /usr/sbin/export-oracle</userinput></programlisting>
              </listitem>

              <listitem><para>
              Setup the export directory; this is the directory where backups will
              be stored. We recommend the directory
              <filename>/ora8/m02/oracle-exports</filename>.</para>

              <programlisting>[root ~]# <userinput>mkdir <replaceable>/ora8/m02/</replaceable>oracle-exports</userinput>
[root ~]# <userinput>chown oracle:dba <replaceable>/ora8/m02/</replaceable>oracle-exports</userinput>
[root ~]# <userinput>chmod 770 <replaceable>/ora8/m02/</replaceable>oracle-exports</userinput></programlisting>
              </listitem>

              <listitem><para> 
              Now edit
              <filename>/usr/sbin/export-oracle</filename> and
              change the <computeroutput>SERVICE_NAME</computeroutput> and
              <computeroutput>DATABASE_PASSWORD</computeroutput> fields to
              their correct values. If you want to use a directory other than
              <filename>/ora8/m02/oracle-exports</filename>, you
              also need to change the
              <computeroutput>exportdir</computeroutput> setting.
              </para>
              <para>
                Test the export procedure by running the command:
              </para>

              <programlisting>[root ~]# <userinput>/usr/sbin/export-oracle</userinput>
mv: /ora8/m02/oracle-exports/oraexport-service_name.dmp.gz: No such file or directory

Export: Release 8.1.6.1.0 - Production on Sun Jun 11 18:07:45 2000

(c) Copyright 1999 Oracle Corporation.  All rights reserved.

Connected to: Oracle8i Enterprise Edition Release 8.1.6.1.0 - Production
With the Partitioning option
JServer Release 8.1.6.0.0 - Production
Export done in US7ASCII character set and US7ASCII NCHAR character set
  . exporting pre-schema procedural objects and actions
  . exporting foreign function library names for user SERVICE_NAME 
  . exporting object type definitions for user SERVICE_NAME 
  About to export SERVICE_NAME's objects ...
  . exporting database links
  . exporting sequence numbers
  . exporting cluster definitions
  . about to export SERVICE_NAME's tables via Conventional Path ...
  . exporting synonyms
  . exporting views
  . exporting stored procedures
  . exporting operators
  . exporting referential integrity constraints
  . exporting triggers
  . exporting indextypes
  . exporting bitmap, functional and extensible indexes
  . exporting posttables actions
  . exporting snapshots
  . exporting snapshot logs
  . exporting job queues
  . exporting refresh groups and children
  . exporting dimensions
  . exporting post-schema procedural objects and actions
  . exporting statistics
Export terminated successfully without warnings.</programlisting>
              </listitem>
            </itemizedlist>
          </listitem>

          <listitem>
            <formalpara id="postgres-snapshot-backup">
              <title>PostgreSQL</title>
              <para>Create a backup file and verify that it was created and has a reasonable size (several megabytes).</para>
            </formalpara>
            <screen>[root root]# <userinput>su - $OPENACS_SERVICE_NAME</userinput>
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>pg_dump -f /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/database-backup/before_upgrade_to_4.6.dmp <replaceable>$OPENACS_SERVICE_NAME</replaceable></userinput>
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>ls -al /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/database-backup/before_upgrade_to_4.6.dmp </userinput>
-rw-rw-r-x    1 $OPENACS_SERVICE_NAME  $OPENACS_SERVICE_NAME   4005995 Feb 21 18:28 /var/lib/aolserver/$OPENACS_SERVICE_NAME/database-backup/before_upgrade_to_4.6.dmp
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>exit</userinput>
[root root]#
<action>su - $OPENACS_SERVICE_NAME
pg_dump -f /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/database-backup/before_upgrade_to_4.6.dmp <replaceable>openacs-dev</replaceable>
ls -al /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/database-backup/before_upgrade_to_4.6.dmp
exit</action></screen>
          </listitem>
        </itemizedlist>
      </listitem>
      <listitem id="backup-file-system">
        <formalpara>
          <title>Back up the file system</title>
          <para>Back up all of the files in the service, including the
          database backup file but excluding the auto-generated
          <filename>supervise</filename> directory, which is
          unneccesary and has complicated permissions.  </para>
        </formalpara>
        <para>In the tar command,</para>
        <itemizedlist>
          <listitem>
            <para><computeroutput>c</computeroutput> create a
            new tar archive</para>
          </listitem>
          <listitem>
            <para><computeroutput>p</computeroutput> preserves permissions.</para>
          </listitem>
          <listitem>
            <para><computeroutput>s</computeroutput> preserves file sort order</para>
          </listitem>
          <listitem>
            <para><computeroutput>z</computeroutput> compresses the output with gzip.</para>
          </listitem>
          <listitem>
            <para>The <computeroutput>--exclude</computeroutput> clauses skips some daemontools files that
            are owned by root and thus cannot be backed up by the
            service owner.  These files are autogenerated and we don't
            break anything by omitting them.</para>
          </listitem>
          <listitem>
            <para>The <computeroutput>--file</computeroutput> clause
            specifies the name of the output file to be generated; we
            manually add the correct extensions.</para>
          </listitem>
          <listitem>
            <para>The last clause,
            <filename>/var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/</filename>,
            specifies the starting point for backup.  Tar defaults to
            recursive backup.</para>
          </listitem>
        </itemizedlist>
        <screen>[root root]# <userinput>su - <replaceable>$OPENACS_SERVICE_NAME</replaceable></userinput>
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>tar -cpsz --exclude /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/daemontools/supervise \
   --file /var/tmp/<replaceable>$OPENACS_SERVICE_NAME</replaceable>-backup.tar.gz /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/</userinput>
tar: Removing leading `/' from member names
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$</screen>
      </listitem>
      <listitem>
        <formalpara>
          <title>Suffer a catastrophic failure on your production system</title>
          <para>(We'll simulate this step)</para>
        </formalpara>
        <screen>[root root]# <userinput>svc -d /service/$OPENACS_SERVICE_NAME</userinput>
[root root]# <userinput>mv /var/lib/aolserver/$OPENACS_SERVICE_NAME/ /var/lib/aolserver/$OPENACS_SERVICE_NAME.lost</userinput>
[root root]#<userinput> rm /service/$OPENACS_SERVICE_NAME</userinput>
rm: remove symbolic link `/service/$OPENACS_SERVICE_NAME'? y
[root root]# <userinput>ps -auxw | grep $OPENACS_SERVICE_NAME</userinput>
root      1496  0.0  0.0  1312  252 ?        S    16:58   0:00 supervise $OPENACS_SERVICE_NAME
[root root]#<userinput> kill<replaceable> 1496</replaceable></userinput>
[root root]# <userinput>ps -auxw | grep $OPENACS_SERVICE_NAME</userinput>
[root root]# <userinput>su - postgres</userinput>
[postgres pgsql]$ <userinput>dropdb $OPENACS_SERVICE_NAME</userinput>
DROP DATABASE
[postgres pgsql]$ <userinput>dropuser $OPENACS_SERVICE_NAME</userinput>
DROP USER
[postgres pgsql]$ <userinput>exit</userinput>
logout
[root root]#</screen>
      </listitem>
      <listitem>
        <formalpara id="recovery">
          <title>Recovery</title>
          <para></para>
        </formalpara>
        <orderedlist>
          <listitem>
            <para>Restore the operating system and required software.
            You can do this with standard backup processes or by
            keeping copies of the install material (OS CDs, OpenACS
            tarball and supporting software) and repeating the install
            guide.  Recreate the service user (<replaceable>$OPENACS_SERVICE_NAME</replaceable>).</para>
          </listitem>
          <listitem>
            <para>Restore the OpenACS files and database backup file.</para>
            <screen>[root root]# <userinput>su - <replaceable>$OPENACS_SERVICE_NAME</replaceable></userinput>
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>cd /var/lib/aolserver</userinput>
[$OPENACS_SERVICE_NAME aolserver]$<userinput> tar xzf /var/tmp/$OPENACS_SERVICE_NAME-backup.tar.gz</userinput>
[$OPENACS_SERVICE_NAME aolserver]$ <userinput>chmod -R 775 <replaceable>$OPENACS_SERVICE_NAME</replaceable></userinput>
[$OPENACS_SERVICE_NAME aolserver]$ <userinput>chown -R <replaceable>$OPENACS_SERVICE_NAME.web</replaceable> <replaceable>$OPENACS_SERVICE_NAME</replaceable></userinput></screen>
          </listitem>
          <listitem>
            <para>Restore the database</para>
            <itemizedlist>
              <listitem>
                <formalpara>
                  <title>Oracle</title>
                  <para></para>
                </formalpara>
                <orderedlist>
                  <listitem>
                    <para>Set up a clean Oracle database user and
                    tablespace with the same names as the ones exported from (<link
                    linkend="install-openacs-prepare-oracle">more information</link>).</para>
                  </listitem>
                  <listitem>
                    <para>Invoke the import command</para>
                    <screen><action>imp <replaceable>$OPENACS_SERVICE_NAME</replaceable>/<replaceable>$OPENACS_SERVICE_NAME</replaceable> FILE=/var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/database-backup/nighty_backup.dmp FULL=Y</action></screen>
                  </listitem>
                </orderedlist>
              </listitem>
              <listitem>
                <formalpara id="restore-postgres">
                  <title>Postgres</title>
                <para>If the database user does not already exist, create it.</para>
</formalpara>
                <screen>[root root]# <userinput>su - postgres</userinput>
[postgres ~]$ <userinput>createuser <replaceable>$OPENACS_SERVICE_NAME</replaceable></userinput>
Shall the new user be allowed to create databases? (y/n) <userinput>y</userinput>
Shall the new user be allowed to create more new users? (y/n) <userinput>y</userinput>
CREATE USER
[postgres ~]$ <userinput>exit</userinput>
</screen>
                <para>Because of a bug in Postgres backup-recovery, database objects are not guaranteed to be created in the right order.  In practice, running the OpenACS initialization script is always sufficient to create any out-of-order database objects.  Next, restore the database from the dump file.  The restoration will show some error messages at the beginning for objects that were pre-created from the OpenACS initialization script, which can be ignored.</para>

<screen>[root root]# <userinput>su - <replaceable>$OPENACS_SERVICE_NAME</replaceable></userinput>
[$OPENACS_SERVICE_NAME ~]$ <userinput>createdb <replaceable>$OPENACS_SERVICE_NAME</replaceable></userinput>
CREATE DATABASE
[$OPENACS_SERVICE_NAME ~]$<userinput> psql -f /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/packages/acs-kernel/sql/postgresql/postgresql.sql <replaceable>$OPENACS_SERVICE_NAME</replaceable></userinput>
<emphasis>(many lines omitted)</emphasis>
[$OPENACS_SERVICE_NAME ~]$ <userinput>psql <replaceable>$OPENACS_SERVICE_NAME</replaceable> &lt; /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/database-backup/<replaceable>database-backup.dmp</replaceable></userinput>
<emphasis>(many lines omitted)</emphasis>
[$OPENACS_SERVICE_NAME ~]$ <userinput>exit</userinput>
[postgres ~]$ <userinput>exit</userinput>
logout</screen>
              </listitem>
            </itemizedlist>
          </listitem>
          <listitem>
            <para>Activate the service</para>
            <screen>[root root]# <userinput>ln -s /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/daemontools /service/<replaceable>$OPENACS_SERVICE_NAME</replaceable></userinput>
[root root]# <userinput>sleep 10</userinput>
[root root]# <userinput>svgroup web /service/<replaceable>$OPENACS_SERVICE_NAME</replaceable></userinput></screen>
          </listitem>
        </orderedlist>
      </listitem>
    </orderedlist>
  </sect1>

  <sect1 id="automated-backup">
    <title>Automated Backup</title>
    <para>The recommended backup strategy for a production sit is to use an automated script which first backs up the database to a file in <filename>/var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/database-backup</filename> and then backs up all of <filename>/var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable></filename> to a single zip file, and then copies that zip file to another computer.</para>
    <orderedlist>
      <listitem>
        <para>Make sure that the manual backup process described above works.</para>
      </listitem>
      <listitem>
        <para>Customize the default backup script.  Edit <filename>/var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/backup.sh</filename> with your specific parameters.</para>
      </listitem>
      <listitem>
        <para>
        Make sure the file is executable:</para>
<programlisting>chmod +x backup.sh</programlisting>
      </listitem>
      <listitem>
        <para>
        Set this file to run automatically by adding a line to root's crontab. (Typically, with <computeroutput>export EDITOR=emacs; crontab -e</computeroutput>.)  This example runs the backup script at 1:30 am every day.</para>
        <programlisting>30 1 * * *        sh /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>/etc/backup.sh</programlisting>
      </listitem>
    </orderedlist>

  </sect1>
  <sect1 id="backups-with-cvs">
    <title>Using CVS for backup-recovery</title>
    <para>CVS-only backup is often appropriate for development sites.  If you are already using CVS and your data is not important, you probably don't
      need to do anything to back up your files.  Just make
      sure that your current work is checked into the system.
      You can then roll back based on date - note the
      current system time, down to the minute.  For maximum
      safety, you can apply a tag to your current
      files.  You will still need to back up your database.</para>
    <para>  Note that, if you did the CVS options in this document, the <filename>/var/lib/aolserver/$OPENACS_SERVICE_NAME/etc</filename> directory is not included in cvs and you may want to add it.</para>
    <screen>[root root]# <userinput>su - $OPENACS_SERVICE_NAME</userinput>
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>cd /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable></userinput>
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>cvs commit -m "last-minute commits before upgrade to 4.6"</userinput>
cvs commit: Examining .
cvs commit: Examining bin
<emphasis>(many lines omitted)</emphasis>
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>cvs tag before_upgrade_to_4_6</userinput>
cvs server: Tagging bin
T bin/acs-4-0-publish.sh
T bin/ad-context-server.pl
(many lines omitted)
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>exit</userinput>
[root root]# 
<action>su - $OPENACS_SERVICE_NAME
cd /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>
cvs commit -m "last-minute commits before upgrade to 4.6"
cvs tag before_upgrade_to_4_6
exit</action></screen>
    <para>To restore files from a cvs tag such as the one used above:</para>
    <screen>[root root]# <userinput>su - $OPENACS_SERVICE_NAME</userinput>
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>cd /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable></userinput>
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>cvs up -r current</userinput>
[$OPENACS_SERVICE_NAME $OPENACS_SERVICE_NAME]$ <userinput>exit</userinput>
<action>su - $OPENACS_SERVICE_NAME
cd /var/lib/aolserver/<replaceable>$OPENACS_SERVICE_NAME</replaceable>
cvs up -r current</action></screen>

  </sect1>

</chapter>
