Omi

Table of Contents

• What Is Omi?
• Compilation
• Quick Start
• Command Line Options
• Configuration Files and Packages
• Debug Settings
• Mirror Option Settings
• Progress Display
• Excludes and Includes
• Timeout Settings
• Retry and Delay Settings
• Examples
• Troubleshooting

What Is Omi?

It is a software to mirror subtrees of FTP servers. It can be used to make a one-time mirror, and it can be used to update a mirror regularly (for example using cron, see the manual page of crontab).

Compilation

Just type "make" for a short help text. Normally you should not have to edit the Makefile. Note that the compilation is done without optimization, but with "-g", because this is an alpha version.

Quick Start

If you want to mirror the directory tree /pub/foo from the FTP server ftp.bar.org (this host does not exist and is only used as an example) to your local directory /pub/mirror/foo, use the following command line:

omi -s ftp.bar.org -r /pub/foo -l /pub/mirror/foo

Command Line Options

-h

Display a short overview of all options.

-s server

("Server") Specify the remote FTP server. This can be a host name or a numerical IP address. The -s option starts a new "server section", which means that all following options apply to this server only, until another -s option is used.
You can also specify an optional port number here, separated from the host name or IP by a colon (":") or space. The default port number is 21.

-r path

("RemoteDir") Specify the path of the remote directory. It must be an absolute path, i.e. it must start with a slash "/". The -r option starts a new "directory section", which means that all following options apply to this remote directory tree only, until another -r option is used. There can be multiple -r options per -s option.

-l path

("LocalDir") Specify the path of the local directory, which is used as target for the previous -r option. There can be only one -l option per -r option. If there is no -l option for an -r option, the local path is the same as the remote path. If there are multiple -l options for the same -r option, only the last one is used. The local path can be an absolute path or relative to the current working directory.

-b address

("Bind") Specify a local IP address to bind to. The default is to let the operating choose an appropriate address, which is usually the first or primary address configured on the respective interface. This option is useful if you want to access a server that has access restrictions based on the IP address which you're coming from.

-d debug_options

("Debug" or "Verbose") Specify debug options (see section Debug Settings). The default is "report,stats,progress". Specify "help" to get a short overview of all debug options.

-g group

("Group") Specify the group to which all files will be chgrp'ed. It can be a group name or a numerical GID. Specifying an empty string or -1 will disable chgrp (this is the default).

-m mode

("Mode") Specify the modes to be used for local files and directories. They must be octal. The default is "644,755" (the first mode is applied to plain files, the second mode is applied to directories). You can also specify only one number for both (the x-bits will be cleared for plain files, and copied from the r-bits for directories).
Note: If you use the "Permissions" mirror option, this mode setting is ignored, except for the base directory of the mirrored tree (and in cases when the permission modes of the FTP server cannot be parsed correctly).

-u login

("User" or "Login") Specify the login (user name) to use for the FTP server. The default is "ftp".

-e email

("Email") Specify the email address which is used for the FTP login (sent as password). The default is "yourlogin@". Warning: Do not specify a real password here, because it is inherently insecure!

-a account

("Account") Specify additional account information. Some servers require this for the FTP login. There is no default. If the server requires account information, but you didn't specify any, omi will print an error message and exit.

-t time

("Deviation") Set a maximum time deviation (in seconds) for file time comparison (the default is 0, i.e. no deviation). The meaning is this: If a file is present locally and has the same size as the remote file, but a different time stamp, it will normally be retrieved. However, if the time difference is smaller than the maximum deviation (set with the -t option), the file will not be retrieved (but the time stamp of the local file will be set to the time of the remote file). This is useful if you used a different mirror software before which had problems with the time stamps.

-n num

("Newest") Only mirror the num newest files in every directory. num must be a non-negative number. If num is 0 (which is the default), all files are mirrored, no matter how old they are.

-P percent

("RemoveLimit") Specify a removal limit in percent. If the number of files that disappeared from the remote site in a certain directory is greater or equal to this value, no existing files will be removed from the local directory. This is useful if you're mirroring a server that is not reliable (i.e. things disappear and then reappear shortly thereafter). The default is 100, which means that local files will be removed if they disappeared from the remote site, no matter what. If you specify 0, nothing will be removed from the local mirror (this is useful if you want to run an accumulating mirror which never deletes anything, or if you want to delete stuff manually).

-o mirror_options

("Options") Specify mirror options. See the section Mirror Option Settings for details. The default is none. Specify "help" to get a short overview of all mirror options.

-x regex

("Exclude" or "RemoteExclude") Specify an exclude pattern (regular expression). See the section Excludes and Includes for details.

-y regex

("Include" or "RemoteInclude") Specify an include pattern (regular expression). See the section Excludes and Includes for details.

-X regex

("LocalExclude") Specify an exclude pattern for files and/or directories on the local machine.

-Y regex

("LocalInclude") Specify an include pattern for files and/or directories on the local machine.

-T timeout_options

("Timeout") Specify timeout values (e.g. for network accesses). See the section Timeout Settings for details.

-R retry_options

("Retry") Specify retry parameters in case of errors. See the section Retry and Delay Settings for details.

-D delay_options

("Delay") Specify delay parameters for "hard retries". See the section Retry and Delay Settings for details.

-I directory

("IncludeDir" or "PackageDir") Specify an include directory for config files (or so-called "package" files). See the section on configuration files.

-i includefile

("Include" or "Package") Load the specified config file (or so-called "package"). The -i is optional, you can omit it: Any file names on the command line which are not preceded by an option letter are interpreted as packages.

-p pidfile

("PidFile" or "LockFile") Specify the name of a PID file, which is also used as a lockfile. If this file is open and locked by another process, an error message is printed, and the mirror program exits. Note that you can have a global lockfile, a per-server lockfile, and a per-directory lockfile. By default there are no lockfiles.

Configuration Files and Packages

You can store options in configuration files. Each command line option has a corresponding config file entry, and vice versa. Config files consist of config entries and comments. A comment is either an empty line, or a line which starts with a "#" character (optionally preceded by whitespace). The syntax of a config entry is:

key value

There can be an optional ":" or "=" after the key. Any whitespace before the key, between key and value, and after the value is ignored, but the value itself may contain whitespace. If you want a value to contain whitespace at the beginning or at the end, you have to enclose it in double quotes. Double quotes within the value have no special meaning. The value can contain the usual escape sequences: \n (newline), \t (tab), \\ (backslash), \NNN (octal character code), \xNN (hex character code), etc. You can use the shell syntax for including environment variables, for example $HOME or ${HOME}, or -- according to Makefile syntax -- $(HOME) (the latter is depreciated and will not be supported in future versions). You can also use $$ to get the PID (process ID) of the mirroring process. Use \$ if you need a literal "$" character.

The following keys are recognized (note that keys are not case-sensitive), the corresponding command line option is given on the right side:

      Server server              -s server
      RemoteDir path             -r path
      LocalDir path              -l path
      Bind address               -b address
      Debug debug_options        -d debug_options
      Group group                -g group
      Mode mode                  -m mode
      Login login                -u login
      User login                 -u login
      Email email                -e email
      Account acount             -a account
      Deviation time             -t time
      Newest num                 -n num
      RemoveLimit percent        -P percent
      Options options            -o options
      Exclude regex              -x regex
      RemoteExclude regex        -x regex
      Include regex              -y regex
      RemoteInclude regex        -y regex
      LocalExclude regex         -X regex
      LocalInclude regex         -Y regex
      Timeout timeout            -T timeout
      Retry retry_options        -R retry_options
      Delay delay_options        -D delay_options
      IncludeDir directory       -I directory
      PackageDir directory       -I directory
      Include includefile        -i includefile
      Package includefile        -i includefile
      PidFile pidfile            -p pidfile
      LockFile pidfile           -p pidfile

After start-up, omi first reads configuration entries from /usr/local/etc/omirc (if it exists), then it reads entries from ${HOME}/.omirc (if it exists), then it reads options and package names from the command line (they are used and read in the same order as they appear on the command line). See the examples section at the end.

Note that the names of the standard config files depend on the name of the binary. For example, if you call the binary "foobar" instead of "omi", it will look for the files /usr/local/etc/foobarrc and ${HOME}/.foobarrc instead.

Debug Settings

There are a lot of possible debug settings, and describing them all here would probably be confusing. Therefore only the most useful and important ones are explained. The debug keywords are not case sensitive. All output is sent to the standard error output ("stderr"), except possibly for the progress display (see below).

Report

Report only on changes to the local mirror, and why they have changed. This will not produce any output at all if no updates have to be done.

Progress

Display the progress of file transfers in bytes and percent (where applicable). Note that this only works if stdout or stderr is associated with a terminal. See the section Progress Display below for details.

Verbose

Verbosely report mirroring activity and FTP server control input/output per file.

Stats

Print final summary statistics if mirroring activity took place. This will not produce any output at all if no updates had to be done.

Statistics

Print statistics for every remote / local directory tree pair, as well as final summary statistics.

All

Report on everything. Warning: this can generate several megabytes of debug output!

None

Report on nothing, be quiet while mirroring.

You can combine multiple debug keywords by separating them with commas, for example "Report,Stats,Progress" (which is the default). You can also subtract debug keywords, for example "All,-Statistics,-Stats" -- this will print everything, but except for final and per-directory statistics.

If you specify debug options multiple times, the latter will replace (i.e. completely override) the former. So, if you want to add debug options to the existing ones, start the next set of options with a plus sign ("+"). Similarly, you can remove debug options from the existing set by starting them with a minus sign ("-"). Example:

      Debug  A,B,C,D           now in effect:  A,B,C,D
      Debug  +E,+F             now in effect:  A,B,C,D,E,F
      Debug  -B,-C             now in effect:  A,D,E,F
      Debug  A,B               now in effect:  A,B

In the second line, "+E,F" would have had the same effect, because "+" is the default for the second and following keywords on a line.

Note that error messages will always be printed, no matter which debug flags are used.

Mirror Option Settings

You can set a few mirror option flags, using "-o" on the command line, or using "Options" in a configuration file. The syntax is exactly the same as for debug options (see above). These mirror options change certain aspects of the default behaviour. Currently, the following option flags are supported:

GetOlderFiles

Normally, if a file on the local host has a newer datestamp than the remote file, it won't be retrieved. If you set this flag, it will be retrieved if the timestamp is different (no matter whether it's older or newer).

PreserveDirs

Normally, if a local directory does not exist anymore on the remote server, it is removed locally (including all files and subdirectories in it). If you set this flag, it will be preserved, and a warning will be printed (unless you have set the debug flags to "none"). You will then have to remove that directory manually, or (if you want to keep it) add it to the local exclude list. This is useful if the remote server is unreliable (i.e. if parts of it disappear sometimes and then are back the next day).

IgnoreTime

Ignore the timestamps of remote and local files when mirroring. I.e. only retrieve a remote file if it doesn't already exist locally, or if the file sizes mismatch. Useful if some stupid cronjob touches all files on the remote site, even though their contents don't change.

DontTouch

Do not update the time stamps of local files to be the same as those of the remote files. This only makes sense if you also use the "IgnoreTime" option flag, or the "deviation" config entry (-t option).

LocalTime

Assume that the time stamps of the remote server are local time. The default (if this flag is not set) is to use GMT. Most UNIX-based FTP servers seem to use GMT, so this flag should not be used normally.

ExcludeAll

Exclude all files and directories from mirroring by default. Use "Include" or -y to include files and directories to be mirrored. See also the section Excludes and Includes.

Test

This is for testing / debugging purposes. If this flag is set, no modifications are performed (no transmission or deletion of files). Useful to test exclusion patterns etc. without the risk to kill any files if something doesn't work as expected.

Active

Use "active" FTP transfers (the default is to use "passive" FTP transfers). This can be useful if the FTP server is behind a firewall, of if the passive mode support of the server is broken. Note that omi does not automatically switch between active and passive transfer mode.

NoStat

This option can be used to work-around a bug in some FTP servers. It causes omi not to use the RFC959 "STAT" command to retrieve a directory listing (it uses "LIST" instead). "STAT" (the default) is more efficient, but some FTP servers don't implement it correctly. Note that omi automatically falls back to "LIST", but sometimes it requires a rather long timeout, so better use this flag if needed.

Permissions

Try to parse the permission modes of remote files and apply them to the local files. Only r/w/x bits are used; but set-id bits are ignored for security reasons.
If this flag is not set, the default is to use the modes specified by the "Mode" config entry (-m option), and if that's not set either, the defaults are 0755 for directories and 0644 for files (rwxr-xr-x and rw-r--r--, respectively).
Note that omi requires access to all directories, so the permission modes for directories must be at least 0700. Otherwise omi will fail to run correctly.

None

None of the above options. This is the default.

Progress Display

Omi features a progress display, which is useful to watch transferring long files over slow connections. It starts if the transfer of a file takes longer than 10 seconds, and it is updated every 5 seconds. The display consists of four parts:

• The number of bytes transferred for this file so far.
• How man percent of this file have been transferred.
• The average transfer rate (bytes per second).
• The ETA (estimated time of arrival) in hh:mm:ss.

If the connection is idle for more than 10 seconds, it is declared as "stalled". In this case, the transfer rate and ETA are replaced by the duration of the idle time.

Please note that the transfer rate and ETA are calculated based on the elapsed time so far, which means that they are not very accurate at the beginning, but they should become more accurate while the transfer is running (provided that the transfer rate doesn't change significantly).

The progress display is enabled by default. You can disable it with "-d -progress" (command line) or "Debug -Progress" (configuration file).

Note: The progress display is disabled automatically if neither stdout nor stderr points to a regular terminal. This is the case if you redirect both stdout and stderr to a file or into another command (using a pipe). This can be a problem if you want to capture the regular output in a file, but you still want to watch the progress on the terminal. In this case, you have to redirect stdout to the terminal explicitly, and pipe stderr into the "tee" command (type "man tee" for information about tee). If you use an sh-like shell (sh, ksh, zsh, bash), the following command will do the trick:

( omi whatever-options-you-need > /dev/tty ) 2>&1 | tee logfile

For csh-like shells (csh, zsh, tcsh), use this one:

( omi whatever-options-you-need > /dev/tty ) |& tee logfile

If you do that frequently, it is probably more convenient to write a small shell script wrapper for the above commands. Please refer to the manual page of your shell to learn more about pipes and redirections.

If both stdout and stderr are regular terminals (i.e. not redirected), the progress display will be sent to stdout. If only one of them is a terminal, the progress display will be sent to that one. If neither is a terminal, progress display is disabled. Note that normal diagnostic output (reports, warnings, statistics, debug output etc.) always goes to stderr.

There are two common cases in which omi automatically does the right thing: (1) If you just call omi on the command line, without any redirections and pipes, you'll get the regular output and progress display as expected. (2) If you call omi via cron, there is no progress display. This makes sense, of course, because you cannot watch the progress anyway. Programs which are executed by cron don't have a terminal assoziated with stdout or stderr. Instead you will receive the output via email as soon as the job is finished (unless you redirect the output somewhere else, of course).

Exludes and Includes

Normally, omi mirrors the complete directory tree and makes an identical copy on the local machine. However, you often want to exclude certain files or directories from mirroring, or maybe you only want to mirror a small part of the whole directory tree.

To support this, omi offers the "Exclude" and "Include" commands (-x and -y command line options, respectively). They take a pattern (an extended regular expression) as argument. A detailed description of extended regular expressions can be found in the following manual pages, depending on your operating system:

Important: The backslash "\" and the dollar sign "$" have a special meaning in omi's configuration files, so you have to escape them with a backslash if you want to use them in regular expressions inside a configuration file.

There is one special thing: If the first character of the pattern is an exclamation mark, the whole pattern is negated. Example: the pattern "!stable|current" matches all files and directories that do not contain the word "stable" or "current" somewhere. Use this feature whith care, since it might produce surprising results. For example, the exclude pattern "!\.mp3$" excludes everything which does not have an .mp3 extension -- however, this will not mirror all files with an .mp3 extension, because it excludes all directories. See the examples section below for information about how to do this correctly.

Every single file and directory name (the complete path) is compared with the list of excludes and includes (in the same order in which they have been specified) until a match is found. If it matches an exclude pattern, the respective file or directory is not mirrored, otherwise it is mirrored. Note that always the complete path is compared (for example "/pub/foo/bar/file.tar.gz"), and that directories have a slash appended (for example "/pub/foo/bar/directory/"), so you can distinguish directories from plain files by matching the slash at the end.

If a file or directory does not match any of the patterns, it is mirrored by default. You can reverse this by using the "ExcludeAll" option flag. If you do this, be sure to include something explicitely, otherwise omi will not mirror anything, because everything is excluded by default. If you include certain files, omi automatically includes the directories which they're in, unless you explicitely exclude them.

There is no limit on the number of exclude and include patterns. However, note that omi takes more CPU load and processing time if there are a lot of patterns and/or very complicated patterns.

Here are a few examples of patterns (remember that you have to quote backslashes ("\\") and dollar signs ("\$") if used in config files!):

exe

Matches all files or directories that contain the string "exe" somewhere (in the path or in the actual name).

\.exe$

Matches all plain files with extension .exe.

/pub/win

Matches all files or directories having "/pub/win" in their path (this would also match "/pub/unix/emulators/pub/winblows").

^/pub/win$

Matches the plain file "/pub/win".

^/pub/win/$

Matches the directory "/pub/win".

^/pub/win/

Matches the directory "/pub/win" and everything inside it.

Note that the last two examples are equivalent if used as exclusion patterns: If you explicitely exclude a directory, everything in it is excluded, too. However, if you use them as inclusion patterns, they are different: The first one matches only the directory itself, the second one matches everything inside it (everything starting with "/pub/win/").

Here are some "real world" examples for certain mirror tasks that require using exlude and/or include patterns.

• We want to mirror everything, except for files called ".notar" or ".nocompress", or "lost+found" directories:

  Exclude /\\.no(tar|compress)\$
  Exclude /lost\\+found/
  

• We want to mirror only two files, called "ls-lR.Z" and "Index.complete", and nothing else. In this case it would be difficult (or maybe even impossible) to exclude all other files, so we use the "ExcludeAll" option flag:

  Options ExcludeAll
  Include /ls-lR.Z\$
  Include /Index.complete\$
  

• We want to mirror all files with an ".mp3" extension. Again, it's easiest to use the "ExcludeAll" option flag. As noted above, the directories that contain these files are included automatically.

  Options ExcludeAll
  Include \\.mp3\$
  

• We want to exclude all files which contain the word "hate", except for those which also contain the word "love". Note that the order matters in this case, because the first pattern that matches takes effect.

  Include love
  Exclude hate
  

• This is a similar example: We want to exclude everything containing the word "linux" or "Linux", except for the files in a directory "FreeBSD". So "FreeBSD/emu/linux" would be mirrored, but "somewhere/else/linux" would not. Again, the order matters in this case.

  Include /FreeBSD/
  Exclude [Ll]inux
  

• We want to mirror all files with a ".jpg" or a ".png" extension, except for those containing the word "sex" or "xxx". The order matters in this case, too.

  Options ExcludeAll
  Exclude sex|xxx
  Include \\.jpg\$
  Include \\.png\$
  

Timeout Settings

The mirror software uses several timeout values when accessing the network. If a timeout expires, the connection is closed and re-opened, and the operation which has been interrupted is repeated (unless a certain maximum number of retries has been reached; in this case the current mirror process is aborted with a fatal error). The intention of this timeout mechanism is to avoid situations in which the network connection hangs for a long time. Currently, the following timeout values are supported:

• DNS timeout ("D"), default 2 minutes: The maximum time allowed for a name server look-up.
• Connect timeout ("N"), default 2 minutes: The maximum time allowed for an FTP connection to be established. This applies to the control connection as well as to the data connection.
• Command timeout ("M"), default 3 minutes: The maximum time for an FTP command to be sent to the FTP server.
• Read timeout ("R"), default 10 minutes: This is the timeout for reading data from the FTP server. If the software does not receive at least one byte within this time, this timeout expires.

You can change the timeout values, using "-T" on the command line, or using "Timeout" in a configuration file. First you have to specify the letter for the type of the timeout ("D" = DNS, "N" = connect, "M" = command, "R" = read), followed by the number of seconds for this timeout. You can also specify it in minutes or hours if you append an "M" or "H", respectively. You can specify multiple timeouts, separating them by commas. Example:

Timeout D60, R5m

This sets the DNS timeout to 60 seconds and the read timeout to 5 minutes. The letters are not case-sensitive.

Note that if you set a timeout value to zero, that timeout is disabled, i.e. the respective operation may hang forever.

Retry and Delay Settings

If an error occurs while accessing the remote server, the software tries to repeat the command which caused the error (this is called a "soft retry"). If a certain number of soft retries occured in a row, the software closes the connection to the server, waits for some time, then re-opens the connection and tries to repeat the failed command again (this is called a "hard retry"). A hard retry is also performed if a soft retry wouldn't help (for example, if the connection to the server was lost or timed out). If a certain number of hard retries occured for the same server, the software "gives up".

You can change the retry parameters, using "-R" on the command line, or using "Retry" in a configuration file, followed by one or more specifications (separated by commas) which consist of a letter followed by a number. The letter indicates which parameter is to be changed:

• Soft retries ("S"), default 10: At most this many soft retries will be performed in a row, before a hard retry is enforced. (The internal counter for soft retries is reset to zero when the command is successul or when a hard retry is done.)

  Note: Do not set this number too high, otherwise the software might waste lots of time and bandwidth with soft retries in cases when a hard retry is necessary.

• Hard retries ("H"), default 10: At most this many hard retries will be performed before the software gives up trying the respective command (unrecoverable failure). (The internal counter for hard retries is reset to zero when the command is successul or when an unrecoverable failure occurs.)

• Unrecoverable failures ("F"), default 10: If this many failures occured for a server, the software assumes that this server is broken, or the connection to it is too bad right now. In this case, mirroring from this server is stopped completely.

• Local errors ("L"), default 10: This one is different from the above ones, as it affects local errors. If a local error occurs, it's usually one of these:

  • disk quota exceeded
  • no space left on device
  • permission denied

  All of these are normally unrecoverable and affect the complete mirror (not only one file). Therefore the software stops mirroring from the current server if the number of local errors reaches this value.

The following example sets the default retry parameters (the letters are not case-sensitive, and whitespace is optional):

Retry s 10, h 10, f 10, l 10

Note that with the default values there will be at most 110 retries for a single operation (for example to retrieve a file): First 10 soft retries, then a hard retry, then another 10 soft retries, etc., until 10 hard retries have been done (each of them preceded by 10 soft retries).

If a hard retry occurs, the software waits some time (the default is 25 seconds) before the connection is re-opened. If multiple hard retries occur in a row, this delay is increased until a maximum value is reached. You can change the delay settings, using "-D" on the command line, or using "Delay" in a configuration file, followed by one or more specifications (separated by commas) which consist of a letter followed by a time value (similar to the timeout specifications, see above). The letter indicates which delay setting is to be changed:

• Initial delay ("D"), default 25 seconds. This is the delay for the first hard retry that occurs for a certain command.
• Delay increment ("I"), default 25 seconds. Every time another hard retry for the same command occurs, the delay increases by this value.
• Maximum delay ("M"), default 15 minutes. The delay will not increase beyond this value.

The following example specifies the default delay settingss (the letters are not case-sensitive, and whitespace is optional):

Delay d 25s, i 25s, m 15m

With these settings, the first delay in a row of hard retries will be 25 seconds, the seconds one will be 50 seconds, the third one 75 seconds, etc., until the maximum of 15 minutes is reached.

Examples

This is a sample ${HOME}/.omirc configuration file:

      #
      #   This is my ~/.omirc
      #
      
      Debug      report,statistics
      Group      ftpadm
      Mode       644,2755
      PackageDir $HOME/mirror

Then you can put "package files" for different mirrors into your directory ${HOME}/mirror. This is a sample package file, which could be called ${HOME}/mirror/freebsd:

      #
      #   This is my ~/mirror/freebsd
      #
      
      Server     ftp7.de.freebsd.org
      Group      bsd
      Mode       664,2775
      
      RemoteDir  /pub/FreeBSD/ports-current
      LocalDir   /pub/mirror/FreeBSD/ports-current
      Exclude    /astro/|/biology/

      RemoteDir  /pub/FreeBSD/newsletter
      LocalDir   /pub/mirror/FreeBSD/newsletter

      RemoteDir  /pub/FreeBSD/docs
      LocalDir   /pub/mirror/FreeBSD/docs

Note that in this example, the group and permission modes are overridden for this package. If the "Group" entry would have been specified after the first "RemoteDir" entry, it would apply only to that directory tree, but not to the two other ones. Likewise, the "Exclude" line only applies to the first directory tree (the one which mirrors "ports-current").

In this sample scenario, you can just type "omi freebsd" to start the mirror (or put it in your crontab if you want to run the mirror regularly).

Troubleshooting

First of all, be sure you have read and understood the omi documentation. If you haven't done that, you certainly will run into trouble.

When some problem occurs, check whether it is reproducible. If it is not, then probably the FTP server had a temporary failure (a shut-down or something similar). If the problem persists, read on.

Basically, all problems fall into one of these classes:

1. User errors
2. Incompatibilities between omi and the FTP server
3. FTP server errors (network problems or ftpd bugs)
4. Omi errors (bugs)

Class 1 has already been discussed in the first paragraph of this section.

Class 2 is difficult. Unfortunately, RFC 959 (the standard which specifies the FTP protocol) is not very clear in some parts. For example, it does not specify a reliable way to retrieve machine-readable directory listings. Therefore, omi has to rely on certain assumptions. One of them is the directory format, which is assumed to be more or less UNIX- style "ls -l" output. If this is not the case, there isn't much that can be done about it. Bad luck.

Class 3 might be easier (or maybe not). If the connection to the server is very bad, try increasing the timeout values and the retry counts. If the connection is not that bad, but you still get timeouts right from the start, this might be caused by the fact that the FTP server is behind a fire- wall which doesn't allow arbitrary FTP data connections. In this case, try using the option for "active" FTP transfer mode. However, if you're behind a firewall yourself, active FTP transfers will probably not work.

Class 4: Bugs in omi. If you find one of these, please submit a bug report to the author, and try to include as much information as possible. If I can't reproduce the problem, I can't fix it. Of course, source patches are very welcome (I prefer diffs in unified format).

Here are a few hints which are helpful when trying to solve problems.

The most important thing is to use omi's debug settings, in order to increase the amount of debugging output (see the section Debug Settings). If you suspect a problem with the FTP protocol between omi and the FTP server, use this setting:

-d +connect,output,input

This will display the status of the connection to the FTP server, all commands that are sent to it, and and all replies that are received from it. Some knowledge of the FTP standard (RFC 959) is necessary to interpret the result. Note that the server sometimes sends multi-line messages, of which only the last line is displayed (which is usually the important one). If you want to see the complete messages, add the debug setting "multi" to those given above.

If you have trouble with regular expressions (exclude or include patterns), the following setting might shed some light on it:

-d +regex

For every local and remote file, it will be displayed if it matched any pattern. Also, the exact file name used for matching is displayed. If you're only interested in remote files, use "+remote-regex" instead (likewise "+local-regex" for local files only).

If you suspect that omi does not parse the directory listing of the FTP server correctly, the following might help:

-d +parse

It displays the parse tree, i.e. the internal representation of the directory structure after omi has parsed the listing of the FTP server.

Finally, the following setting will give you everything that omi could possibly say:

-d all

Note that this produces huge amounts of debug output, most of which is probably not useful for you.