There are several cases where you will need to manually clean up a build:
You have manually interrupted it.
The head node has been rebooted while a build was running.
qmanager has crashed and has been restarted.
Manually interrupting a build is a bit messy. First you need to identify the tty in which it's running (either record the output of tty(1) when you start the build, or use ps x to identify it. You need to make sure that nothing else important is running in this tty, e.g., ps -t p1 or whatever. If there is not, you can just kill off the whole term easily with pkill -t pts/1; otherwise issue a kill -HUP in there by, for example, ps -t pts/1 -o pid= | xargs kill -HUP. Replace p1 by whatever the tty is, of course.
The package builds dispatched by make to the client machines will clean themselves up after a few minutes (check with ps x until they all go away).
If you do not kill make(1), then it will spawn more jobs. If you do not kill dopackages, then it will restart the entire build. If you do not kill the pdispatch processes, they'll keep going (or respawn) until they've built their package.
To free up resources, you will need to clean up client machines by running build cleanup command. For example:
% /var/portbuild/scripts/build cleanup i386 8-exp 20080714120411 -full
If you forget to do this, then the old build jails will not be cleaned up for 24 hours, and no new jobs will be dispatched in their place since pointyhat thinks the job slot is still occupied.
To check, cat ~/loads/* to display the status of client machines; the first column is the number of jobs it thinks is running, and this should be roughly concordant with the load average. loads is refreshed every 2 minutes. If you do ps x | grep pdispatch and it is less than the number of jobs that loads thinks are in use, you are in trouble.
You may have problem with the umount commands hanging. If so, you are going to have to use the allgohans script to run an ssh(1) command across all clients for that buildenv. For example:
% ssh gohan24 df
will get you a df, and
% allgohans "umount -f pointyhat.freebsd.org:/var/portbuild/i386/8-exp/ports" % allgohans "umount -f pointyhat.freebsd.org:/var/portbuild/i386/8-exp/src"
are supposed to get rid of the hanging mounts. You will have to keep doing them since there can be multiple mounts.
Note: Ignore the following:
umount: pointyhat.freebsd.org:/var/portbuild/i386/8-exp/ports: statfs: No such file or directory umount: pointyhat.freebsd.org:/var/portbuild/i386/8-exp/ports: unknown file system umount: Cleanup of /x/tmp/8-exp/chroot/53837/compat/linux/proc failed! /x/tmp/8-exp/chroot/53837/compat/linux/proc: not a file system root directoryThe former two mean that the client did not have those mounted; the latter two are a bug.
You may also see messages about procfs.
After you have done all the above, remove the ${arch}/lock file before trying to restart the build. If you do not, dopackages will simply exit.
If you have to do a ports tree update before restarting, you may have to rebuild either duds, INDEX, or both.
Here are the rest of the options for the build command:
build destroy arch branch - Destroy the build id.
build list arch branch - Shows the current set of build ids.