You can use qclient command to monitor the status of build nodes, and to list the currently scheduled jobs:
% python path/qmanager/qclient jobs % python path/qmanager/qclient status
The scripts/stats ${branch} command shows the number of packages already built.
Running cat /var/portbuild/*/loads/* shows the client loads and number of concurrent builds in progress. The files that have been recently updated are the clients that are online; the others are the offline clients.
Note: The pdispatch command does the dispatching of work onto the client, and post-processing. ptimeout.host is a watchdog that kills a build after timeouts. So, having 50 pdispatch processes but only 4 ssh(1) processes means 46 pdispatches are idle, waiting to get an idle node.
Running tail -f ${arch}/${branch}/build.log shows the overall build progress.
If a port build is failing, and it is not immediately obvious from the log as to why, you can preserve the WRKDIR for further analysis. To do this, touch a file called .keep in the port's directory. The next time the cluster tries to build this port, it will tar, compress, and copy the WRKDIR to ${arch}/${branch}/wrkdirs/.
If you find that the system is looping trying to build the same package over and over again, you may be able to fix the problem by rebuilding the offending package by hand.
If all the builds start failing with complaints that they cannot load the dependent packages, check to see that httpd is still running, and restart it if not.
Keep an eye on df(1) output. If the /var/portbuild file system becomes full then Bad Things™ happen.
The status of all current builds is generated periodically into the packagestats.html file, e.g., http://pointyhat.FreeBSD.org/errorlogs/packagestats.html. For each buildenv, the following is displayed:
updated is the contents of .updated. This is why we recommend that you update .updated for -exp runs (see below).
date of latest log
number of lines in INDEX
the number of current build logs
the number of completed packages
the number of errors
the number of duds (shown as skipped)
missing shows the difference between INDEX and the other columns. If you have restarted a run after a ports tree update, there will likely be duplicates in the packages and error columns, and this column will be meaningless. (The script is naive).
running and completed are guesses based on a grep(1) of build.log.