31.7. Diskless Operation
Prev | Chapter 31. Advanced Networking | Next

## 31.7. Diskless Operation

Updated by Jean-François Dockès.
Reorganized and enhanced by Alex Dupre.

A FreeBSD machine can boot over the network and operate without a local disk, using file systems mounted from an NFS server. No system modification is necessary, beyond standard configuration files. Such a system is relatively easy to set up because all the necessary elements are readily available:

The Intel® Preboot eXecution Environment (PXE) can be used to load the kernel over the network. It provides a form of smart boot ROM built into some networking cards or motherboards. See pxeboot(8) for more details.

A sample script (`/usr/share/examples/diskless/clone_root`) eases the creation and maintenance of the workstation's root file system on the server. The script will probably require a little customization.

Standard system startup files exist in `/etc` to detect and support a diskless system startup.

Swapping, if needed, can be done either to an NFS file or to a local disk.

There are many ways to set up diskless workstations. Many elements are involved, and most can be customized to suit local taste. The following will describe variations on the setup of a complete system, emphasizing simplicity and compatibility with the standard FreeBSD startup scripts. The system described has the following characteristics:

- The diskless workstations use a shared, read-only `/` and `/usr`.

  The root file system is a copy of a standard FreeBSD root, with some configuration files overridden by ones specific to diskless operation or, possibly, to the workstation they belong to.

  The parts of the root which have to be writable are overlaid with md(4) file systems. Any changes will be lost when the system reboots.

### Caution:

As described, this system is insecure. It should live in a protected area of a network and be untrusted by other hosts.

### 31.7.1. Background Information

Setting up diskless workstations is both relatively straightforward and prone to errors. These are sometimes difficult to diagnose for a number of reasons. For example:

- Compile time options may determine different behaviors at runtime.

- Error messages are often cryptic or totally absent.

In this context, having some knowledge of the background mechanisms involved is useful to solve the problems that may arise.

Several operations need to be performed for a successful bootstrap:

- The machine needs to obtain initial parameters such as its IP address, executable filename, server name, and root path. This is done using the DHCP or BOOTP protocols. DHCP is a compatible extension of BOOTP, and uses the same port numbers and basic packet format. It is possible to configure a system to use only BOOTP and bootpd(8) is included in the base FreeBSD system.

- DHCP has a number of advantages over BOOTP such as nicer configuration files and support for PXE. This section describes mainly a DHCP configuration, with equivalent examples using bootpd(8) when possible. The sample configuration uses ISC DHCP which is available in the Ports Collection.

- The machine needs to transfer one or several programs to local memory. Either TFTP or NFS are used. The choice between TFTP and NFS is a compile time option in several places. A common source of error is to specify filenames for the wrong protocol. TFTP typically transfers all files from a single directory on the server and expects filenames relative to this directory. NFS needs absolute file paths.

- The possible intermediate bootstrap programs and the kernel need to be initialized and executed. PXE loads pxeboot(8), which is a modified version of the FreeBSD third stage loader, loader(8). The third stage loader will obtain most parameters necessary to system startup and leave them in the kernel environment before transferring control. It is possible to use a `GENERIC` kernel in this case.

- Finally, the machine needs to access its file systems using NFS.

Refer to diskless(8) for more information.

### 31.7.2. Setup Instructions

#### 31.7.2.1. Configuration Using ISC DHCP

The ISC DHCP server can answer both BOOTP and DHCP requests.

ISC DHCP is not part of the base system. Install the `net/isc-dhcp42-server` port or package.

Once ISC DHCP is installed, edit its configuration file, `/usr/local/etc/dhcpd.conf`. Here follows a commented example for PXE host `corbieres`:

default-lease-time 600; max-lease-time 7200; authoritative; option domain-name "example.com"; option domain-name-servers 192.168.4.1; option routers 192.168.4.1; subnet 192.168.4.0 netmask 255.255.255.0 { use-host-decl-names on; option subnet-mask 255.255.255.0; option broadcast-address 192.168.4.255; host corbieres { hardware ethernet 00:02:b3:27:62:df; fixed-address corbieres.example.com; next-server 192.168.4.4; filename "pxeboot"; option root-path "192.168.4.4:/data/misc/diskless"; } }

| This option tells dhcpd to send the value in the `host` declarations as the hostname for the diskless host. An alternate way would be to add an `option host-name `corbieres`` inside the `host` declarations.
| The `next-server` directive designates the TFTP or NFS server to use for loading loader(8) or the kernel file. The default is to use the same host as the DHCP server.

The `filename` directive defines the file that PXE will load for the next execution step. It must be specified according to the transfer method used. PXE uses TFTP, which is why a relative filename is used here. Also, PXE loads `pxeboot`, not the kernel. There are other interesting possibilities, like loading `pxeboot` from a FreeBSD CD-ROM `/boot` directory. Since pxeboot(8) can load a `GENERIC` kernel, it is possible to use PXE to boot from a remote CD-ROM.

| The `root-path` option defines the path to the root file system, in usual NFS notation. When using PXE, it is possible to leave off the host's IP address as long as the BOOTP kernel option is not enabled. The NFS server will then be the same as the TFTP one.

#### 31.7.2.2. Booting with PXE

By default, pxeboot(8) loads the kernel via NFS. It can be compiled to use TFTP instead by specifying the `LOADER_TFTP_SUPPORT` option in `/etc/make.conf`. See the comments in `/usr/share/examples/etc/make.conf` for instructions.

There are two other `make.conf` options which may be useful for setting up a serial console diskless machine: `BOOT_PXELDR_PROBE_KEYBOARD`, and `BOOT_PXELDR_ALWAYS_SERIAL`.

To use PXE when the machine starts, select the `Boot from network` option in the BIOS setup or type a function key during system initialization.

#### 31.7.2.3. Configuring the TFTP and NFS Servers

If PXE is configured to use TFTP, enable tftpd(8) on the file server:

1. Create a directory from which tftpd(8) will serve the files, such as `/tftpboot`.

2. Add this line to `/etc/inetd.conf`:

   tftp dgram udp wait root /usr/libexec/tftpd tftpd -l -s /tftpboot

   ### Note:

   Some PXE versions require the TCP version of TFTP. In this case, add a second line, replacing `dgram udp` with `stream tcp`.

3. Tell inetd(8) to reread its configuration file. Add `inetd_enable="YES"` to `/etc/rc.conf` in order for this command to execute correctly:

   `#` `service inetd restart`

Place `tftpboot` anywhere on the server. Make sure that the location is set in both `/etc/inetd.conf` and `/usr/local/etc/dhcpd.conf`.

Enable NFS and export the appropriate file system on the NFS server.

1. Add this line to `/etc/rc.conf`:

   nfs_server_enable="YES"

2. Export the file system where the diskless root directory is located by adding the following to `/etc/exports`. Adjust the mount point and replace `corbieres` with the names of the diskless workstations:

   `/data/misc` -alldirs -ro `margaux corbieres`

3. Tell mountd(8) to reread its configuration file. If NFS is enabled in `/etc/rc.conf`, it is recommended to reboot instead.

   `#` `service mountd restart`

#### 31.7.2.4. Building a Diskless Kernel

When using PXE, building a custom kernel with the following options is not strictly necessary. These options cause more DHCP requests to be issued during kernel startup, with a small risk of inconsistency between the new values and those retrieved by pxeboot(8) in some special cases. The advantage is that the host name will be set. Otherwise, set the host name in a client-specific `/etc/rc.conf`.

options BOOTP # Use BOOTP to obtain IP address/hostname options BOOTP_NFSROOT # NFS mount root file system using BOOTP info

The custom kernel can also include `BOOTP_NFSV3`, `BOOT_COMPAT` and `BOOTP_WIRED_TO`. Refer to `NOTES` for descriptions of these options.

These option names are historical and slightly misleading as they actually enable indifferent use of DHCP and BOOTP inside the kernel.

Build the custom kernel, using the instructions in Chapter 9, Configuring the FreeBSD Kernel, and copy it to the place specified in `/usr/local/etc/dhcpd.conf`.

#### 31.7.2.5. Preparing the Root File System

Create a root file system for the diskless workstations in the location listed as `root-path` in `/usr/local/etc/dhcpd.conf`.

##### 31.7.2.5.1. Using `make world` to Populate Root

This method is quick and will install a complete virgin system, not just the root file system, into `DESTDIR`. Execute the following script:

#!/bin/sh export DESTDIR=/data/misc/diskless mkdir -p ${DESTDIR} cd /usr/src; make buildworld && make buildkernel make installworld && make installkernel cd /usr/src/etc; make distribution

Once done, customize `/etc/rc.conf` and `/etc/fstab` placed into `DESTDIR` according to the system's requirements.

#### 31.7.2.6. Configuring Swap

If needed, a swap file located on the server can be accessed via NFS.

##### 31.7.2.6.1. NFS Swap

The kernel does not support enabling NFS swap at boot time. Swap must be enabled by the startup scripts, by mounting a writable file system and creating and enabling a swap file. To create a swap file:

`#` `dd if=/dev/zero of=`/path/to/swapfile` bs=1k count=1 oseek=`100000``

To enable the swap file, add the following line to `/etc/rc.conf`:

swapfile=`/path/to/swapfile`

#### 31.7.2.7. Miscellaneous Issues

##### 31.7.2.7.1. Running with a Read-only `/usr`

If the diskless workstation is configured to run Xorg, adjust the XDM configuration file as it puts the error log on `/usr` by default.

##### 31.7.2.7.2. Using a Non-FreeBSD Server

When the server for the root file system is not running FreeBSD, create the root file system on a FreeBSD machine, then copy it to its destination, using tar(1) or cpio(1).

In this situation, there are sometimes problems with the special files in `/dev`, due to differing major/minor integer sizes. A solution to this problem is to export a directory from the non-FreeBSD server, mount this directory onto a FreeBSD machine, and use devfs(5) to allocate device nodes transparently for the user.

Prev | Up | Next
31.6. Link Aggregation and Failover | Home | 31.8. PXE Booting with an NFS Root File System

All FreeBSD documents are available for download at http://ftp.FreeBSD.org/pub/FreeBSD/doc/

Questions that are not answered by the documentation may be sent to <freebsd-questions@FreeBSD.org>.
Send questions about this document to <freebsd-doc@FreeBSD.org>.