Setup of high availability NFS servers

This page was originally created by GuochunShi but is now maintained primarily by DaveDykstra.

Please refer to one of these papers:

• AlanRobertson wrote an article entitled Highly-Affordable High-Availability which appeared in the November, 2003 issue of the US publication Linux Magazine

• Andre Bonhote wrote an article on HA-NFS entitled

  High-Availability NFS Server with Linux Heartbeat (pdf) in the August, 2003 issue of the European publication Linux Magazine.

• Also the How-To Setting Up A Highly Available NFS Server by Falko Timme may be helpful.

For those who want a quick setup, here are the steps to go (We assume you already know how to setup Heartbeat.):

1. Make sure you are using a shared disk device, either shared by hardware between your two servers or by DRBD.

2. Mount your shared device in one of the machines. For example, if the device is /dev/sdb1 and the directory you want to mount to is /data:

   mount -t ext3 /dev/sdb1 /data 
   

   If the directory you will export is /data/userdata, create it if it is not there yet:

   mkdir /data/userdata
   

3. Move /var/lib/nfs to your shared disk, make a link (/var/lib/nfs) to it:

   mv /var/lib/nfs /data
   ln -s /data/nfs /var/lib/nfs
   

   On the other machine, remove the /var/lib/nfs directory and create a link instead:

   rm -fr /var/lib/nfs
   ln -s /data/nfs /var/lib/nfs
   

4. You need to supply rpc.statd your cluster name (hostname for your floating IP address). On Debian the locking service is started by /etc/init.d/nfs-common and you can avoid editting the common file by putting STATDOPTS="-n <cluster_host_name>" in /etc/default/nfs-common. Otherwise, it is in the /etc/init.d/nfslock script on many Linux systems:

   start() {
           ...
           echo -n $"Starting NFS statd: "
           ...
           daemon rpc.statd "$STATDARG" -n <cluster_host_name>
           ....
   }
   

5. Export the user directory. Add the following line to /etc/exports:

   /data/userdata  *(rw,sync)
   

   and run

   exportfs -a
   

Hints

1. NFS-mounting any filesystem on your NFS servers is highly discouraged. DaveDykstra wanted both servers to NFS-mount the replicated filesystem from the active server, and through a lot of trouble mostly got it working but still saw scenarios where "NFS server not responding" could interfere with heartbeat failovers and he finally gave up on it. The biggest problem was with the fuser command hanging. For more details see the archives of the mailing list discussions beginning here and here.

2. If your kernel defaults to using TCP for NFS (as is the case in 2.6 kernels), switch to UDP instead by using the 'udp' mount option. If you don't do this, you won't be able to quickly switch from server "A" to "B" and back to "A" because "A" will hold the TCP connection in TIME_WAIT state for 15-20 minutes and refuse to reconnect.

Device Numbering

Note The device major/minor numbers are embedded in the NFS filehandle, making the filehandle go stale if the major/minor numbers change as the failover happens. If that is the case, you can either magically change your configuration to make it the same if you are using DRBD , or if you have shared disks you can use either EVMS or LVM to create a shared volume which will have the same major/minor numbers. You can download EVMS in http://evms.sourceforge.net/ and LVM in http://www.sistina.com/products_lvm.htm

Another alternative to deal with the NFS device numbering problem is afforded by later versions of NFS (commonly included with 2.6 Linux kernels). In these versions, you can specify an integer to be used in place of the major/minor of the mount device through the fsid parameter. For more details see exports(5).

Locking

Unfortunately, on recent kernels locks do not survive HA failovers. It is recommended that you mount NFS filesystems with the "nolock" option. For more details see this mailing list post and also this confirmation using a more recent kernel. The former information about setting up locking that was on this page is now at HaNFSOldLocking.

Active/Active HA NFS

The above directions and comments apply mainly to ActivePassive arrangements. For information on ActiveActive setups, please refer to Matt Schillinger's NFS page.

Linux-HA site built by the Linux-HA crowd. HTML4 / CSS2