LAN Mirroring Technologies

LAN Mirroring technologies are an important HA technology now, and will remain a key ingredient in many inexpensive HA systems. They allow two machines to mirror a filesystem between themselves with only two NICs and a crossover cable - yielding a system with no SPOF[1]s for a very low price.

• DRBD[2]: system by Philipp Reisner[3] and LarsEllenberg[4] for mirroring filesystems across the LAN. It is RAID and LAN aware with a quick sync option for bringing the mirror up to date quickly. Recent versions even have a very fast full-sync option. This is the general replication technology which is most commonly used by members of the Linux-HA community, because it provides a complete solution, is sensitive to data integrity issues, and integrates easily with Linux-HA.

• Enhanced Network Block Device[5] project by P.T. Breuer. ENBD is a project to make the NBD work right. It doesn't combine the NBD and RAID together like DRBD does.

  • Linux Journal[6] article by P.T. Breuer, A. Marin Lopez and Artuo Garcia Ares on The Network Block Device[7] A network block device (NBD) driver makes a remote resource look like a local device in Linux, allowing a cheap and safe real-time mirror to be constructed.

• NBD server software[8] - the complement to the in-kernel NBD driver.

-- AlanRobertson[9]

Volume Managers

• Sistina's LVM[10]: Linux Volume Manager. LVM allows the growth and shrinkage of partitions, as one would expect, but also allows the creation of snapshot partitions, enabling consistent online backups or recovery in filesystems and database systems.

• IBM's EVMS[11] Enterprise volume management system. The EVMS Project has the goal of providing unparalleled flexibility and extensibility in managing storage. It represents a new approach to logical volume management, as the architecture introduces a plug-in model that allows for easy expansion or customization of various levels of volume management. See information about the combined setup of Heartbeat v2 on the Xensource Wiki page on a SAN based EVMS-HA Xen solution[12]

-- AlanRobertson[9]

Filesystem technologies

Three types of filesystem technologies are important for Linux-HA: Journalling filesystems, Cluster filesystems and Mirror filesystems. Journalling filesystems allow takeover of shared/mirrored filesystems to occur rapidly in cases of failover. Cluster filesystems allow disks to be shared read-write between all members of a cluster simultaneously - which is highly desirable for many parallel applications. Mirror filesystems replicate, or mirror, files to multiple remote locations on the network in real time, ensuring that any given file is always up-to-date and always available.

Journalling and Log-structured Filesystems

• Reiserfs[13]: A tree-structured filesystem which journals updates. It is available with most 2.4 and later kernels.

• Ext3[14]: Stephen Tweedie's journalling version of the ext2 filesystem. It is available with most 2.4 and later kernels.

• JFS[15]: IBM's journalled filesystem technology. It is available with many Linux distributions.

• XFS[16]: SGI's journalled filesystem technology.

• LinLogFS[17]: A log structured filesystem for Linux

• Intermezzo[18]: A distributed file system which lets systems replicate directory trees. It is a standard part of 2.4 and later kernels. It isn't precisely what most people would call a cluster filesystem, but it's related.

Cluster Filesystems

• Lustre[19]: Lustre is a novel storage and file system architecture and implementation which does not follow the block device paradigm. The central target in this project is the development of a next-generation cluster file system which can serve clusters with 10,000's of nodes, petabytes of storage, move 100's of GB/sec. Lustre is in trial production in several sites across the world.

• GFS[20]: The Global Filesystem - fault tolerant, distributed, very cool, etc. GFS is both a cluster filesystem and a journalled filesystem.

• The CODA[21] distributed, fault-tolerant filesystem

• Oracle Cluster File System (OCFS)[22]: enterprise-class open-source cluster file system

Mirror Filesystems

• MFS[23]: The Mirror File System links and manages file systems on two or more geographically-dispersed machines. When a file is updated, the changed portion is mirrored in real time to a copy residing on a remote location. MFS is a new solution for HA, load balancing, DR, and online file backup built on top of ext3 and NFS. It is completely transparent to users and applications

Application-level HA Clustering

• Samba: Active-Active HA with shared storage

  (pdf[24]), (ogg audio[25]).

Related Projects

Below is a list of projects with complementary goals to those of the Linux-HA project.

• The Linux Virtual Server[26] project (LVS includes Network Address Translation, tunneling, and direct routing). LVS recommends using the Linux-HA "heartbeat" code. There is a document[27] describing how to configure them together.

• UltraMonkey[28]: LVS[29] and Heartbeat[30] and a GUI put together with nice documentation. Created and supported by long-time Linux-HA contributor horms[31].

• HA-OSCAR[32] “High Availability Open Source Cluster Application Resources” is an open source project aims to provide a combined power of High Availability and Performance Computing solution. Our goal is to enhance a Beowulf cluster system for mission critical grade applications. To achieve high availability, component redundancy is adopted in HA OSCAR cluster to eliminate this single point of failure. HA-OSCAR incorporates a self-healing mechanism; failure detection & recovery, automatic failover and fail-back. (HA-OSCAR uses Linux-HA technology)

• High Availability RAID[33] from Software RAID HOWTO

• The Linux Scalability Project[34]. The primary goal of this research is to improve the scalability and robustness of the Linux operating system to support greater network server workloads more reliably. We are specifically interested in single-system scalability, performance, and reliability of network server infrastructure products running on Linux, such as LDAP directory servers, IMAP electronic mail servers, and web servers, among others.

• OSCAR[35] Open Source Cluster Application Resources. The OSCAR is a snapshot of the best known methods for building, programming, and using clusters. It will hopefully bring some form of uniformity to clusters, foster commercial versions of the cluster software, and make clusters more broadly acceptable.

• The Linux/IR[36] project

• Linux Enterprise Computing[37] overview

• IP Channel Bonding[38] for Linux. Channel bonding is the technique of using more than one ethernet link to increase bandwidth and availability on IP networks. See IpFailoverChannelBonding[39] for how to use this with Linux-HA.

Monitoring and Administration Software for Linux

• lm-sensors[40]: hardware health monitoring project.

• safte-monitor[41]. SAFTE stands for SCSI Accessed Fault-Tolerant Enclosure device. If you have a SAF-TE compatible storage enclosure, safte-monitor will let read the enclosure configuration fetching things such as the number of fans, power supplies, slots, and also the mapping of slots to scsi ids. safte-monitor reads disk enclosure status information from SAF-TE capable enclosures (SCSI Accessible Fault Tolerant Enclosures). SAF-TE is a component of SES (SCSI Enclosure Services) which is common on most SCSI disk enclosures these days. saftemon can monitor multiple SAF-TE devices and will automatically probe and detect them. The information retreived includes power supply and fan status, temperature, audible alarm, drive faults, array critical / failed / rebuilding state and door lock status. saftemon logs changes in the status of these enclosure elements to syslog and can optionally execute an alert help program with details of the component failure. This could send a pager message for example. Temperature alert limits also be set.

  The SAF-TE spec is on the web[42] and an addendum is at this location[43]. More information about the specification here[44].

• Linux RAS[45] project. Ambitious. Mailing list information can be found on the web[46].

• Linux RAM ECC monitoring[47] with a corresponding Mailing list[48].

• Chris Brady's x86 Memory Testing program[49] (memtest86). This ships with newer versions of SuSE Linux.

• Mon[50]: service monitoring daemon.

• HAPM[51]: another service monitoring daemon. High Availability Port Monitor (HAPM) is a local port status check. It is a simple, light and fast daemon to check TCP/UDP ports. If one or more monitored ports (per IP) downs then the primary Heartbeat will be killed by HAPM.

• OpenNMS[52] is an open-source project dedicated to the creation of an enterprise grade network management platform.

• Spumoni[53] enables any program which can be queried via local commands to be health-checked via SNMP. This allows admins to use enterprise-level monitoring programs such as OpenNMS, Tivoli, OpenView, MRTG and RRDTool for even non-SNMP-enabled applications.

• Monit[54]: Monit is a utility for monitoring daemons or similar programs running on a Unix system. It will start specified programs if they are not running and restart programs not responding.

• VACM[55]: VA Cluster Manager. VACM provides cluster monitoring and control at a very fundamental level.

• PIKT[56]: Problem Informant Killer Tool

• NOCOL/SNIPS[57] - system and network monitoring tool

• Big Brother[58] - Systems and Network monitor. It monitor both network and system information.

• Big Sister[59] - a real time system and network health monitoring application

• Nagios[60] - Network monitor (formerly Netsaint)

• MAT[61] - Monitoring and Administration Tool. MAT is an easy to use network enabled UNIX configuration and monitoring tool. It provides an integrated tool for many common system administration tasks, including Backups, and Replication

• WebRAT[62] - a web based administration tool, to administer several nodes on a network, from a central host (administration server). The main purpose of WebRAT is to administer a network with many nodes, remotely. The more the nodes on the network, the more WebRAT will seem to be irreplaceable.

• xCAT (Extreme Cluster Administration Toolkit)[63] - A tool kit that can be used for the deployment and administration of (primarily high-performance) Linux clusters.

• dwatch[64] - daemon watching program -- last updated in 2001 -- appears to be dead

High-Availability-Related Protocols and Standards

• The Transis[65] group communications system - providing efficient group multicast for high-availability applications.

• SCTP[66] Stream Control Transmission Protocol (RFC 2960). From Joe Grimm: The HA features of SCTP revolve around using multi-homed hosts to provide failover paths implemented at the transport layer, essentially this becomes a transparent issue to SCTP applications. This is just one of the features in SCTP but one network standard being driving by the Carrier Grade community to provide 5 nines networking on top of the Internet. The linux kernel project to implement this can be found at: http://www.sf.net/projects/lksctp[67].

• The IETF's Reliable Server Pools[68] working group. The purpose of the working group is to develop an architecture and protocols for the management and operation of server pools supporting highly reliable applications, and for client access mechanisms to a server pool

• Werner Almesberger's TCP Connection Passing[69] project. tcpcp is a mechanism that allows cooperating applications to pass ownership of TCP connection endpoints from one Linux host to another one. Such functionality should be useful in load-balancing and failover applications. tcpcp can be used between hosts using different architectures and does not need the other endpoint of the connection to cooperate (or even to know what's going on).

• Extended Transmission Control Protocol (ETCP)[70]. ETCP helps within migration clusters because by using it, it becomes possible to have a network connection follow a process, instead of tcp IO being relayed through the original node. Given application state synchronization, this technique could also useful for failover clusters as well.

• UCARP - Common Address Redundancy Protocol (CARP) for Unix[71]. UCARP allows a couple of hosts to share common virtual IP addresses in order to provide automatic failover.

• VRRP (Virtual Router Redundancy Protocol)[72] is a redundancy protocol designed to increase the availability of the default gateway.

High-Performance Clusters

• Beowulf (Linux supercomputing clusters) home page[73]

• OSCAR[35] Open Source Cluster Application Resources. The OSCAR is a snapshot of the best known methods for building, programming, and using clusters. It will hopefully bring some form of uniformity to clusters, foster commercial versions of the cluster software, and make clusters more broadly acceptable.

• HA-OSCAR[32] “High Availability Open Source Cluster Application Resources” is an open source project aims to provide a combined power of High Availability and Performance Computing solution. Our goal is to enhance a Beowulf cluster system for mission critical grade applications. To achieve high availability, component redundancy is adopted in HA OSCAR cluster to eliminate this single point of failure. HA-OSCAR incorporates a self-healing mechanism; failure detection & recovery, automatic failover and fail-back. (HA-OSCAR builds on Linux-HA).

• Scyld Computing[74]. The original Beowulf team providing second generation Beowulf Clustering.

• OpenMosix[75] (scalable computing clusters, including process migration). Unfortunately, it's not HA :-(.

• Amata[76] high-availability for Beowulf clusters.

• Grid Engine[77]. The Grid Engine project provides distributed resource management software for wide ranging requirements from compute farms to grid computing.

Other Cluster-Related Portal-type sites

• Joe Greenseid's excellent Linux Clustering Information Center[78]

• IEEE Task Force On Cluster Computing High Availability Information Page[79]

• IEEE Task Force On Cluster Computing[80]

• High Performance Cluster Computing[81] resource page. High Performance Cluster Computing is a book edited by Dr. Rajkumar Buyya of Monash University's School of Computer Science and Software Engineering.

Other HA Management Software for Linux

• Linux Cluster Project[82] a community project sponsored by Red Hat[83]. Includes a DLM[84], a clustered file system (GFS) and a High-Availability component.

• Keepalived[85]: keepalive facility for LVS[29]. The main goal of the keepalived project is to add a strong and robust keepalive facility to the Linux Virtual Server project. This project is similar to the MON project, but it is in C with multilayer TCP/IP stack checks. Keepalived implements a framework based on three family checks : Layer3, Layer4 & Layer5.

• EZHA[86]: open HA clustering package. An "easy to use" cluster HA tool, able to manage up to 16 nodes, using network links or raw devices for heartbeat links, working fine on Linux and SUN Solaris. It has a small GUI.

• Simon Edwards'

  Linux Replicated High Availability Manager[87] (freshmeat[88]). Linuxha.net is a toolset that attempts to allow two node high availability clusters to be built under Linux using simple inexpensive hardware. Rather than using a single data image (made available via SCSI or Fibre Channel connected storage) Linuxha.net instead replicates the data between the two machines.

• Mission Critical Linux's Kimberlite[89] Open Source High-Availability system.

• Jerome Etienne's VRRP implementation[90]. Jerome implemented VRRPv2 as specfied by rfc2338[91].

• Fake[92]: Redundant server switch software home page

• SGI's Linux FailSafe[93] project. Linux FailSafe[94] is the most complete and functional open source HA software. Unfortunately, it's pretty much dead on Linux

• HP's Cluster Infrastructure for Linux[95]. This project is developing a kernel-based infrastructure for Linux clustering by extending the Cluster Membership and Internode Communication Subsystems from HP's NonStop Clusters for SCO Unixware code base.

• HP's Single System Image (SSI) Clusters for Linux[96]. The SSI project incorporates HP's NonStop Clusters for SCO Unixware technology and other open source technology to provide a full, highly available SSI environment for Linux.

• HAPPI[97] High Availability Peer-to-Peer Implementation. HAPPI is a simple Perl-based high availability clustering technology that uses IP aliasing for IP-failover across a number of machines.

• Mysql Cluster[98].

• Dominque Chabord's Shaman-X[99] software - providing opensource and free high-availability disaster tolerance, and crisis management software.

• Ericsson's Eddie[100] open source High-Availability Server farm project. It is written in Ericsson's own programming language Erlang/OTP[101].

• Mnesia[102] is a distributed Database Management System designed to handle failovers as part of Eddie.

• Clusterit[103]: parallel clustering software for *BSD systems.

• Andreas Muller's HA-failover[104] project. It seems like a nice implementation, mainly targeted at Solaris. Here is a summary from email he sent me:

  • Consists of two components: a daemon that stores state information about services served by the machine, and a shell that checks the state of all the services on remote machines (heartbeat).
  • Shell is programmable in Tcl.
  • Heartbeat uses either TCP or UDP, TCP always non-blocking.
  • Monitoring interfaces: text base, curses and html
  • The complete documentation is on the web site. There is also a detailed example on the server that shows how to do IP address failover, including sending a gratuitous ARP on Solaris (a tool for sending gratuitous ARPs using DLPI is included, but there is also a less carefully tested Linux version). As the thing is programmable in Tcl, it doing failover for disk systems or other kinds of services is quite easy to implement. The only missing component is that machine must be able to find out about the state of a disk, and it must publish it in the daemon. I guess Mon can do such things, I usually use scripts written for the occasion.

• Andrew Barnett's pl-cluster[105] software. Andrew wasn't happy with heartbeat, so he wrote his own version. It uses multicast, and can run in listen-only mode. According to Andrew "It is alpha at best, YMMV".

• Life (LInux Failover Extension[106]) software for the Linux Virtual Server.

• SRRD[107] is the Service Routing Redundancy Daemon. SRRD was designed and developed starting September 2002 by Amir Guindehi <amir@srrd.org> as a master thesis at the Swiss Federal Institute of Technology in Zurich.

• Failoverd[108] (discontinued): Failover daemon home page. Failoverd is now VRRP-based.

• Ninja[109]: Highly Available, Scaleable Computing Environment.

Other White Papers and Related Materials

Below is a list of white papers which have been brought to our attention. We don't agree with everything said here, but they are still probably interesting to many people.

• There are current High Availability reports, white papers, product literature, webcasts, and case studies at Bitpipe[110]

• A High-Availability Cluster Checklist[111] A Linux Journal Article by Tim Burke

• A High-Availability Cluster for Linux[112] A Linux Journal Article by Phil Lewis

• Linux Journal[6] #55 (November, 1998)[113] article: High Availability Linux Web Servers by Aaron Gowatch

• MontaVista HA white paper[114]. by John Mehaffey <mehaf@mvista.com>

• Ron Resnick's Taxonomy of High-Availability[115] (originally prepared for Nortel). Basic definitions, and discussion of a few technologies.

References