This site best when viewed with a modern standards-compliant browser. We recommend Firefox Get Firefox!.

Linux-HA project logo
Providing Open Source High-Availability Software for Linux and other OSes since 1999.

USA Flag UK Flag

Japanese Flag

Homepage

About Us

Contact Us

Legal Info

How To Contribute

Security Issues

This web page is no longer maintained. Information presented here exists only to avoid breaking historical links.
The Project stays maintained, and lives on: see the Linux-HA Reference Documentation.
To get rid of this notice, you may want to browse the old wiki instead.

1 February 2010 Hearbeat 3.0.2 released see the Release Notes

18 January 2009 Pacemaker 1.0.7 released see the Release Notes

16 November 2009 LINBIT new Heartbeat Steward see the Announcement

Last site update:
2017-12-11 23:01:27

Tiebreaker Server

For supporting split-site clusters, we envision the need for a tiebreaker server. This page provided some of the background thinking which led to our quorum server. As such, this page mainly provides background information, and the design proposed below wasn't exactly the one implemented.

This server would give out a token to one subcluster which had contacted it and asked to have quorum. If it was granted the exclusive privilege of being the primary subcluster it would receive a HAVEQUORUM token, and if it were not granted quorum, it would be given a NOQUORUM token.

If a node did not renew its HAVEQUORUM token, then the server would begin sending the node at the other end of the cluster a NOQUORUM token until the beginning of a new bidding cycle for selecting a new primary subcluster.

It is important that the tiebreaker server effectively have its own heartbeat mechanism (through the continual token renewal), so that if one node dies when it has the quorum token, that the tiebreaker service know that it has died. Otherwise it would never be able to pass the quorum token along to another node when this happens. Since UDP has no connection mechanism, and TCP connections normally take around 10 minutes to notify you when they are disconnected, unfortunately one can't rely on network disconnect notifications for this function.

Once this happens, no subcluster would be able to receive the HAVEQUORUM token for a period of time.

Then after that, all connections in good standing would be eligible to be selected as the primary subcluster.

The process for obtaining quorum during a bidding interval would happen like this:

  • the tiebreaker server would announce an open bidding cycle

  • Each client (subcluster) would bid on being selected as the primary subcluster
  • Based on the bids, the server would select one of the subclusters to become the primary subcluster

Once a primary subcluster was selected, the process the winner would be:

  • From then on, the primary subcluster continue to acknowledge its continued selection by sending acknowledgement tokens to the tiebreaker server on a periodic basis.
  • If a winning client receives a NOQUORUM token, or fails to hear from the tiebreaker server for a period of time, it must consider that it has lost quorum, and notify the CCM through some as-yet-to-be-designed asynchronous (callback) notification system.
  • If the process performing the quorum computation is shut down gracefully, it should notify the tiebreaker server that it is going away.

Once a primary subcluster was selected, the process all the losers would follow would be:

  • stay connected and wait for a new bid announcement.

When a client subcluster first connects to the server, it provides its identification and bid information to the tiebreaker server, and waits for a new bid announcement. If the server wishes to, it can terminate the current holder of the quorum token and begin a new bid process as though the current holder had failed to renew in time.

Thoughts about the QUORUMACK packets

Should they contain a 'deadtime' value? Or should they be slaved to the server? Obviously some kind of common expectation (negotiation) has to happen here...

Open Issues

When a node which had a connection open goes down gracefully, can its connection be transferred gracefully? or is it just something which will cause a rebid for quorum?

  • On the one hand, a unique token could be passed along to the new instance when it shuts down gracefully.
  • On the other hand, changes in membership probably changes the "bid" amount for quorum.
  • If the handoff is graceful, there doesn't need to be a deadtime notice, but there need to be a delay if there is a second subcluster bidding for quorum for the case that quorum changes subclusters as a result.

zhenh: Can we don't let the client keep the connection? for example, every minutes the clients connect to the tiebreaker, send the data of itself, then get the quorum status from tiebreaker, then disconnect.

  • AlanR: I thought I addressed this above, but I see that I didn't. You could theoretically design it to allow disconnect and reconnect, but TCP/IP connections are both slow to start and fairly expensive to start, but they are much more efficient once they've been established for a while (like 4 seconds or so), and they're cheap to keep up. In UDP, there is no connection - so you can't connect or disconnect. My first guess is that it is both noticeably more efficient and more reliable to stay connected if you choose TCP, and irrelevant if you choose UDP. On a related note, one of the most common criticisms of HTTP 1.0 is that it has very short connection times, which both increase server load, and slow down data transfer. So, subsequent versions of HTTP make it possible to reuse server connections to avoid this problem. We can avoid it completely right from the beginning if we just choose to stay connected.

TCP or UDP

  • TCP is oriented to bulk data transfer and stream connections
  • UDP is oriented to connectionless communication and low latency
  • TCP is easier to manage through firewalls - because its connection orientation makes it possible to allow the firewall to better tell what's going on with the connection.
  • Many of these connections will go through firewalls.
  • You can run SSL over TCP. Is SSL necessary??

I would conclude that UDP is the better protocol when no firewalls or encryption are involved. I would conclude that TCP may be necessary when dealing with firewalls, because corporate security directives are often inflexible. If one allowed both TCP and allowed people to choose port numbers then this would allow one the maximum flexibility to conform to corporate guidelines.

  • [The more I think about this protocol the more sense UDP seems to make, since there is no bulk data transfer, and latency is more important than throughput. But, I'm not really an expert in this]

Practical Considerations

It is necessary for to be accessible from both sites, and highly desirable that the tiebreaker server be located on a third site, and both the server (or HA server pair) and the networks to it from the other sites be as reliable as possible. Low latency and high band and the should be highly reliable.

Since the quorum server may serve many clusters, and many versions of cluster software, it is necessary for it to be very tolerant of different versions of the software running on either end of the connection. It might also prove desirable to have different versions of the server be running on different ports at the same time.