This site best when viewed with a modern standards-compliant browser. We recommend Firefox Get Firefox!.

Linux-HA project logo
Providing Open Source High-Availability Software for Linux and other OSes since 1999.

USA Flag UK Flag

Japanese Flag

Homepage

About Us

Contact Us

Legal Info

How To Contribute

Security Issues

This web page is no longer maintained. Information presented here exists only to avoid breaking historical links.
The Project stays maintained, and lives on: see the Linux-HA Reference Documentation.
To get rid of this notice, you may want to browse the old wiki instead.

1 February 2010 Hearbeat 3.0.2 released see the Release Notes

18 January 2009 Pacemaker 1.0.7 released see the Release Notes

16 November 2009 LINBIT new Heartbeat Steward see the Announcement

Last site update:
2018-06-20 00:58:58

Types of a message

There are different types of messages which have different journeys.

  1. status (heartbeat) messages every node generates in certain period of time.
  2. control messages, e.g., failover messages
  3. client messages
  4. fifo messages

1. status (heartbeat) messages

sending

detail follows:

the function hb_send_local_status() are put in the main looping function to send a heartbeat message in each config->heartbeat_ms miliseconds. The config->heartbeat_ms is the value you set in /etc/ha.d/ha.cf for keepalive times 1000. A status message sent out contains the following fields:

  • t=status
  • st=<curnode->status> (usually it is "up") (the following fields are common to all messages)

  • src=<source node name>

  • srcuuid=<source node uuid>

  • seq=<message sequence number>

  • hg=<heartbeat generation number>

  • ts=<time stamp>

  • ld=<load average>

  • ttl=<time to live>

  • dest=<destination>

  • destuuid=<destination uuid>

  • auth=<authentication string>

A status message is send out via send_cluster_msg() to all medias.

receiving

A status message is read by a read child process. Every read child process is running the function read_child(). Heartbeat create one read process per media. Once the message is read, the read child process feeds the message into the IPC channel between main process and itself.

The main process will check time to time if there is any message available in read_dispatch() function. This function is put in the main loop. After it gets a message, it will process it through process_clustermsg(). If the status from the source node is the same, nothing will be changed, otherwise /etc/ha.d/rc.d/status will be invoked to update the status of the source node.

flow chart for a status message:

                             sending node                                              receiving node

                      _____________________________                            ____________________________________
                      |                           |                            |                                  |
                      |     main process          |                            |      main process                |
                      |    hb_send_local_status() |                            |     process_clustermsg()         |
                      -----------------------------                            ------------------------------------
                                  |                                                            ^
                                  |                                                            |
                                  |                                                            |
                      ___________\|/_______________                            ________________|___________________
                      |                            |                           |                                  |
                      | write child process        |--------------------------&gt;|     read child process           |
                      |    write_child()           |                           |        read_child()              |
                      -----------------------------                            -----------------------------------


sending function call chain:

main process:
       hb_send_local_status() -&gt; send_local_status() -&gt; send_cluster_msg() -&gt; process_outbound_packet() -&gt; send_to_all_media() -&gt; socket_send()

write child process:

       write_child()
                     -&gt; ipcmsgfromIPC() -&gt; socket_recv()
                     -&gt; media_write_func ( e.g. bcast_write() )


receiving function call chain:


read child process:

        read_child()
                    -&gt; media_read_func ( e.g. bcast_read() )
                    -&gt; socket_send()


main process:

        read_child_dispatch()
                     -&gt; msgfromIPC()
                     -&gt; process_clustermsg()


2. control messages

3. client messages

A client message is send out by a client process through IPC to the main process. It will then be replied or forwarded to the whole cluster depending what kind of request this client message is. The main process will generate an answer message if the request message can be answered immediately. Specifically the following message can be answered without broadcasting the whole cluster:

  • (t=hbapi-req for api request messages)
  • signon request (reqtype=signon)
  • signoff request (reqtype=signoff)
  • nodelist request (reqtype=nodelist)
  • node status request (reqtype=nodestatus)
  • node type request (reqtype=nodetype)
  • iflist request (reqtype=iflist)
  • ifstatus request (reqtype=ifstatus)
  • get uuid request (reqtype=getuuid)
  • get name request (reqtype=getnodename)
  • setmsgsignal request (reqtype=setsignal)
  • setfmode request (reqtype=setfilter)
  • get parameter request (reqtype=getparm)
  • get resources request (reqtype=getrsc)

The following is flow chart for these messages:

                         ________________________    request message          _______________________________
                        |                        |---------------------------&gt;|                              |
                        | client process         |                            |   master process             |
                        |                        |&lt;-------------------------- |                              |
                        --------------------------  reply message (if any)    --------------------------------


client process function call chain:
           client_api -&gt;  msg2ipcchan() -&gt; socket_send()

master process function call chain:
            APIclients_input_dispatch() -&gt; ProcessAnAPIRequest()
                                                             -&gt; msgfromIPC_noauth()
                                                             -&gt; api_process_request() -&gt; corresponding_handler_function


The following messages sent out by a client usually will be forwarded to the whole cluster.

  • client status query (t=query-cstatus)
  • sendclustermsg
  • sendnodemsg
  • send_ordered_clustermsg
  • send_ordered_nodemsg

A message is first send from the client process to the master process through IPC. Upon receiving the message, the master process decides to broadcast the message to the whole cluster through communction media. On the receiving node, the master process looks at the F_TOID("to_id") field. If it matches any client id in the local node, it will send the message to the client.

The flow chart follows:

                             sending node                                              receiving node


                      _____________________________                            ____________________________________
                      |                           |                            |                                  |
                      |     client process        |                            |      client process              |
                      |                           |                            |                                  |
                      -----------------------------                            ------------------------------------
                                  |                                                            ^
                                  |                                                            |
                                  |                                                            |
                                  |                                                            |
                      ___________\|/______________                            _________________|_________________
                      |                           |                            |                                  |
                      |     main process          |                            |      main process                |
                      |                           |                            |                                  |
                      -----------------------------                            ------------------------------------
                                  |                                                            ^
                                  |                                                            |
                                  |                                                            |
                      ___________\|/_______________                            ________________|___________________
                      |                            |                           |                                  |
                      | write child process        |--------------------------&gt;|     read child process           |
                      |                            |                           |                                  |
                      -----------------------------                            -----------------------------------

function call chain in the sending node client process:

   &lt;api_func&gt; -&gt; msg2ipcchan() -&gt;socket_send()

function call chain in the sending node master process:

   APIclients_input_dispatch() -&gt; ProcessAnAPIRequest()
                                                   -&gt; msgfromIPC_noauth()
                                                   -&gt; api_process_request() -&gt; send_cluster_msg() -&gt;process_outbound_packet()-&gt;send_to_all_media()

function call chain in the receiving node master process:
   read_child_dispatch()
                -&gt; msgfromIPC()
                -&gt; process_clustermsg() -&gt; heartbeat_monitor() -&gt; api_heartbeat_monitor() -&gt;  api_send_cliennt_msg() -&gt; msg2ipcchan()

function call chain in the receiving node client process:
   read_msg_w_callbacks() -&gt; read_hb_msg() -&gt; msgfromIPC()

4. FIFO messages

FIFO is used in heartbeat to send cluster messages from a shell script --- /var/lib/heartbeat/fifo is the file name. On start heartbeat create a seperate process to read messages from FIFO, send them to the master process and then to the cluster. Scripts which send messages to the cluster through FIFO include: /usr/lib/heartbeat/req_resource, /usr/lib/heartbeat/hb_takeover, /usr/lib/heartbeat/hb_standby, etc. The function used to send messages in scripts is ha_clustermsg(), which is defined in /etc/ha.d/shellfuncs.

The following is the fifo message flow chart

                               scripts
                                  |
                                  |
                                  |
                                  |
                      ___________\|/______________
                      |                           |
                      |   fifo read process       |
                      |                           |
                      -----------------------------
                                  |
                                  |
                                  |
                      ___________\|/_______________
                      |                            |
                      |    master process          |
                      |                            |
                      -----------------------------
                                  |
                                  |
                                 \|/

                              to the cluster


The fifo read process is running on fifo_child() function. The master process read messages from fifo read process thourgh IPC by scheduling fifo_child_msg_dispatch() into its main loop.