This site best when viewed with a modern standards-compliant browser. We recommend Firefox Get Firefox!.

Linux-HA project logo
Providing Open Source High-Availability Software for Linux and other OSes since 1999.

USA Flag UK Flag

Japanese Flag

Homepage

About Us

Contact Us

Legal Info

How To Contribute

Security Issues

18 August 2008 Heartbeat release 2.1.4 is now out Download it and install it!

11 October 2007 NEW educational HA/DR Blog hosted by Alan Robertson

9 April 2007 Check out the Cool Heartbeat Screencasts: Installation, Intro to the GUI Part of the Heartbeat Education project

Last site update:
2008-08-20 19:11:08

Contents

  1. Introduction
  2. Extra Options
  3. Simple Example
  4. What if?
  5. FAQ
  6. Writing a Clone Resource Agent
    1. Which Resource Instance is Promoted
    2. Requirements for ResourceAgents
    3. Extra Operations
    4. Extra Environment Variables
    5. Proper Interpretation of Master Environment Variables
  7. Advanced Example
  8. Location Constraints for Masters
  9. Location Constraints for Slaves
  10. Co-locational and Ordering Constraints related to Masters
  11. Co-locational and Ordering Constraints related to Slaves
  12. Finding Masters in the Status Section

Related Pages: v2/Concepts v2/Concepts/Clones outdated documentation

Introduction

Superset of a cloned resource.

  • Can be either anonymous or unique

Currently supports two states

  • The states mean whatever you want them to mean

Extra Options

  • master_max (default: 1)

    • The maximum number of resource masters that should be run in the cluster.
  • master_node_max (default: 1)

    • The maximum number of resource masters that should be run on any given node.

Simple Example

In the example below we have specified that the WebServerIP resource should be a ClusteredIP. It will be started on at most 3 nodes and at most there can be 2 clone active on any given node. Instance 1 will serve the first "bucket", instance 2 the second and so on.

  • <master_slave id="MyCorpWWW" ordered="false" interleave="false" notify="false">
      <instance_attributes>
        <attributes>
          <nvpair name="clone_max" value="3"/>
          <nvpair name="clone_node_max" value="2"/>
          <nvpair name="master_max" value="2"/>
          <nvpair name="master_node_max" value="1"/>
        </attributes>
      </instance_attributes>
      <primitive id="WebServerIP" class="ocf" type="IPaddr" provider="heartbeat">
        <instance_attributes>
          <attributes>
            <nvpair name="ip" value="127.0.0.26"/>
          </attributes>
        </instance_attributes>
      </primitive>
    </master_slave>
     

What if?

  • If there are >=1 nodes available:

    • Two nodes will be chosen and each node will run 1 master
  • If there are 2 nodes available:
    • 1 node will run 2 clones (1 master, 1 slave), the other 1 master
  • If there is 1 nodes available:
    • 1 node will run 2 clones (1 master, 1 slave), the clone will not be started

FAQ

  1. Will the master clone instances always be an unbroken sequence?
    • No.
  2. Will the master clone instances always start at 0?
    • No.
  3. Does the CRM keep track of the clone instances' state?
    • No.

      It is the responsibility of the OCFResourceAgent to know what state the clone instance is in. The CRM knows what state it thinks the resource should be in, but in all cases it is the script which must understand the resource and determine it's true state.

  4. Why does monitoring stop after my resource is promoted to master?
    • You must define a monitor action for every role (e.g. Master, Slave, Started, Stopped) you wish to monitor. Please refer to the examples given in ClusterInformationBase/Actions.

Writing a Clone Resource Agent

Which Resource Instance is Promoted

During the start operation, Resource Agent scripts should call the crm_master command-line tool. This tool automatically detects both the resource and host and should be used to set a preference for being promoted. Based on this, master_max, and master_node_max, the instance(s) with the highest preference will be promoted.

BeekhofG5::beekhof pengine # crm_master 
usage: crm_master [-?VQ] -(D|G|v) [-l]
Options
        --help (-?)     : this help message
        --verbose (-V)  : turn on debug info. additional instances increase verbosity
        --quiet (-Q)    : Print only the value on stdout (use with -G)
        --get-value (-G)        : Retrieve rather than set the attribute
        --delete-attr (-D)      : Delete rather than set the attribute
        --attr-value (-v) <string>      : Value to use (ignored with -G)
        --lifetime (-l) <string>        : How long the preference lasts (reboot|forever)

Requirements for ResourceAgents

This items listed here are in addition to the requirements for Clones.

ResourceAgents are required to accurately report how many instances are active, inactive, masters and slaves. In monitor operations:

  • Active masters should return $OCF_RUNNING_MASTER
  • Failed masters should return $OCF_FAILED_MASTER
  • Inactive instances should return $OCF_NOT_RUNNING
  • All other active instances should use
    • $OCF_SUCCESS if they are active, or
    • one of the $OCF_ERR_* values if they have failed

All OCF variables can normally be found in /usr/lib/ocf/resource.d/heartbeat/ocf-shellfuncs which should be the first thing included by a resource agent.

Extra Operations

A Master/Slave Resource Agent must implement the promote and demote operations. The implementation is up to the user.

Extra Environment Variables

In addition to the environment variables added for clones, these variables are available to notify actions.

  • OCF_RESKEY_notify_operation (start|stop|promote|demote)

  • OCF_RESKEY_notify_{x}_resource

  • OCF_RESKEY_notify_ {x}_uname

  • {x} : promote, demote, master, slave

Starting with 2.0.4, they will also be available to start, stop, promote and demote actions if notify="true" is set.

Proper Interpretation of Master Environment Variables

  1. Pre-notification (demote)
    • Active resources: $OCF_RESKEY_notify_active_resource
    • Master resources: $OCF_RESKEY_notify_master_resource
    • Slave resources: $OCF_RESKEY_notify_slave_resource
    • Inactive resources: $OCF_RESKEY_notify_inactive_resource
    • Resources to be started: $OCF_RESKEY_notify_start_resource
    • Resources to be promoted: $OCF_RESKEY_notify_promote_resource
    • Resources to be demoted: $OCF_RESKEY_notify_demote_resource
    • Resources to be stopped: $OCF_RESKEY_notify_stop_resource
    • Resources that were started: NA
    • Resources that were promoted: NA
    • Resources that were demoted: NA
    • Resources that were stopped: NA
  2. Post-notification (demote) / Pre-notification (stop)
    • Active resources: $OCF_RESKEY_notify_active_resource
    • Master resources: $OCF_RESKEY_notify_master_resource
      • minus $OCF_RESKEY_notify_demote_resource

    • Slave resources: $OCF_RESKEY_notify_slave_resource
    • Inactive resources: $OCF_RESKEY_notify_inactive_resource
    • Resources to be started: $OCF_RESKEY_notify_start_resource
    • Resources to be promoted: $OCF_RESKEY_notify_promote_resource
    • Resources to be demoted: $OCF_RESKEY_notify_demote_resource
    • Resources to be stopped: $OCF_RESKEY_notify_stop_resource
    • Resources that were started: NA
    • Resources that were promoted: NA
    • Resources that were demoted: $OCF_RESKEY_notify_demote_resource
    • Resources that were stopped: NA
  3. Post-notification (stop) / Pre-notification (start)
    • Active resources: $OCF_RESKEY_notify_active_resource
      • minus $OCF_RESKEY_notify_stop_resource

    • Master resources: $OCF_RESKEY_notify_master_resource
      • minus $OCF_RESKEY_notify_demote_resource

    • Slave resources: $OCF_RESKEY_notify_slave_resource
      • minus $OCF_RESKEY_notify_stop_resource

    • Inactive resources: $OCF_RESKEY_notify_inactive_resource
      • plus $OCF_RESKEY_notify_stop_resource

    • Resources to be started: $OCF_RESKEY_notify_start_resource
    • Resources to be promoted: $OCF_RESKEY_notify_promote_resource
    • Resources to be demoted: $OCF_RESKEY_notify_demote_resource
    • Resources to be stopped: $OCF_RESKEY_notify_stop_resource
    • Resources that were started: NA
    • Resources that were promoted: NA
    • Resources that were demoted: $OCF_RESKEY_notify_demote_resource
    • Resources that were stopped: $OCF_RESKEY_notify_stop_resource
  4. Post-notification (start) / Pre-notification (promote)
    • Active resources: $OCF_RESKEY_notify_active_resource
      • minus $OCF_RESKEY_notify_stop_resource
        plus $OCF_RESKEY_notify_start_resource

    • Master resources: $OCF_RESKEY_notify_master_resource
      • minus $OCF_RESKEY_notify_demote_resource

    • Slave resources: $OCF_RESKEY_notify_slave_resource
      • minus $OCF_RESKEY_notify_stop_resource
        plus $OCF_RESKEY_notify_start_resource

    • Inactive resources: $OCF_RESKEY_notify_inactive_resource
      • plus $OCF_RESKEY_notify_stop_resource
        minus $OCF_RESKEY_notify_start_resource

    • Resources to be started: $OCF_RESKEY_notify_start_resource
    • Resources to be promoted: $OCF_RESKEY_notify_promote_resource
    • Resources to be demoted: $OCF_RESKEY_notify_demote_resource
    • Resources to be stopped: $OCF_RESKEY_notify_stop_resource
    • Resources that were started: $OCF_RESKEY_notify_start_resource
    • Resources that were promoted: NA
    • Resources that were demoted: $OCF_RESKEY_notify_demote_resource
    • Resources that were stopped: $OCF_RESKEY_notify_stop_resource
  5. Post-notification (promote)
    • Active resources: $OCF_RESKEY_notify_active_resource
      • minus $OCF_RESKEY_notify_stop_resource
        plus $OCF_RESKEY_notify_start_resource

    • Master resources: $OCF_RESKEY_notify_master_resource
      • minus $OCF_RESKEY_notify_demote_resource
        plus $OCF_RESKEY_notify_promote_resource

    • Slave resources: $OCF_RESKEY_notify_slave_resource
      • minus $OCF_RESKEY_notify_stop_resource
        plus $OCF_RESKEY_notify_start_resource
        minus $OCF_RESKEY_notify_promote_resource

    • Inactive resources: $OCF_RESKEY_notify_inactive_resource
      • plus $OCF_RESKEY_notify_stop_resource
        minus $OCF_RESKEY_notify_start_resource

    • Resources to be started: $OCF_RESKEY_notify_start_resource
    • Resources to be promoted: $OCF_RESKEY_notify_promote_resource
    • Resources to be demoted: $OCF_RESKEY_notify_demote_resource
    • Resources to be stopped: $OCF_RESKEY_notify_stop_resource
    • Resources that were started: $OCF_RESKEY_notify_start_resource
    • Resources that were promoted: $OCF_RESKEY_notify_promote_resource
    • Resources that were demoted: $OCF_RESKEY_notify_demote_resource
    • Resources that were stopped: $OCF_RESKEY_notify_stop_resource

Advanced Example

Location Constraints for Masters

It is sometimes desirable to preferentially constrain the master instance of a master/slave resource to run on a particular node or set of nodes in the cluster.

The CIB snippet below will add prefer the node ace to be master of the resource ms-drbd1 by 100 points.

<rsc_location id="loc:ms-drbd1_likes_ace" rsc="ms-drbd1">
  <rule id="rule:ms-drbd1_likes_ace" role="master" score="100">
    <expression  attribute="#uname" operation="eq" value="ace"/>
  </rule>
</rsc_location>

Location Constraints for Slaves

It is sometimes desirable to preferentially constrain the slave instance of a master/slave resource to run on a particular node or set of nodes in the cluster.

However, since all resources go through slave status before becoming promoted, what we have to do is constrain it to avoid becoming master.

The CIB snippet below will prefer to promote any node but fred to be master of the resource drbd1 by 100 points. This is effectively the same as saying "we want fred to avoid becoming master if possible".

<rsc_location id="loc:drbd1_likes_fred" rsc="drbd1">
  <rule id="rule:drbd1_likes_fred" role="master" score="100">
    <expression  attribute="#uname" operation="ne" value="fred">
  </rule>
</rsc_location>

Co-locational and Ordering Constraints related to Masters

It is very common to want to have a certain resource run only on a node which is master for a given resource.

You can either make sure the resource is run on the master, which is the propably more common approach:

<rsc_order id="ms-drbd0_before_fs0" from="fs0" action="start" to="ms-drbd0" to_action="promote"/>
<rsc_colocation id="fs0_on_ms-drbd0" to="ms-drbd0" to_role="master" from="fs0" score="infinity"/>

Translation:

  • Start fs0 after ms-drbd0 has been promoted
  • Only start fs0 when ms-drbd0 is running as master

Or you can make sure the resource is not run on the slave:

<rsc_order id="ms-drbd0_before_fs0" from="fs0" action="start" to="ms-drbd0" to_action="promote"/>
<rsc_colocation id="fs0_not_on_stopped_ms-drbd0" to="ms-drbd0" to_role="stopped" from="fs0" score="-infinity"/>
<rsc_colocation id="fs0_not_on_slave_ms-drbd0" to="ms-drbd0" to_role="slave" from="fs0" score="-infinity"/>

Translation:

  • Never run fs0 on a node where ms-drbd0 is stopped.

  • Never run fs0 on a node where ms-drbd0 is running as slave.

  • Only start fs0 after promoting ms-drbd0 to master.

Since the the total set of states for a master/slave resource is {stopped, slave, master}, this only allows it to run on a node which is running as master.

Co-locational and Ordering Constraints related to Slaves

The idiom to use if you have a resource which you want to run on the slave node is similar to the colocation with master case:

<rsc_colocation id="fs_on_drbd0" to="drbd0-partition"
  to_role="stopped"
  from="mount-drbd0" score="-infinity"/>
<rsc_colocation id="fs_on_drbd0" to="drbd0-partition"
  to_role="master"
  from="mount-drbd0" score="-infinity"/>

Translation:

  • Never run fs_on_drbd0 on a node where drbd0-partition is stopped.

  • Never run fs_on_drbd0 on a node where drbd0-partition is running as master.

Since the the total set of states for a master/slave resource is {stopped, slave, master}, this only allows the resource to run on a node which is running as slave.

Question:
I don't know how to write the corresponding ordering rule for this case - to ensure that it only runs after being started OR after being demoted from master. Is it necessary? If it isn't necessary, then probably the ordering rule for the master case isn't needed either.

Finding Masters in the Status Section