Print v2/Concepts/Clones: from Linux HA

Contents

Introduction
Extra Options
Simple Example
What if?
FAQ
Writing a Clone Resource Agent
Advanced Example
Location Constraints for Clones
Ordering Constraints and Clones
Co-locational Constraints and Clones
Finding Clones in the Status Section

Related Pages v2/Concepts[1] v2/Concepts/MultiState[2]

Introduction

Clones were initially conceived as a convienient way to start N instances of an IP resource and have them distributed throughout the cluster for load balancing. They have turned out to quite useful for a number of purposes including integrating with Red Hat[3]'s DLM, the fencing subsystem and OCFS2.

In fact you can clone any resource provided the ResourceAgent[4] supports it. Like regular resources you may even have it configured differently depending on which host it is active on.

Three types of cloned resources exist.

Anonymous Clones.
- The simplest type of clones, these resources behave completely identically everywhere they are running.
Globally Unique Clones.
- These resources are distinct entities. An instance of the clone running on one machine is not equivalent to another instance on another node. Nor would any two instances on the same node be equivalent.
Stateful Clones.
- Active instances of these resources are divided into two states active and passive (aka. primary/secondary, master/slave, choose whichever term suits you). Stateful clones can be either anonymous or globally unique.
  You can read more about them here: v2/Concepts/MultiState[2]

Extra Options

ordered (default: FALSE)
- Start (or stop) each clone only after the operation on the previous clone completed.
interleave (default: FALSE)
- This relates to action ordering... see "Ordering Constraints and Clones" below.
notify (default: FALSE)
- Notify the an instance of the clone before and after stopping/starting its siblings.
  This requires the notify action to be implemented. See "Writing a Clone Resource Agent" for details below.
clone_max (default: 1)
- The maximum number of copies of the resource that should be run in the cluster.
clone_node_max (default: 1)
- The maximum number of copies of the resource that should be run on any given node.
globally_unique (default: TRUE)
- Is this clone a Globally Unique Clone or not.

Simple Example

In the example below we have specified that the WebServerIP resource should be a ClusteredIP. It will be started on at most 3 nodes and at most there can be 2 clone active on any given node. Instance 1 will serve the first "bucket", instance 2 the second and so on.

<clone id="MyCorpWWW" ordered="false" interleave="false" notify="false">
  <instance_attributes>
    <attributes>
      <nvpair name="clone_max" value="3"/>
      <nvpair name="clone_node_max" value="2"/>
    </attributes>
  </instance_attributes>
  <primitive id="WebServerIP" class="ocf" type="IPaddr" provider="heartbeat">
    <instance_attributes>
      <attributes>
        <nvpair name="ip" value="127.0.0.26"/>
      </attributes>
    </instance_attributes>
  </primitive>
</clone>

What if?

If there are >=3 nodes available:
- Three nodes will be chosen and each node will run 1 clone
If there are 2 nodes available:
- 1 node will run 2 clones, the other 1
If there is 1 nodes available:
- 1 node will run 2 clones, the clone will not be started

FAQ

Will the active clone instances always be an unbroken sequence?
- No.
Will the active clone instances always start at 0?
- No.
Start order (with ordered=false)
- All clone instances not yet active would be started approximately at the same time.
Stop order (with ordered=false)
- All clone instances still active would be stopped approximately at the same time.
Start order (with ordered=true)
- If nothing was yet started, the order would be like this:
  1. Start WebServerIP:0
  2. Start WebServerIP:1
  3. Start WebServerIP:2
Stop order (with ordered=true)
- If all instances were active, the order would be like this:
  1. Stop WebServerIP:2
  2. Stop WebServerIP:1
  3. Stop WebServerIP:0

Writing a Clone Resource Agent

All OCF variables can normally be found in /usr/lib/ocf/resource.d/heartbeat/ocf-shellfuncs which should be the first thing included by a resource agent.

Requirements for ResourceAgents

- Only OCF ResourceAgents[5] are supported for cloning. This is due to the extra variables that need to be sent to the agent.
- ResourceAgents[5] are required to accurately report how many instances are active. In order to recover from possible failures or administrative accidents, the CRM checks for active resources (generally this is done at startup). In CRM terminology, these are called "probes" and one is sent for every possible clone instance. Probes can be recognized in shell scripts with the following code:
  [ "${__OCF_ACTION}" = "monitor" -a "${OCF_RESKEY_CRM_meta_interval}" = "0" ]
- . For anonymous clones, the ResourceAgent[4] must only respond with ${OCF_SUCCESS} to N of these probes. Where N is the number of active instances of the clone on that node. This is the minimum requirement but it is preferable for the ResourceAgent[4] to act as defined for globally unique clones below.
- . For globally unique clones, the ResourceAgent[4] must only respond with ${OCF_SUCCESS} if the node has that exact instance active. Clone instances are identified as clone number OCF_RESKEY_CRM_meta_clone of OCF_RESKEY_CRM_meta_clone_max. These variables are available for ResourceAgents[5] to use. All other probes for instances of the clone should result in ${OCF_NOT_RUNNING}. Unless of course they are failed, in which case they should return one of the other OCF error codes.

v2/Concepts/MultiState[2] resources have additional requirements not discussed here.

Extra Operation (notify)

A Clone Resource Agent can optionally implement the notify operation.

Notification can be enabled by adding notify="true" to the resource's definition.

Once enabled, your resource will receive pre and post notification events telling the resource what is or did just take place. Currently these are informational updates only and are not permitted to fail. Future version may allow notify actions to fail.

Extra Environment Variables

OCF_RESKEY_globally_unique (default: true)

These variables are available to notify actions.

OCF_RESKEY_CRM_meta_notify_type (pre|post)
OCF_RESKEY_CRM_meta_notify_operation (start|stop)
OCF_RESKEY_CRM_meta_notify_{x}_resource
OCF_RESKEY_CRM_meta_notify_ {x}_uname
{x} : start, stop, active, inactive

Starting with 2.0.4, they will also be available to start and stop actions if notify="true" is set; notify_type and notify_operation is not provided there.

Proper Interpretation of Clone Environment Variables

Pre-notification
- Active resources: $OCF_RESKEY_CRM_meta_notify_active_resource
- Inactive resources: $OCF_RESKEY_CRM_meta_notify_inactive_resource
- Resources to be started: $OCF_RESKEY_CRM_meta_notify_start_resource
- Resources to be stopped: $OCF_RESKEY_CRM_meta_notify_stop_resource
Post-notification (stop)
- Active resources: $OCF_RESKEY_CRM_meta_notify_active_resource
  - minus $OCF_RESKEY_notify_stop_resource
- Inactive resources: $OCF_RESKEY_CRM_meta_notify_inactive_resource
  - plus $OCF_RESKEY_CRM_meta_notify_stop_resource
- Resources that were started: $OCF_RESKEY_CRM_meta_notify_start_resource
- Resources that were stopped: $OCF_RESKEY_CRM_meta_notify_stop_resource
Post-notification (start)
- Active resources: $OCF_RESKEY_CRM_meta_notify_active_resource
  - minus $OCF_RESKEY_CRM_meta_notify_stop_resource
    plus $OCF_RESKEY_CRM_meta_notify_start_resource
- Inactive resources: $OCF_RESKEY_CRM_meta_notify_inactive_resource
  - plus $OCF_RESKEY_CRM_meta_notify_stop_resource
    minus $OCF_RESKEY_CRM_meta_notify_start_resource
- Resources that were started: $OCF_RESKEY_CRM_meta_notify_start_resource
- Resources that were stopped: $OCF_RESKEY_CRM_meta_notify_stop_resource

Advanced Example

An IP address on its own isnt much use. More than likely we'd like something like an apache instance to be started as well. What we can do is define the target of the clone to be a resource group.

<clone id="MyCorpWWW">
  <instance_attributes>
    <attributes>
      <nvpair name="clone_max" value="3"/>
      <nvpair name="clone_node_max" value="1"/>
    </attributes>
  </instance_attributes>
  <group id="MyCorpWWW_group">
    <primitive id="WebServerIP" class="ocf" type="IPaddr" provider="heartbeat">
      <instance_attributes>
        <attributes>
          <nvpair name="ip" value="127.0.0.26"/>
        </attributes>
      </instance_attributes>
    </primitive>
    <primitive id="WebServerApache" class="ocf" type="Apache" provider="heartbeat">
      <instance_attributes>
        <attributes>
          <nvpair name="apache_config" value="/home/www.mycorp.com/conf/apache.conf"/>
        </attributes>
      </instance_attributes>
    </primitive>
  </group>
</clone>

This will allocate one bucket and one apache instance to up to three nodes. If nothing was yet started and ordered=true, interleave=true, the startup order would be like this:

Start WebServerIP:0
Start WebServerApache:0
Start WebServerIP:1
Start WebServerApache:1
Start WebServerIP:2
Start WebServerApache:2

If instead ordered=true, interleave=false, the startup order would be like this:

Start WebServerIP:0
Start WebServerIP:1
Start WebServerIP:2
Start WebServerApache:0
Start WebServerApache:1
Start WebServerApache:2

Location Constraints for Clones

Using constraints for the clone as a whole is no different to a native resource... you just use the id of the group.

<rsc_location id="run_MyCorpWWW" rsc="MyCorpWWW">
    <rule id="pref_run_WebServerIP" score="100" boolean_op="or">
        <expression attribute="#uname" operation="eq" value="c001n01"/>
        <expression attribute="#uname" operation="eq" value="c001n02"/>
        <expression attribute="#uname" operation="eq" value="c001n03"/>
    </rule>
</rsc_location>

We specify 3 nodes since we will effectivly have 3 resources to place.

Ordering Constraints and Clones

If we defined two more resources and had one start before MyCorpWWW and one after, the results are pretty much what you would expect.

Example:

<primitive id="Oracle" class="lsb" type="oracle"/>
<primitive id="WwwStats" class="lsb" type="apache_stats"/>
<rsc_order id="order_oracle_www" from="MyCorpWWW" action="start" type="after" to="Oracle"/>
<rsc_order id="order_stats_www" from="MyCorpWWW" action="start" type="before" to="WwwStats"/>

If nothing was yet started, the order would be like this:

Start Oracle
Start WebServerIP:0
Start WebServerApache:0
Start WebServerIP:1
Start WebServerApache:1
Start WebServerIP:2
Start WebServerApache:2
Start WwwStats

and the reverse for stop.

If however, the "Oracle" resource was also a clone,

<clone id="Oracle">
  <instance_attributes>
    <attributes>
      <nvpair name="clone_max" value="3"/>
      <nvpair name="clone_node_max" value="1"/>
    </attributes>
  </instance_attributes>
  <primitive id="Oracle_clone" class="lsb" type="oracle"/>
</clone>

the orders would be:

Start Oracle_clone:0
Start Oracle_clone:1
Start Oracle_clone:2
Start WebServerIP:0
Start WebServerApache:0
Start WebServerIP:1
Start WebServerApache:1
Start WebServerIP:2
Start WebServerApache:2
Start WwwStats

and the reverse for stop.

And if interleave=true was specified,

<clone id="Oracle_clones" interleave="true">
  <instance_attributes>
    <attributes>
      <nvpair name="clone_max" value="3"/>
      <nvpair name="clone_node_max" value="1"/>
    </attributes>
  </instance_attributes>
  <primitive id="Oracle" class="lsb" type="oracle"/>
</clone>

then it would be:

Start Oracle:0
Start WebServerIP:0
Start WebServerApache:0
Start Oracle:1
Start WebServerIP:1
Start WebServerApache:1
Start Oracle:2
Start WebServerIP:2
Start WebServerApache:2
Start WwwStats

and the reverse for stop.

Co-locational Constraints and Clones

Co-locational -INFINITY constraints (ie. the resources must not be co-located) are the same as for any other resource.

<rsc_colocation id="not_same" from="www_prod_clones" to="www_test_clones" score="-INFINITY"/>

Dont run production and test environments on the same machine.

Co-locational INFINITY constraints (ie. the resources must be co-located) for clones only make sense when the other resource is also a clone. Even then you are better off adding the resource to the resource group. All other cases are ignored.

Finding Clones in the Status Section

to be filled in...

References

[1]	http://www.linux-ha.org/v2/Concepts
[2]	http://www.linux-ha.org/v2/Concepts/MultiState
[3]	http://en.wikipedia.org/wiki/Red_Hat
[4]	http://www.linux-ha.org/ResourceAgent
[5]	http://www.linux-ha.org/ResourceAgents