There is a node in your CIB which keeps reappearing after you delete the node definition.
The undead node is problably a leftover from somewhere in the past (you created it sometime ago). In order to solve the problem, shutdown heartbeat, delete the /var/lib/heartbeat/hostcache file on all your nodes and start heartbeat. Then delete the node from your CIB using cibadmin once and for all.
You are trying to change an attribute of a resource and all the calls to crm_resource fail with:
ERROR: cib_native_perform_op: Call failed: Application of an update diff failed, requesting a full refresh
A call to cibadmin --cib_sync does not help either.
Check the resource you are trying to modify with a call to crm_resource -x -r RESOURCE_ID and see if target_role is present in both meta_attributes and instance_attributes like it is in the following snippet:
<primitive class="stonith" type="external/ibmrsa-telnet" provider="heartbeat" id="resource_stonith_asterix02"> <meta_attributes id="resource_stonith_asterix02_meta_attrs"> <attributes> <nvpair name="target_role" id="resource_stonith_asterix02_metaattr_target_role" value="started"/> </attributes> </meta_attributes> <instance_attributes id="resource_stonith_asterix02_instance_attrs"> <attributes> <nvpair id="1e2ced8d-fcb5-49dc-a0e1-988279531431" name="nodename" value="asterix02"/> <nvpair id="5bc3e50b-f2f5-450a-901a-2b84d2c928c9" name="ip_address" value="10.4.4.67"/> <nvpair id="e9f35ac1-3e32-4f23-9434-329de48de9f5" name="username" value="USER"/> <nvpair id="9c6484b4-f722-4446-8072-b670b45d85d4" name="password" value="PASSWORD"/> </attributes> </instance_attributes> <operations> <op id="3ba4b45f-35bc-4684-a1b7-ab26ab3aced1" name="monitor" description="monitor" interval="5" timeout="20" prereq="nothing" start_delay="0" disabled="false" role="Started" on_fail="restart"/> </operations> <instance_attributes id="resource_stonith_asterix02"> <attributes> <nvpair id="resource_stonith_asterix02-target_role" name="target_role" value="stopped"/> </attributes> </instance_attributes> </primitive>
According to the linux-ha devs this can happen by using the GUI (one more reason not to use it). If that is the case then remove the instance attributes section with a call to (make sure to copy the correct XML snippet from your output) on your DC node:
echo '<instance_attributes id="resource_stonith_asterix02"><attributes><nvpair id="resource_stonith_asterix02-target_role" name="target_role" value="stopped"/></attributes></instance_attributes>' | sudo cibadmin -D -o resources -p
Ignore the error message Application of an update diff failed, requesting a full refresh, the operation should succeed anyway. Now set the target_role value using:
sudo crm_resource --meta -r resource_stonith_asterix02 --set-parameter target_role --property-value stopped
Note the --meta.