What Is HACMP Heartbeating? – [joshd.ca] [~] $

If you have worked with AIX you may have come across the term HACMP (High Availability Cluster Multi-Processing). This is IBM’s solution for high-availability clusters. One of the first things to understand about HACMP is heartbeating, which I will discuss in this blog post.

HACMP heartbeating is essentially like another other type of application using a ‘keep alive’ type functionality. What a heartbeat is, is a User Datagram Protocol (UDP) packet used to monitor the availability of network interfaces, communication devices and IP labels (service, non-service and persistent). It can be used on both IP and non-IP networks. HACMP will maintain information about the status of the interfaces, devices or adapters and through that information, the availability of the cluster nodes.

When HACMP is started on a node it passes the network topology stored in the ODM configuration to Reliable Scalable Cluster Technology (RSCT). RSCT uses the information it receives to construct a ‘heartbeat ring’ and will in turn provide any failure notifications back to HACMP.

Heartbeating takes place by each node sending a heartbeat packet with the expectation to receive a packet over each network within the interval determined by the network sensitivity. Each host will only communicate with it’s two neighbors on each network ring, so it will only receive one packet from a particular node every two heartbeat intervals. This is the important factor in the calculation of how long it takes HACMP to determine a failure.

RSCT will determine that one of the interfaces or adapters on a node has failed if it is no longer receiving heartbeat packets from it, but still receiving heartbeat packets through other interfaces or adapters on the same node. In this type of situation HACMP will preserve the communication to the node by transferring the service labels to another interface on the same network on that same node. This will help prevent against data loss or duplication.

In the situation where all interfaces on that specific HACMP network are unavailable on the same node then HACMP will transfer all resource groups containing those specific IP labels to another node with an available interface. RSCT will consider this node to have failed and HACMP will bring all affected resource groups online on that other operational node.

RSCT will use particular nodes to manage the communications around these groups. The nodes which fulfill the tasks are chosen dynamically and can change each time a node enters or leaves a cluster. There is the group leader which keeps information about the other nodes in the cluster and the network configuration. The group leader backup, which keeps a backup copy of the group leaders topology data and will take over as the leader should the leader leave the cluster. Then you have the mayor which is responsible for ensuring that all nodes in the cluster are informed of any changes to the cluster topology.

HACMP relies heavily on the data received in the heartbeat packets through RSCT for information about the state of the cluster topology. Now there can be situations where HACMP can make an incorrect assumption about the state of the node, for example is RSCT was relying only on TCP/IP network then a failure of a network component between the nodes, such as a switch, would be incorrectly interpreted as a failure of one or more nodes. This is the situation where it is recommended that each cluster have at least one non-IP network defined for each of the nodes in the cluster to prevent any ‘network partitioning’ that could occur due to the failure of a network component, such as the switch mentioned above. A serial network would help HACMP distinguish between an actual node failure and an IP network/subsystem failure.

Enjoy,
Josh Diakun
http://www.joshd.ca