show sidebar
Configuring a Cluster Virtual IP Address (CVIP)

Pre-requisite
-------

Once /etc/cvip.conf is created the ha-lvs init-script will enable
ha-lvs on all nodes in the cluster. See details below on how to set
this file up. From 1.0.0-rc2 on, the ipvsadm package
(http://www.linuxvirtualserver.org/software/ipvs.html) will 
automatically be included as part of OpenSSI.  For older releases,
it must be installed by hand.

Configuration files involved.
-----------------------------

CVIP is cluster virtual IP address, which would be used by outside
world to connect to the cluster. This IP address needs to be different
from the set of IP addresses, within the cluster - ie. the CVIP cannot
be the IP address of the root node. The /etc/cvip.conf is an XML file
that contain the information of the CVIP, director nodes, real
servers, internal gateway and the lvs routing method. The director nodes redirect 
the requests for an HA-LVS configured service to the nodes, also known as real 
servers, running the service. Director nodes has dual functionality as real server and
director.

NOTE: In the case of multiple CVIPs, they should either share all the
director nodes or share nothing. That means more than one CVIP can be
configured on the cluster provided they have the same set of director
nodes or mutually exclusive set.


Example Configuration file  format
----------------------------
<?xml version="1.0"?>
<cvips>
        <routing>DR</routing>
        <cvip>
                <ip_addr>192.168.200.10</ip_addr>
                <gateway>192.168.200.1</gateway>
                <director_node>
                        <node_num>1</node_num>
                        <garp_interface>eth0</garp_interface>
                        <sync_interface>eth0</sync_interface>
                </director_node>
                <director_node>
                        <node_num>2</node_num>
                        <garp_interface>eth0</garp_interface>
                        <sync_interface>eth0</sync_interface>
                </director_node>
                <real_server_node>
                        <node_num>1</node_num>
                </real_server_node>
                <real_server_node>
                        <node_num>2</node_num>
                </real_server_node>

        </cvip>
</cvips>

In the above configuration is for a two node cluster with both the
nodes acting as director and real server. 

The lvs routing method can be DR (direct routing) or NAT.  For NAT based 
configuration you need to specify the <gateway>. DR configuration 
doesn't need the gateway to be specified. <gateway> specified will be configured
on the cluster interconnect interface and the IP specified should be different from 
the already existing interconnect address

The sync_interface is the interface used to sync the load level information
between potential LVS director nodes. Mostly it will be the cluster interconnect. 

garp_interface is the interface through which the gratious ARP messages are sent.That
should be the  interface through which the cluster connect to outside world .


Example of WRONG configuration 
---------------------------
<?xml version="1.0"?>
<cvips>
        <routing>DR</routing>
        <cvip>
                <ip_addr>192.168.200.10</ip_addr>
                <gateway>192.168.200.100</gateway>
                <director_node>
                        <node_num>1</node_num>
                        <garp_interface>eth0</garp_interface>
                        <sync_interface>eth0</sync_interface>
                </director_node>
                <director_node>
                        <node_num>2</node_num>
                        <garp_interface>eth0</garp_interface>
                        <sync_interface>eth0</sync_interface>
                </director_node>
                <real_server_node>
                        <node_num>1</node_num>
                </real_server_node>
                <real_server_node>
                        <node_num>2</node_num>
                </real_server_node>

        </cvip>
        <cvip>
                <ip_addr>192.168.200.11</ip_addr>
                <gateway>192.168.200.100</gateway>
                <director_node>
                        <node_num>1</node_num>
                        <garp_interface>eth0</garp_interface>
                        <sync_interface>eth0</sync_interface>
                </director_node>
                <real_server_node>
                        <node_num>1</node_num>
                </real_server_node>
                <real_server_node>
                        <node_num>2</node_num>
                </real_server_node>

        </cvip>
</cvips>

The above example shows a WRONG configuration for CVIP on openssi
cluster. As you can see the CVIP2 doesn't have the same director node
set (node2 is not in the director node set for CVIP2).


/etc/lvs.VIP.active -> This file is used by the ha-lvs daemon for
keeping track of the CVIP and the current master director node for
that CVIP. User should not modify this file. That same information can
be found from /proc/cluster/lvs if procfs is mounted


Starting the cluster ip service:
--------------------------------

ha-lvs is configured as a service, by the same name. The daemon can be
started, stopped etc using service ha-lvs start or server ha-lvs stop
on RedHat or invoke-rc.d ha-lvs start or invoke-rc.d ha-lvs stop on
Debian. The ha-lvs daemon needs to be started on all nodes in the
cluster, possibly immmediately after the networking. The daemon would
exit after configuring the required network interfaces on pure real
servers, while it would continue to run and monitor the services on
director nodes. This startup script is installed as a part of
openssi-tools installation .


Cluster event handlers:
-----------------------

During the cluster node up/down event, reconfiguration of the lvs cluster is 
taken care by the ha-lvs daemon.

To verify the configuration us the ip command (/sbin/ip addr)
The real server and backup director node will  have the CVIP attached to the 
loopback interface lo.

Configuring of the above mentioned interfaces are done by ha-lvs daemon.
Currently, Failover is not supported in LVS-NAT configuration.

kernel subsystem:
-----------------

The listen and release routines for TCP protocol is modified in such a
way that those IP address get registered with the ipvs master director
node. Any service registration happening in the master director node
is further replicated on all slave director nodes so that the
fail-over happens smoothly.

NOTE: To enable automatic port binding ipvs should be built as a part
of the kernel.

Limitation:
-----------

As of now only if the listen happen on a port that is in the range
specified by setport_weight command it get registered with ipvs. This
is supposed to be removed in the future releases (see below).


Load balancing :
----------------

In order to achieve network based load balacing between different
nodes in a cluster one need to set the port range that need to be load
balanced This can be done using the command
/usr/sbin/setport_weight. At any point the information regarding load
balancing port range with associated weight can be found in
/proc/cluster/ip_vs_portweight. This file is node specific and its
content will be different on different nodes.


When a listen ( listen() ) happen on the load balancing port on any of 
the nodes, it get automagically registered with ipvs master director 
node. This information is then replicated on all slave director node by
the master director.(All the director node have the information regarding
the slave director node for a CVIP. This is done using the set_mdirector
and set_pdirector APIs).  This is needed so that all the services registered
with master director node continues to be up even after the director node 
failover

From the user perspective any bind happening on the 
load balancing port is made cluster wide with master director node 
fail-over capabilities

----------------------------------------------------------
Hope this help in configuring SSI and LVS. If you have any problem 
mail ssic-linux-devel@lists.sourceforge.net/aneesh.kumar@gmail.com

This page last updated on Wed Feb 16 05:05:26 2005 GMT
privacy and legal statement